While the rest of the world counted the days while waiting for our favorite TV shows and movies to return, Charlie Crawford ’24 found inspiration for his thesis in last year’s Writers Guild of America strike.
Throughout the 148-day strike, which ground Hollywood and production studios to a halt, the union continually expressed its concern over the rise of AI and large language models like ChatGPT and its impact on the livelihood of its membership. Crawford wanted to explore unintended biases ChatGPT might introduce when it is asked to generate a script for a TV show.
He credits his advisor, Shibulal Family Computer Science Professor Sorelle Friedler, for channeling his interest in machine bias and the ethical use of AI into a thesis centered on the writers guild’s chief concern. The resulting work, “Scene and Unseen: GPT Bias in Script Writing,” was developed with Friedler and a small team of researchers at the University of Pennsylvania and Pennsylvania State University.
“The research I did for my thesis was always alongside Sorelle and this team, so I have them to thank for the work we were able to accomplish,” says Crawford, a computer science major and environmental studies minor.
Through its research, the team found that models produced by OpenAI, the tech company behind ChatGPT, have a pronounced tendency to censor themselves at a very low threshold, Crawford says. Out of the 81 real scripts they fed into the models, about 70% were flagged for content violations, including 50% of PG-rated scripts. Those results, he says, prove that, at least for now, screenwriters are irreplaceable.
“These findings showed us that GPT models cannot adeptly navigate the balance of representing shows that people have come to love and preventing toxic content,” Crawford says. “That’s a balance only humans can find.”
For Crawford, the exploration of ChatGTP became much more than a thesis. With support from the team, his work grew into a full-fledged research paper that was presented at the seventh annual ACM Conference on Fairness, Accountability, and Transparency in early June. In witnessing his thesis’ growth from its earliest stages, Crawford says he gained a much deeper understanding of how a research paper is created, from ideation to publication. The team’s small size, he says, Crawford played an integral role in developing the structure and code for its research, giving him an added level of responsibility within the project.
“My thesis has sparked a love of research within me, though I have not decided what direction that will take me in. I am currently interested in pursuing more environmental research opportunities, but ultimately my dream would be to work at the intersection of environmental studies and computer science,” Crawford says. “Whether or not that intersection involves screenwriting or AI, I know that being part of this project has given me the skills I would need to make that dream a reality.”