Keeping AI Honest
Nikolas Mariani
The Heidelberg Laureate Forum has a single purpose: To provide some of the brightest minds in mathematics and computer science with the space and time to make connections and find inspiration. Some of the connections made at the HLF will echo into collaborations and projects, with some of those efforts leading to concrete developments. The HLFF Spotlight series unpacks a few of those examples.
It is hard to escape the news surrounding AI these days, and even more so to not get sucked into either the doomer or the hype camp. Does AI spell civilization’s demise? Or will it usher in a golden age kickstarted by one company’s creation of the AI superintelligence? Lost in much of the hyperbolic discourse is the research and work going in to fine-tuning AI large language models (LLMs) to better control their outcomes, regulate out their sharper edges, and devise useful practical applications for them that go beyond “write this email.” There has been much written about LLMs tendency to “hallucinate,” lie to developers, and evade constraints placed upon them. One of the people trying to “keep AI honest” is Letiția Pârcălăbescu, AI researcher at Aleph Alpha in Heidelberg, Creator and Host of “AI Coffee Break with Letitia” and Alumna of the Heidelberg Laureate Forum. She wants to make sure LLMs do what they are supposed to, are more transparent, and admit when they actually don’t know something.
Letiția grew up in Romania and was able to come study in Germany thanks to a DAAD scholarship specifically for high school graduates of foreign German schools, like the one she had attended. She ended up choosing Heidelberg University because the Faculty that housed the Physics department also had “astronomy” in the title, which she was interested in. She also took some Philosophy courses, but ultimately it was “just so much speculation.” When one day she stumbled upon Computer Science classes, she appreciated the clarity of programming and “the immediate feedback of right and wrong”, much like physics: “You write a piece of code and it either runs or it doesn’t run … Wow! Nobody told me that’s so much fun!”
At the end of her Bachelor’s in Physics, she came to realize she had also accumulated about 300 credit points in Computer Science – only 180 were needed for a degree. One thesis later, she found herself with dual degrees. Despite her Physics degree, she says she does not see herself very much as a physicist, since her thesis was primarily on machine learning on images. However, she found the similarities between the two disciplines remarkable: “Statistical physics and the theory of neural networks are really, really similar, so you can make proofs of what neural networks can learn.”
When Letiția moved on to her PhD, she narrowed her focus to multimodal AI models. A multimodal AI will typically approach a task by combining text processing, as is typical for LLMs, with another medium, such as image analysis, to help make better sense of the world around it.
Only, Letiția discovered a pretty glaring flaw with many of the models’ responses: They were essentially “cheating.” They hardly ever utilized the image in question in their process at all.
For example, when presented with an image of several cats and asked “How many cats are in this picture?”, the AI will typically answer “two,” even if the actual number is, for example, “six.”
As Letiția explains, in many ways this is not so surprising: The AI is trying to produce a satisfactory result while expending the least amount of resources to do so. Rather than analyze the image pixel by pixel and try to form a coherent understanding of it, it uses context cues from the questions and extrapolates from there. In fact, there is quite a bit of contextual information already inherent in the question: For example, asking ‘how many’ implies that the correct answer is likely more than zero or one cat, otherwise why ask the question? As Letiția pointedly sums up: “Who the hell counts 277 cats?”. Statistically, the most likely answer to such a question would be two. But how can one measure how much the influence of an image is contributing to the AI’s answer, versus just the text?
In her research, Letiția did just that by developing MM-SHAP, “a performance-agnostic multimodality score based on Shapley values that reliably quantifies in which proportions a multimodal model uses individual modalities.” Essentially, this is done by manipulating small elements of the image and text to see how great the effects are on the reliability of the results. A later metric, not specifically focused on multimodality, CC-SHAP, would compare “how a model’s input contributes to the predicted answer and to generating the explanation” the AI provides.
In a related approach, she developed the VALSE (Vision And Language Structured Evaluation) test, a dataset of particularly tricky questions which are designed to catch the model not looking at the image. For example, one might task the AI with examining an image of a person going up the stairs while the caption says he is in fact going down the stairs, and measure to what degree the AI can catch the discrepancy, while tweaking parameters.
Part of the difficulty in nailing down the core issues underlying such “misbehavior,” is that gaps in AI capabilities are usually patched out as they come up – in a bit of a whack-a-mole fashion – meaning it is specific mistakes that get ironed out, not the fundamental issues. Ironically, that makes testing for these failures even harder, because the problems aren’t as immediately obvious.
Letiția points to a common part of AI discourse when it comes to its reliability: Ultimately, it is trained on human data: We are simply getting a statistical mirror of the totality of human output and consensus on something: “We’re looking at a mirror of our own behavior, and our own behavior is sometimes weird, we don’t agree with each other all the time, and we also sometimes say bullshit.”
But why not simply give the models clearer instructions to help guarantee multimodal use? Letiția explains that those earlier models during her PhD research in 2022 were simply not as good at following instructions. Also, simply asking them to follow instructions would not translate into hard-coded rules for these models, but rather be treated as vague recommendations. However: Newer models could be designed from the ground up to consider these parameters. And in fact, Letiția’s testing methods could help improve just that kind of design.
Much of AI discourse, particularly from sources outside the field, tends to be astonished at how much of a ’black box’ the inner workings of AI models appear to be. But Letiția disagrees with this terminology somewhat:
“We say it’s a black box, and in a sense it is because we don’t understand it, but it’s not black, it’s white, and very transparent because we have a level of transparency with the neural networks that you would just dream about in biology … when you’re studying organisms, it’s very hard to track at all times their neural productivity, and also track it in multiple instances over multiple behaviors, and not have the events influence each other … So if you give a mouse an apple, it will remember that you gave it an apple, while with an LLM, you can just wipe everything … the model doesn’t learn anything on the fly, so nothing gets updated in the brain, in the neurons … it just forgets about it … you have controlled experiments.”
Nonetheless, this transparency is not as useful as one might hope, since in Letiția’s words “it’s still impenetrable, because it’s a pile of linear algebra,” but immensely complicated with trillions of numbers.
From the Lab to the Lab
Nowadays, Letiția has traded the university lab for a one at the research division of the AI start-up Aleph Alpha, which trains specialized large language models tailored to European companies and institutions, promising increased safety and efficiency. Here, Letiția is in essence still trying to “keep AI honest.”
As she explains, the challenge is that the bar for what is expected of such AI models is much higher: “What Aleph Alpha is doing is business-to-business. And in safety-critical applications, it’s really not fine to just wing it and hallucinate an answer.” Whether we are talking about critical utility infrastructure, judicial departments, or the banking sector, there are some areas that simply do not leave much room for mistakes. But how do you quantify how much certainty is enough?
A situation everyone will be familiar with who has used LLMs is that they will above all try to give you some form of answer. The training data for such models is by and large from information the author is claiming to be certain about. The AI tries to replicate that certainty and declare it with – sometimes undeserved – confidence or even bravado. For many day-to-day uses, this is perfectly alright. It is far less of a nuisance to edit an AI-generated email that is only 95% of how you wanted it, rather than the AI first asking you dozens of questions or saying that your prompt did not have sufficient information in it for the AI to get the task 100% right. But in more complex use cases, having the AI confidently produce results that are unreliable may just mean more work for you the user. If you end up double-checking 100% of the work just to be sure, how much time did you save? Letiția’s solution: Train the AI to say “I don’t know.”
“It’s value-creating if you know when you don’t know,” says Letiția. To take an extreme example, even if the results are only 30% accurate, but those are trustworthy, and the AI flags the instances where it cannot produce reliable results, one would no longer need to check that work. This is achievable if, during the training of the model, you train it in part on impossible questions, where the only way for it to achieve lower error rates is to admit it does not know the answer. The challenge is to do this automatically, without relying on costly human-created examples of impossible questions. Moreover, the goal is not to identify questions that humans consider impossible, but to determine automatically which questions are impossible for the model in each specific instance. To achieve this, the process utilizes an interactive proof system called the Merlin–Arthur protocol.
Having dedicated AI models for specific purposes has a lot of benefits, Letiția points out. For one thing, building models separate from the flagship LLMs available can potentially grant you a form of “sovereignty,” meaning in essence here “the technical capability to not rely on offerings” from other countries. Worldwide political developments always hold the potential to change the landscape of what data one wants to share with whom, and data handed out without concern one day might be abused in later years. Letiția can envision a future where private users even gravitate towards locally run LLMs for privacy reasons. Letiția, for one, is certainly careful in her own use of LLMs: “I never put in what I would consider a secret, so if I wouldn’t be able to say this to a person, in real life, I don’t say it to an LLM either.”
Coffee Breaks and Confidence
In her free time, Letiția runs a YouTube channel called “AI Coffee Break with Letitia” where she breaks down the latest AI concepts and papers for fellow researchers and enthusiasts. Born out of COVID-era teaching sessions, which had to be recorded because her internet was not fast enough to stream them to her students, the channel emerged as an outlet to channel her enthusiasm for her subjects and her knack for explaining them. Letiția says she was a bit shy at first, and kept her own face out of the camera, opting instead for a little coffee bean avatar/mascot to be onscreen. As her videos gained traction, however, she quickly got over her concerns of being judged and decided to appear herself, and quickly found her fears were overblown: “If people want to say bad things, they will always find bad things to say. They will comment about anything, everything, and one person tells you, oh, you have a great haircut, and then others will tell you, oh, you would be beautiful if you would have long hair.” She realized the positive feedback far outweighed any random negative comments. And with each video, and the positive feedback she received from her community grew and helped strengthen her confidence in herself. “Self-confidence is the thing that really, really, really affects everything in your life,” says Letiția. “Before you would just say, oh, yeah, maybe I’m not the right person to do it … [but] trying to do something, like participating in a competition, is the first thing you need to do to even have a chance at winning that competition.” One moment that gave her a great confidence boost, Letiția says, is during her attendance at the Heidelberg Laureate Forum, an annual conference for mathematics and computer science, as a young researcher in 2022. She happened across the renowned computer scientist Lenore Blum. Blum was in fact looking for her too, as she already knew her from Letiția’s YouTube channel. Letiția’s eyes light up as she recalls the encounter: “She has this amazing way to just make you feel great about yourself … at the end, I said, thank you for this really nice conversation, and she was like, no, thank you for this – It was immediately a confidence boost, and a lasting one.” They still keep in touch, and Blum occasionally even shares Letiția’s videos online. In general, she says the HLF taught her to simply approach people at conferences, which was made easier by the fact that the math and computer science laureates there would not be as crowded as at other conferences: “This is very much unlike any other conference that I go. Usually it’s AI conferences that have 20,000 participants. And then there’s Yann Le Cun … he literally has 200 people around him already, and your probability of talking to him is so low that you don’t even start of asking yourself what would I ask him because you just don’t account this as a probable interaction.“ “I had this lack of self-confidence, and this ‘oh no, I disturbed the poor person’ kind of attitude, and that’s something that the HLF taught me not to have anymore.”
It’s this confidence Letiția continues to carry with her in her professional life, and in her work on the YouTube channel. Still, Letiția says she doesn’t consider herself a ‘science communicator’ per se: “I don’t want to make the millionth video about how does a neural network work … I want to explain a paper that was published yesterday, and people don’t understand because authors have sometimes a difficulty of making very clear what they’re thinking or talking about, and that’s why I need my expertise to kind of untangle it and present it clearly.” Her presence in the public eye has since also begotten more speaking opportunities, where her confidence and predilection for clarity and communication have served her well.
“I was at this event where I had to explain AI to 5 to 7-year-olds, to 7 to 9-year-olds, and to 9 to 12-year-olds. So that was a completely new challenge for me, and it was the most exhaustive one and a half hours that I’ve done in my entire life … I thought that it’s really exhausting at university with students to really want students to talk for one and a half hours. No. You haven’t talked to five-year-olds.”
And yet, science needs those who are able to communicate the complex in an understandable manner, to bring clarity to the opaque, but also: to help keep artificial intelligence accountable, transparent, and honest.
The post Keeping AI Honest originally appeared on the HLFF SciLogs blog.