Individual responsibility for ethical AI use? Nope.

Markus Pössel

The present and future role of AI for mathematics, computer science, and science in general was such a pervading topic at this year’s Heidelberg Laureate Forum that at least one laureate felt moved to apologize on stage that AI would not play a role in their talk. Between the recent claim by OpenAI to have solved one of the six Millennium Problems, namely showing that the Navier–Stokes equation for fluid dynamics can produce unphysical solutions, and the various researchers’ (and their students’/mentees’) own experiences with using generative and agentive AI, there is a general sense that mathematics in particular is in the midst of a major upheaval (“mathematics is hosed,” as one senior participant put it). Andrei Mihai has documented some of the on-stage contributions so far in his blog post “Can We Turn AI Into Scientists? And Most Importantly, Can We Make Them Safe?”.

For this blog post, I was initially all set to explore what AI can or cannot do when it comes to scientific research, both in maths and computer science and in other sciences. But one particular part of the panel disscussion on “AI in Mathematical Research” on Tuesday (which I couldn’t attend live due to the requirements of my day job, but which is available on YouTube here) stopped me in my tracks. As part of the opening statements, panelist Michael Harris quoted from science fiction writer Ted Chiang’s article in the Atlantic: “Many people feel that LLMs are a fundamentally unethical technology because they are built on the theft of intellectual property, rely on exploited labor, waste natural resources, spread misinformation, deskill workers, stunt the cognitive development of students, and contribute to a consolidation of power that is unhealthy for a democratic society.”

Whether you agree with all (or any) parts of this statement or not, it’s a challenge. Part of acting ethically is to keep your ears open for signs that you might have gone astray. This doesn’t mean we cannot or shouldn’t filter – like many whose work involves Einstein’s theories of relativity, I regularly get messages accusing me of being part of a worldwide conspiracy against common sense, and yes, I routinely ignore those missives. But if you are in effect getting called out by a fellow panelist at a high-profile event like the Heidelberg Laureate Forum, I think it would be fair to say that it is on you to not just ignore the message. But unfortunately, that is what happened in this case. Suddenly, there was a massive call-out. Which everybody on stage then went on to ignore.

Only much later, about 50 minutes into the panel, the pre-planned list of questions belatedly caught up. Panelists were individually asked “Do you have ethical concerns about the usage of AI in mathematics?” The round of answers started with Australian mathematician Geordie Williamson, whose immediate ethical concern was not with AI itself, but with how “ridiculous” concerns about AI electricity are, in his opinion, likening the GPU sitting next to him to a “little toaster,” pointing to the Heidelberg Laureate Forum as “we’ve all probably flown here” which, he presumes, accounts for “how many orders of magnitude more” in CO2 emissions. Then (to my ears a bit incongruously) he followed up with mathematicians just not being “used to the fact that we need to consider ethics in what we’re doing,” claimed that mathematicians are getting used to “this” now, and expresses his gratitude to fellow panelist Michael Harris for constantly reminding him to “try to do this.”

In other words: Immediately following the being-called-out, no ethical reflections, but reflexive defense, and with a pattern that is all too familiar from many discussions. Claiming that each individual contribution is negligibly small is not something you expect from mathematicians, who should be expected to realize that when the problem is the sum, each part of the sum contributes. When an outlet like Bloomberg, surely above suspicion when it comes to tree-hugging hippieness, projects that within ten years data centers are “are projected to consume more natural gas than most countries,” with all the negative consequences that kind of contribution entails in the current climate crisis, rejecting responsibility altogether is not what one would call an adequate response. And the CO2 contribution of international flights? And in any sensible discussion of ethics, saying “but we all do this other thing, too, and that is worse!” isn’t a get-out-of-jail-free card either. If you’re doing (at least) two problematic things, surely the decent thing is to reflect on both? Not on neither?

I would have loved one of the other participants to respond to Williamson. But that is not how this panel worked. It wasn’t designed to encourage discussion. It was set up to follow a fixed pattern: The moderator asks a question. Each participant answers in their turn. So, no counterarguments for Williamson, but instead, the next answer: by Peter Scholze, brilliant young mathematician and Fields medal winner. Scholze referenced the original list of concerns (which he “broadly agrees with” he said), and followed up with, wait for it: “one possible solution is […] that we could build some kind of academia inside open source whatever kind of large language model, which might address such concerns.” (Scholze, by the way, is one of the panelists who does not use AI in his own research.) And yet, that was it, next panelist please.

Jacob Tsimerman, one of this year’s Fields medalists who shortly after announced that he would be joining OpenAI’s safety department, and shortly after that that he was funding the independent AI Safety Institute MAISI, was, for the greatest part of his answer, remarkably vague. He acknowledged that there are “just so many concerns around AI.” By his own statements, he is more concerned with sweeping “exctinction” scenarios of AI somehow doing away with all of humanity; topics like AI’s contribution to climate change remain discretely in the background. Historically, major shifts have always been accompanied by “turmoil,” norms need to shift, yet we should (somehow?) do the inevitable breaking of “value propositions and standards” “thoughtfully” and “mindfully.” Hm.

Then it’s Michael Harris’s turn. Harris regularly publishes the Silicon Reckoner newsletter, had fired the opening shots in what might have been an ethics debate if this panel were to actually set up for debate, and weirdly enough begins his reply by citing Peter Scholze. It turns out that Scholze, contrary to his own answer in this panel, has quite concisely pointed to the ethical problems of AI, in a quote for the German “Pause AI campaign” (the third quote in the gallery here). In that quote, Scholze calls out AI as harmful to humanity, democracy, and the planet. Harris’s own reply concentrates on two specific aspects: AI frontier labs, he says, are developing a model to sell to somebody. If you interact with those models, you have to assume your input is being used to train them for future applications, including military and surveillance technology. That is what people who use AI for mathematics are contributing to. He also brings up the problem of AI putting those mathematicians working in industry and other businesses out of work (a problem that goes beyond mathematicians).

Then it’s the moderator’s turn: “So, thank you all for your thoughts on this.” And off he goes with the next question. No follow-up. That was it for ethics. Moving on. The questions from the audience in the last part of the panel do not revisit the issues of energy consumptions or intellectual theft either. The audience does not ask anything about environmental impacts. Which is consistent with the survey at the opening ceremony, in which only 7% saw environmental impact as the biggest challenge with the growth of AI over the coming years. Even the question pair “for those of you who aren’t big fans of AI in mathematics, what would it take for you to use it? And then for those of you who do use AI, what events made you start using LLMs yourself?” omits the very natural third question, namely: just how unethical would AI need to be for the current users to stop using it?

To be fair: The question of individual responsibility is always a difficult one when it comes to problems that are in some way or another broader, or even systemic. To pick climate change as an example, there is only so much an individual can do when the choices are just not there – when some locations are only reachable by plane, not by train, or when those who rent their apartment cannot actively influence whether their heating relies on fossil fuels or renewable energies, or, as a consumer, whether or not the manufactured goods on offer are produced in a sustainable manner or not. But while it’s difficult to find the optimal, or even just a good balance for what individuals can do, it’s not difficult to identify certain behaviors as problematic. Looking at the systemic contribution and the difficulties of making a difference, we shouldn’t just shrug and decide that, since it doesn’t matter all that much what we as individuals do, we shouldn’t even try. If we are given a platform, we certainly should take care not to serve as negative role models for others, modeling this kind of attitude. But looking at the role models on display at this panel, it doesn’t seem to me that when it came to those ethical challenges that went beyond the impact on mathematics as a discipline, it was exactly this kind of attitude that was on display. Also, there’s a tendency within our society to just not talk about certain serious problems (climate change again comes to mind as an example), as if that would make them go away. Raising such topics whenever one is given a platform is a simple and straightforward individual contribution. It’s difficult enough for societies to effect systemic change where it’s needed. It doesn’t get easier if those in positions of responsibility act as if there’s nothing they can do on an individual level.

From the perspective of how to adequately react to ethical challenges, I found the panel disappointing. But maybe I shouldn’t have. Maybe it was just a reminder that when it comes to a discussion of wider ethical issues, it would make sense to have a panel drawn from a wider pool of people – beyond the mathematicians and computer scientists that make up the core population of the Heidelberg Laureate Forum. So if I had a wish, it would be this: For a future HLF to address the ethical challenges of AI by inviting a panel that combines mathematicians and computer scientists with philosophers, climate scientists and other stakeholders, in a format that encourages panelists to interact, discuss, and argue with each other.

The post Individual responsibility for ethical AI use? Nope. originally appeared on the HLFF SciLogs blog.