The Problem with AI Discrimination

Andrei Mihai

In 2018, MIT computer scientist Joy Buolamwini discovered something disturbing: Leading facial recognition systems often failed to identify darker-skinned women, with error rates as high as 34%, while barely missing lighter-skinned men (~0.8%). It was relatively early days in the AI race, but it was a concerning find nonetheless. Fast forward to today, and AI models have been repeatedly shown to discriminate based on gender and race in everything from job selection to healthcare. 

The advent of artificial intelligence (AI) has taken the world by storm. It seems to have captured our entire psyche, spilling into every aspect of life we can imagine. Yet, despite all the progress it brings (and some exaggerated hype), AI also brings several challenges. For one, it consumes a lot of water and energy (as much as a sizeable country); secondly, there is a lack of transparency and ethics around the technology; and last but not least, it is often marked by bias and discrimination.

For decades, many hoped that by replacing human discretion with the impartial logic of computers, society could mitigate or maybe even eliminate the unfair discrimination that has long plagued our society. This optimism, however, is challenged by reality. AI systems are not inherently neutral; quite the opposite. They can easily become powerful instruments for perpetuating and even amplifying existing societal, institutional, and human biases.

The Good, the Bad, and the Data

The common maxim “garbage in, garbage out” sounds rough, but it is a useful starting point. The most frequently cited source of algorithmic bias resides in the data used to train machine learning models. If the data is a flawed or skewed representation of reality, the model will learn and systematize those flaws.

This manifests itself in several ways.

Historical bias arises when AI models are trained on data reflecting past prejudices and societal norms that may no longer be considered acceptable or accurate. If you feed an AI data on how a particular race or gender or religion is superior, it will incorporate that idea and perpetuate the bias. Because datasets are often historical collections influenced by society itself, discrimination is effectively encoded before the data is even collected. A hypothetical AI system for loan approvals trained on historical income data would look at past data and see that more men have taken loans in the past and repaid them, and feel more inclined to believe that men are more reliable. Similarly, a woman programmer would be more likely to get rejected for a job application because fewer women have historically held such positions.

Other insidious problems, like proxy bias (where an algorithm uses seemingly neutral variables as proxies for attributes like race, gender, or socioeconomic status), can also cause problems, as highlighted by a 2019 study. In the study, systemic inequalities in healthcare access flagged Black patients as having lower health needs, systematically disadvantaging them. But AI does not just repeat the past; it can magnify it. Models can latch onto group identity as a “shortcut” in making predictions, strengthening stereotypes beyond what the data shows. Once deployed, their decisions can create feedback loops.

Eliminating all of these biases from the data is difficult enough. But “garbage in, garbage out“ doesn’t capture the entire scope of the problem. Bias is not solely a product of flawed data; it can also be introduced and amplified by the design and inner workings of the AI model itself.

Beyond the Data

Even if you somehow feed an AI model the cleanest, most representative data possible, bias can still creep in. This is because algorithms are not neutral vessels. The choices made during their design, the mathematical structures they use, and the way they are tuned can all introduce their own distortions. These are the biases that emerge beyond the data.

One of the most common sources is algorithmic design bias. Every AI system begins with a series of human decisions: what features to include, how to weight them, and how to define “success.” These choices can embed the values and assumptions of the development team, often without their conscious awareness. For example, early speech recognition systems consistently underperformed for women’s voices but not because of the data or because the developers intended it. Rather, it was because the models were trained and optimized mostly on recordings of male voices.

Another challenge is the “black box” problem, or understanding how AI makes decisions. Modern AI, especially deep learning, operates with millions or billions of parameters arranged in ways that defy straightforward interpretation. Developers can tell you that the system produces an answer, but not exactly why. This opacity means the model can discover hidden correlations that no one planned for. Those correlations could sometimes be linked to race, gender, or socioeconomic status, and yet remain invisible to anyone reviewing the system.

A newer frontier of concern is architectural bias. This is a type of bias baked into the mathematical structure of a model, independent of the data it sees. Large language models (LLMs), for instance, use a design called a transformer architecture, which researchers have found can cause “position bias”: a tendency to give more weight to information at the beginning and end of a text, while neglecting the middle. Other studies have found “first-position bias” in AI hiring tools, where the model disproportionately selects the first résumé it’s shown, regardless of content. These are quirks of the machine’s inner workings that are very difficult to even out.

Then, the real-world interaction can also have an effect.

Once an AI system is deployed, its decisions don’t just sit in isolation. The system interacts with its users and/or its environment, which creates feedback loops that strengthen the original bias. Over time, these loops can make discrimination more entrenched, harder to spot, and far more damaging.

Another problem with AI data is measurement bias, a type of bias introduced when the data we collect differs from what we actually want to measure. In predictive policing, for instance, arrest counts are proxies for police activity, not crime itself. If patrols have historically concentrated in certain communities, algorithms trained on those arrests will flag those areas as high-crime, creating a feedback loop of biased enforcement. This dynamic is clearest in “Minority Report”-style prediction.

Suppose an algorithm is trained on historical arrest data. If those records reflect decades of over-policing in certain neighborhoods, the AI will “learn” that these areas are high-crime zones. When police follow the algorithm’s guidance and patrol those areas more heavily, they inevitably make more arrests, often for minor offenses that might go unnoticed elsewhere. Those new arrests feed back into the dataset, “proving” to the AI that its initial prediction was correct. The cycle repeats, each turn deepening the association between certain neighborhoods and criminality, regardless of the true crime rate.

Feedback loops are particularly insidious because they launder bias through the language of objectivity. When a human makes a prejudiced decision, we can call it out. When an algorithm does it, it brings “data” and “predictions” that make the result appear neutral, even scientific, even though it carries a hefty amount of bias.

What Can Be Done?

Eliminating AI bias is so difficult because it’s not just a matter of cleaning bad data, it’s a problem woven into every layer of the system. AI bias is a “wicked problem.” It doesn’t have a single fix or a switch that can make everything better. The bias is so deeply and systemically embedded you need a multiple-pronged approach to even have a chance.

Technical fixes, starting from cleaning and rebalancing of datasets even before training, are an important starting point. Modifying the learning algorithm to optimize for fairness and accuracy is also important. Surprising approaches like “vaccinating” an AI with bad data can also work, as highlighted by recent research.

Yet, ultimately, technical fixes alone will not be sufficient.

Fairness itself, particularly in regards to machine learning, has no universal definition. A model might be “fair” according to one metric, yet blatantly unfair by another. Imagine an algorithm used to decide who should get extra screening for a disease. One definition of fairness might say that, for any given risk score, people from different groups should have the same likelihood of actually having the disease. Another definition might say the system should make the same kinds of mistakes (false alarms and missed cases) at the same rate for every group. If the underlying rates of the disease differ between groups, mathematics makes it impossible to satisfy both definitions at once. Deciding which one to prioritize isn’t something the algorithm can do for us, it is a societal and moral issue.

As it so often happens, technology has evolved faster than our moral code. This is not at all particular to AI, but given how quickly this technology has advanced, it is a striking example.

Governance and oversight have also fallen behind. In the race for AI supremacy, ethics and bias are sometimes considered an afterthought. Inside companies, AI ethics boards and “human-in-the-loop” decision-making are often touted as safeguards. But these mechanisms work only if the humans involved are trained, empowered, and diverse enough to spot potential harms. Without genuine independence and enforcement power, ethics boards can become little more than PR exercises.

In the end, “debiasing” AI is bound to be a process more than a goal. It relies on constant scrutiny, correction, and open debate about the values we want our machines to reflect. Without that vigilance, AI will inherit the problems already present in our society and amplify them for the future.

The post The Problem with AI Discrimination originally appeared on the HLFF SciLogs blog.