Micah Reed HysongArtificial intelligence (AI) has become a hot topic and for good reason. Whether we are aware of it or not, AI is becoming increasingly embedded in our everyday lives. And I am not just talking about chatbots, social media algorithms, or Netflix recommendations: AI is now routinely used to make major decisions in hiring, medical diagnosis, and money lending.
With AI’s rapidly increasing influence in such important decisions, many people are wondering whether we can trust it. Some people assume that because computers rely on math, they must be objective and are incapable of making mistakes. But is this true? Is AI always right?
How Does AI Work?
What about large language models (LLMs) like chatbots? They work differently, but the basic idea of pattern recognition still applies. During training, an LLM is exposed to enormous amounts of text and learns relationships between words and phrases. When you give an LLM a prompt, it uses those learned patterns to predict what text is most likely to come next. Text autocomplete is a good example of this. If you type “the cat is stuck in the”, the computer may predict that “tree” is the word you are most likely going to type next, simply based on past instances where those words have been used in that combination.
So, AI uses existing data its been given (training data) to learn patterns and then applies those patterns to make predictions of new data, but how accurate are those predictions?
Systematic Biases
Bias can enter an AI system at multiple stages of its development, anywhere from the data used to train it to the way its predictions are evaluated and applied. Let’s look at three major sources of bias.
Misclassification
Let’s return to our brain tumor example. Imagine that a high school student was tasked with labeling all the initial brain scans for whether there was a tumor present. Lacking the years of expertise required to assess the scans, the labels would likely not be much better than a random guess. In this case, the AI would probably not be very accurate at making new predictions. This is referred to as misclassification bias. While the AI would likely be more accurate if the initial training labels were generated by experienced neurosurgeons and radiologists, even experts make mistakes, and thus misclassification can still occur.
Underrepresentation
Now, what happens when the training data do not adequately represent the populations or situations in which the AI will eventually be used? The system may learn patterns that work well for the groups it has seen but fail to generalize to groups that are underrepresented in the training data. To continue with the above example, this could look like applying an AI brain tumor detector trained entirely on adults to children. Not having encountered children’s brain scans before, the AI will be more likely to make predictive errors. There are two real-world examples that illustrate this generalizability problem.
While trying to use a facial recognition app for a class project at MIT, Dr. Joy Buolamwini made a striking discovery. The app was unable to recognize her face; that is, until she put on a white mask. She ran many experiments and concluded that the facial recognition technology was far better at recognizing white faces than faces with darker skin or women’s faces. Based on what we learned about how AI works, why might that be? It was because the facial recognition AI was trained almost exclusively on white male faces, so when it came time to make predictions on faces with darker skin or women, it didn’t perform so well.
The same underlying problem appears in personalized genetic medicine. Polygenic risk scores estimate an individual’s genetic risk for disease by adding up the small effects of hundreds to millions of genes. Because these scores have predominately been trained on individuals with European ancestry, they don’t perform as well in groups with non-European ancestry. This means that polygenic risk scores will unequally benefit people with European ancestry and potentially harm people of non-European ancestry by providing potentially inaccurate information, further contributing to health disparities.
Both examples illustrate the same fundamental problem: when important groups are missing or underrepresented from AI training data, the system can reproduce those gaps in its predictions. This is also why large language models often perform better in English than in other languages.
Biased historical patterns
But what happens when the data is “correct” but reflects human biases or inequalities? While AI can recognize patterns, it does not reason whether those patterns are fair or justified. Because AI uses patterns it learns from the training data to make predictions, it can cement discrimination into its decision-making. Let’s look at two examples.
Amazon developed an AI tool to help screen applicants for technical jobs. To train it, the company used resumés from past applicants, and told the AI whether each of those applicants were hired. The system learned patterns in those resumés to score and rank future applicants. But something unexpected happened. The AI demonstrated a gender bias: it favored male applicants. Why? Because in the historical data, men were far more likely to be hired for technical roles. So even though the engineers didn’t tell the AI to prefer male applicants, AI learned this bias from its training data and rated resumés with “women’s” activities or degrees from women’s colleges lower.
Now consider another high-stakes decision: whether someone can get a mortgage to buy a home. Research has found that Black, Hispanic, and other borrowers of color have faced higher mortgage denial rates than White borrowers with similar measured financial characteristics. Why does this matter for AI? If an AI model were to be trained to inform banks of lending risks, it would likely pick up on this racial discrimination and use it in its decision-making. So even if race wasn’t intentionally used to make lending decisions, the AI could be using historical data to reinforce racial inequality.
The Black Box
In Amazon’s hiring example, engineers were eventually able to identify gender bias in the system. But detecting that bias required them to investigate how the model was making decisions and recognize that certain patterns were problematic. And here’s the challenge: even when we can see what information goes into an AI system and what decisions come out, it can be difficult to understand exactly why it made a particular decision. This is called the black box problem. Modern AI systems can identify patterns across millions of variables and complex relationships that humans may not be able to easily interpret. In some cases, even the people who build these systems cannot fully explain why a particular input produces a particular output.
If AI is deciding whether you get a job, a loan, or a medical diagnosis, it is important to be able to understand and justify how it arrived at its decision. Imagine being denied a loan and the people overseeing the system can’t justify the decision or explain to you why you were denied. How could you challenge the decision? And if we don’t know what patterns a system is using, we might not realize that it is relying on misleading information, reproducing historical discrimination, or performing poorly for particular groups of people until real harm has occurred. The problem isn’t simply that AI can make mistakes. It’s that we may not know when it’s making them, why it’s making them, or who should be held accountable.
Control: Who Decides the Truth?
AI’s behavior is shaped not only by its training data but also by the instructions given to it by the people who build and control it. That raises another ethical question: Who gets to decide what information is reliable, what counts as misinformation, and what perspectives an AI system should prioritize? In an era when even basic facts are politically contested, this influence raises important questions.
The media attention surrounding Grok in 2025 provides a perfect example of how AI can be manipulated to push certain perspectives. Grok is an AI chatbot developed by Elon Musk’s company, xAI. Grok pulls information from the social media platform X, also owned by Musk. Many users reported that Grok gave distorted views of its owner Musk, saying Musk was more athletic than LeBron James, could beat Mike Tyson in a fight, and could resurrect faster than Jesus Christ. As ridiculous as these examples may sound, they illustrate a serious concern: the people who control AI systems can influence the information and perspectives those systems present as the “truth” and use it to reflect their interests and worldviews.
So, can we trust AI? Unfortunately, there’s not a simple answer. But, there are steps we can take to make AI more trustworthy. Through regulation, transparency, independent evaluation, and accountability, we can help ensure that these systems are safe, fair, and working in public interest.
How Do We Make AI Trustworthy?
Recently six tech CEOs got together and signed a joint commitment to the safe development of AI. While such an agreement is a step in the right direction, can we trust these companies to regulate themselves? This question is especially relevant because some of the industry’s own leaders have called for slowing the pace of AI development to give safety greater priority. Yet the U.S. federal government has favored a more voluntary approach, emphasizing industry-led safeguards rather than binding federal requirements. So, what would it take to increase trust in AI?
Transparency: Can we understand how AI is being used?
Transparency is fundamental to building trust in all sectors of science. The public have a right to know how companies are developing their AI: what data they are trained on, what instructions and rules they are given, and their limitations. People should be informed when AI is being used to make decisions and should receive meaningful information about how that AI system reached its decision.
Independent evaluation and equity monitoring: Does AI work safely and fairly for everyone?
There should be clear standards for evaluating AI accuracy, safety, and performance across different populations, supported by independent evaluators, like the Algorithmic Justice League, who can identify disparities and hold developers accountable.
Accountability and the right to challenge decisions: What happens when AI gets it wrong?
Companies and institutions that use AI should be responsible for the consequences of their systems. People affected by consequential AI-driven decisions should have a way to appeal those decisions, obtain meaningful human review, and seek remedies when they are harmed. Users can also be expected to understand the limitations of AI tools and take responsibility for how they use them. However, signing a disclaimer acknowledging these risks should not absolve companies or institutions of responsibility for foreseeable harms, particularly when they control how AI systems are developed, deployed, and used to make decisions.
Conclusion
AI is increasingly being used to make high-impact decisions, from hiring and lending to healthcare. Although AI has enormous potential for advancements, it also can cause serious harm. AI can reproduce existing inequalities, make errors that are difficult to detect, and reflect the priorities of the people and organizations that control it.
Making AI more trustworthy requires more than technical improvements or promises from technology companies. It requires transparency about how these systems are developed and used, independent evaluation of their accuracy and fairness across populations, and avenues for accountability when they cause harm.
While important, increasing AI accuracy should not be seen as the ultimate goal: AI must also be used in ways that are safe, equitable, and accountable.
Edited by: Julia Drennan and Karen Wang
Illustrated by: Priscila Santa Rosa
