Q&A: How to Help Students Detect Bias in AI Datasets for Medical Applications

Each year, countless students enroll in courses focused on deploying artificial intelligence (AI) models to assist healthcare professionals in diagnosing diseases and recommending treatments. Despite the importance of this education, many courses overlook a crucial aspect: teaching students how to identify and address biases in the data they use to build these models.

Leo Anthony Celi, a senior research scientist at MIT's Institute for Medical Engineering and Science, physician at Beth Israel Deaconess Medical Center, and Harvard Medical School professor, highlights these gaps in a recent publication. His research emphasizes the necessity for curricula to incorporate thorough evaluations of data quality and bias, aiming to prepare future developers to recognize and mitigate data flaws.

One leading example of bias in medical datasets involves pulse oximeters, which tend to overestimate oxygen saturation levels in people of color. This discrepancy arises because clinical trials for these devices often lacked sufficient representation of diverse populations. Historically, medical devices and equipment have been optimized based on healthy young male subjects, neglecting variations in age, gender, ethnicity, and health conditions, thus limiting their effectiveness across diverse patient groups.

Furthermore, the electronic health records (EHR) systems often serve as unreliable sources for AI data due to their design limitations. These systems weren’t originally intended for machine learning applications, and their inconsistent, incomplete, or biased data can pose significant challenges. Nonetheless, researchers are exploring advanced modeling techniques, such as transformer models, that analyze structured data—including lab results and vital signs—to better address missing or biased information.

Understanding the sources of bias is vital for AI courses. An analysis of existing curricula reveals that many focus primarily on model development techniques, with only a few addressing dataset biases explicitly. To bridge this gap, educators should incorporate questions about data origin, collection methods, demographic representation, and potential sampling biases at the outset.

Effective teaching should emphasize critical thinking about data provenance, understanding who collected the data, the healthcare settings involved, and the societal factors influencing data quality. Participatory efforts like datathons, where multidisciplinary teams analyze local health datasets, exemplify environments fostering critical analysis and awareness of bias. These initiatives illustrate that understanding data context is foundational to producing reliable AI models.

In conclusion, curricula must go beyond technical modeling and include comprehensive education on data integrity and bias mitigation. By cultivating an awareness of data limitations and emphasizing critical evaluation, future healthcare AI practitioners can develop more equitable and effective models, ultimately improving patient outcomes across diverse populations.

Source: https://medicalxpress.com/news/2025-06-qa-students-potential-bias-ai.html