Inside the Brain of AI: An AI Engineer on How Medical Models Learn, Fail, and Earn Our Trust

Sure. Thanks for having me on the podcast. I'm Thomas Proudhon. I'm an AI engineer developing medical devices. I currently work in cardiology, so I build AI ECG biomarkers. More generally, I'm interested in how AI can properly integrate into the clinic, to go beyond the hype and actually improve patients' lives.

I think we should start from the beginning and define what artificial intelligence is. It's actually a term I don't really like, because if you use ChatGPT to discuss a clinical case, or you analyze a brain MRI to segment a tumor, you're using very different technology, yet both are called AI. Artificial intelligence is the broad term for systems that aim to achieve tasks typically associated with human intelligence, such as interpreting an image or generating language. There are multiple ways to achieve it. Most of the systems we use today are based on machine learning, which means letting the system learn patterns from examples rather than programming every rule explicitly. Generative AI, which is very visible at the moment, only describes what the model does: creating an image, for instance, or generating text like ChatGPT. So the key to understanding AI is learning patterns from examples. When you build an AI system, you train it by showing it a great deal of data, for instance millions of medical images or clinical reports, and the system extracts meaningful patterns. Depending on the data you show the model and the techniques you use to extract patterns, you get different AI models.

That's a core part of building an AI system, ensuring it has the right performance, especially for medical applications, and it's hard to summarize quickly. You start with basic metrics. Say you have a model that tells you whether a mammogram is likely to show cancer. The metrics used for diagnostic tests without AI are relevant here: sensitivity and specificity. You want to know what false positive rate and what false negative rate you will have, because every wrong decision has an impact. If you miss a positive, what is the cost? A wrong positive prediction can also have downsides, such as unnecessary monitoring. But for clinical AI, that's not sufficient. You also need to know whether the performance is transportable, meaning it generalizes to other populations, because you train on a sample of the population, and it's not a given that the model will perform on every patient you see in practice. The data used to build a model is usually split into sets. The training set is what the model learns from. The test set is data the model never sees, used only to assess performance. But that's not enough either, because the test set might come from a single clinical center. You want to know whether the model performs well in every center where you'll deploy it, so you collect data at another site and check that performance doesn't drift too much. Validation is really the key for medical applications, and it's a broad topic.

It could. It's not a guarantee, and it depends on the task, but demographics are really important. You want a proper subgroup analysis. Even if sensitivity and specificity are good across the whole population, you still want to know whether the model performs equally well in men and women, in sicker patients compared with healthier ones, and in other subgroups you can think of. The greatest example is a dermatology model that took pictures of the skin. It was developed mostly on white skin, and when it was used on darker skin, it performed much worse. That makes sense, because the model learns from the training population. If you train mostly on men, it isn't guaranteed to work equally well on women.

One hundred percent is never reachable in medicine. Even the best doctor in the world can fail, and, as you know better than I do, diagnosis is sometimes not black or white. There are several grades, and it's not an easy topic. Achieving 100% accuracy won't be possible, and we have to acknowledge that. What we should focus on is a proper understanding of the model's performance on a given population. You might not even want a model that works on the whole world population. Take mammography, which is maybe one of the most advanced clinical tasks in AI at the moment. You want to target the right population. The goal is to define the intended population of the model and describe it very well, and you don't want to use the model on patients who are outside of it.

What we observe now is AI enthusiasts versus anti-AI voices, which I don't think is a constructive approach. As with any science or technological invention, we should be critical, especially in medicine. We're in the era of evidence-based medicine. If you take a drug, it has to go through a randomized controlled trial, a prospective trial, and you wait for proof before you use it. It should be the same with AI. You shouldn't adopt it blindly because there's hype at the moment, and you also shouldn't reject it for no reason. What you mention is true: today there is a general lack of evidence for medical AI. A study published last August found that among approximately 1,400 AI-based medical devices cleared by the FDA, the regulatory agency that evaluates whether a medical device is safe enough, only three, about 0.2%, had been tested on patient-centered outcomes such as mortality, morbidity or readmission. Another recent study, focused on cardiology, showed that only 3% of cardiology AI papers reported a randomized controlled trial. When you know that randomized trials are the gold standard for defining new clinical guidelines, that shows we should be careful about how we trust AI.

The first, as we stressed, is validation. Many papers show that something is technically feasible, but the validation is often skipped. A good example is a sepsis model developed by Epic a few years ago that was deployed without proper validation. In clinical practice, its real-world performance was very low compared with the claims that had been made, and it had to be removed. The second is blind trust. When a model performs very well, it's easy to rely on its output. If you're on an exhausting shift, you won't have time to look closely at the output and do a double review. So there's a real risk of automation bias, which is accepting the output without critical thinking, and it's been documented in several studies. One study showed that some physicians who relied too much on the AI output had lower diagnostic accuracy. There's also a risk among students and young doctors: a systematic review showed that medical students using AI showed reduced development of skills. So AI is probably beneficial in medicine when it's well used, but we have to be careful about how we develop it.

Studies keep showing that it's improving, even in other areas. You probably saw that recently, super hard mathematical problems were solved by a group of AI agents. If I said today that AI will never reach human clinical grade, I would probably be completely wrong if you watched this video in a year or two. The rapid progress shows there's really something there. But maybe what we should focus on is not performance. We have to see how it integrates into clinical care, because medicine isn't only about performance. There's a human relationship between the patient and the doctor, and that's something we might want to keep. So we should focus on what can be automated for the better, rather than going for AI just because it performs well and it's hyped.

That's a good example. Whether it's true empathy is a hard question to answer, because it's partly philosophical: what is intelligence, and what is empathy? What's useful is to remember how a large language model is developed, because understanding how it's built helps answer part of the question. Behind ChatGPT, the core model predicts only the next word of a sentence. It's trained mostly on text, essentially the whole internet, so the model is a kind of compressed version of web content, and it tries to approximate and retrieve what it learned. The more commonly a topic is discussed online, the more likely the model is to be right. If you take the sentence "Paul is my brother, and Sarah is my...", the most likely next word is sister, or cousin, or mother, depending on the context. At that stage, the model has no particular tone or intent. It blankly predicts the next word. The second training phase aligns the model with human values and expectations, because when you interact with ChatGPT, you want to feel you're having a real conversation, not a dry one. This is done with a method called RLHF, reinforcement learning from human feedback. You run the model several times on the same prompt, humans rate each response and select the one they think is most relevant in a human conversation, and over time the model learns human preferences and gives answers humans appreciate. So in your example, when a patient says they were diagnosed with cancer, there are many possible continuations. It could be a very technical medical explanation with many technical terms. But the model is built to mimic a human conversation, and what most people say when someone tells them about a cancer diagnosis is that they're sorry to hear it. The empathy you see is largely the result of RLHF. It gives the sense that you're talking to a human, but it isn't particularly tailored for a clinical application, and that might become tricky.

It could be. We've heard some AI labs say they think these models are very dangerous now. I don't really know what's happening inside those labs, and they might also be generating some hype to sell the model. But coming back to empathy, I'd say that you are essentially training the model to answer like a human, and the danger is mostly in forgetting that this is what it is.

With ChatGPT, it was the first time the public really saw and used AI. Before that, it was more obscure, and most of it happened in research. ChatGPT was the first mass product with a clear use of AI, where you can really sense the intelligence. Before generative AI, AI was already used outside medicine, in ad recommendation, social media algorithms and customer profiling. In medicine, adoption was slower because, as we said, for a product to be used in healthcare it usually has to be cleared by a regulatory agency such as the FDA. But the first AI-based model cleared by the FDA actually dates back to 1995, for cytology slide analysis. Before generative AI, many algorithms focused on radiology, as assisted diagnosis for very narrow tasks: an input image and an output probability, for instance that a brain MRI has a tumor, with no interaction with the model. Another example is diabetic retinopathy in ophthalmology, used as early as 2018. Even now, most AI products used in healthcare are not generative AI, because it's really hard to get them cleared and to show they're safe to use.

That goes outside of medical AI, because what you're describing is AI-assisted coding, which is one of the most used applications of AI at the moment. Companies switching to agentic coding is revolutionizing the whole industry, and it gives people without prior knowledge the power to build things that weren't achievable before. But medicine is super regulated, so if you build an application and don't understand what's happening behind it, it can be risky. There are entire teams focused on building a product. You have to build the core model, but you also have to ensure many other things, like cybersecurity, and follow a number of norms and standards. What's interesting is that you can get to prototyping very fast, and a prototype can be built by one person. If your plan is to build a real product, a startup for example, you can go very fast with AI-assisted coding to develop a prototype, then maybe raise money to build a team and create the actual product, which goes through the complex validation we discussed. It's an era where you can try things, but remember that medical AI is a really regulated environment, and it's not easy even for big companies to create an AI model.

When a physician uses an existing product, you have to focus on how the model was validated, and on which population. Is it the same demographic as the one in the reported claims? You also have to understand a bit of what happens behind the scenes, because you can easily fall into blind trust and automation bias, and if you rely too much on the output, it can erode your critical thinking. I view AI like any other tool you have in medicine. It should serve one purpose, which, depending on the physician's specialty, is to treat the patient and improve their life. That doesn't mean AI should replace physicians. It's a tool that needs to be learned. So focus on how it was validated, how you use it, and beware of blind trust. That would be my summary.