Accountability and Evidence: A Cardiologist on AI, Clinical Reasoning, and the Limits of Medical Knowledge

Sure. My name is Greg Katz. I'm a cardiologist and medical educator at the NYU Grossman School of Medicine. I'm working with AI and medical education across a couple of different domains. We're using it as part of our tools for teaching residents clinical reasoning, and we're thinking about AI as it relates to the evaluation and training of residents. We're also wrestling with how AI may lead to a different skill set being required of a physician, and with the concern that AI could lead to never-skilling of doctors, because we can use it as a crutch for a lot of our clinical reasoning questions. It's a time of a lot of transition in medical education. It's pretty wild to see the current generation of doctors use OpenEvidence the way the old generation used UpToDate. The idea that you can get a bespoke answer to any question you have, which is what a large language model promises, is changing everything about how we think about medical education and patient care. The pace of change in how doctors think is incredibly rapid, and the way we practice medicine is not changing at the same pace.

I think both. New movers in the marketplace of healthcare are building AI-native interfaces and AI-first models. But healthcare in the real world is delivered by big organizations, and there's a lot of regulatory complexity, a lot of bureaucracy, and a lot of billing, liability and legal considerations. There are so many layers of complexity. So right now AI is being deployed much more as an add-on to our current healthcare system. I've spoken with a handful of startups that are trying to build AI-native interfaces to rethink how we practice medicine.

Machine learning has been part of research and of the analytics of how tests are evaluated for a really long time. Even something as simple as a calcium score involves a machine learning tool calculating the amount of calcium. Any doctor who has ever ordered a Zio patch gets an automatically machine-interpreted report, pre-populated before a human ever looks at it, and the AFib detection on these non-invasive rhythm monitors is artificial intelligence. What's changing is that large language models have made it accessible to non-technical people and people who aren't doing research. The speed at which you can get answers and the level of compute you can access are wildly different. It's also changing what patients do and what their expectations of the medical system are. I often have patients bring a printout of what they discussed with ChatGPT. How people interface with the medical system, and how doctors think about medical decision-making, is really different with the easy accessibility of large language models.

It's interesting, I've done work on this. When you give an AI a case and you give a human a case, the inputs you put in, and what the human or the computer focuses on as most important, are going to be different. They're preference-sensitive. If you ask me what Claude Fable is like as a clinical reasoning model, it's pretty damn good, and as good as most doctors in most situations. It's wrong sometimes, but doctors are wrong sometimes. It interprets the evidence overly literally, but doctors do that too. The question about whether AI will eventually replace doctors is one of the least interesting to me. The answer is maybe. I'm much more interested in how AI is being used and deployed now. If you look at all the studies comparing large language models to doctors on patient cases, the ability to work through a whole case hasn't been rigorously shown to be as good as a human's. For individual questions, in terms of information gathering, prioritizing and sorting things out, it's quite good, but there are certainly gaps. I imagine those gaps will close, but AI is improving so fast that predicting what these tools will be like in six months, a year or two years is a fool's errand. I'm not technical enough to know where on the curve of exponential growth we are.

Why isn't it everyone's fault, everyone who contributed? If the doctor trusted the large language model, the doctor is at fault. If the large language model is wrong, then it shouldn't be cleared for use in a clinical setting like that. And whether you get the same answer over and over in the same clinical scenario is something I haven't seen a lot of data on. Whether large language models are consistent in their clinical reasoning hasn't been robustly assessed. The converse question has also been proposed: if a doctor makes a mistake and didn't use AI, is it the doctor's fault for not asking an AI to look through the case? Liability, ultimately, comes down to who is getting paid for the clinical decision-making. If a large language model is licensed by a health system, a hospital or a medical practice and is involved in their decisions, then to me at least it carries some responsibility. How the legal liability shakes out is for legal experts to decide, but in terms of moral culpability, if the people who built a model are selling it as an adjunct to clinical care and it's wrong, there is a moral responsibility even if there isn't a legal one.

You shouldn't trust anything 100%. You shouldn't trust a person 100%. These models are really impressive technology and sometimes they're wrong. Doctors who are experts in their field are really impressive, and sometimes they're wrong. One of the concepts in medicine with the most longevity is the second opinion. In the vast majority of situations in medicine there is no absolute right or wrong answer. There are judgment calls based on an imperfect application of the evidence we have. If you train an AI tool on every paper that's ever been written, you end up with a tool that has a lot of low-quality data in it. They tell you at the start of med school that half of what you learn will be wrong in ten years. The problem is we don't know which half, because medical evidence is very limited and our understanding of the human body is necessarily limited. Even a model that follows guidelines perfectly is following imperfect documents, negotiated by a group of subject matter experts. If you go through any guideline and look at the class one recommendations and then at their level of evidence, the gap is pretty appalling, with so many class one recommendations resting on level B non-randomized or level C evidence. So even an AI that practices medicine entirely by the book will still produce decisions that are wrong, even when they're consistent with conventional wisdom and the standard of care.

It's crazy. AI is in everything. At a conference, everybody has AI-generated slides, and researchers' output is exploding because they're using large language models to write their papers. The editor-in-chief of the Journal of the American College of Cardiology put out a statement recently saying, essentially, that they aren't checking whether AI was involved in the writing process and don't care whether it touched the prose or the graphs, but that the human who submits it is responsible for the output. I suspect that's the direction we'll end up going: you're allowed to use AI for your research, and ultimately you're responsible for every word of what you submit. To me the biggest problem isn't whether AI can answer clinical questions or do statistics correctly. It's that biology is constrained by the human body and by doing experiments. If you use AI to do observational research that just describes the world, that doesn't prove causality, and observational research is almost always hopelessly confounded no matter how well adjusted the models are. AI doesn't speed up the real bottleneck in making discoveries, because at the end of the day you need to test something in humans and follow people for as long as it takes to see a change in events. It speeds up the non-rate-limiting steps of the process, and it lets us generate more unimportant findings. At some point we may develop true models that mimic human biology and let us speed up drug testing and safety evaluation, but right now the bottleneck across biomedical research is how long and how expensive it is to test things in humans. It's not generating hypotheses or targets for drug development. So AI is accelerating low-quality research and the generation of mediocre knowledge, but so far it's not expanding what we understand about human biology. I'm hopeful it will, but we're not there.

That's a hard question. Is using ChatGPT and discussing clinical cases with it enough to make you AI-literate? I'd argue probably not. There's also the question of whether AI is actually expanding what you're doing, or just making your work feel a little more like a game because you're going back and forth and testing it. "AI-literate" is a confusing title and I'm not fully sure what it means. But with the trainees I'm responsible for, I do know that if you're not aware of what these models can do, you're going to fall behind your peers who are. The line between "I understand what this tool is useful for and I'm using it to augment what I'm doing" and "I'm outsourcing all of my thinking to this tool" is really challenging to define, and it's probably moving as we speak. I don't have a pat answer, just a bunch of ideas. In the same way that doctors owe it to our patients to constantly learn what is changing in our field, I think we owe it to them to understand the chatbots they're using and what this technology can do, at least on some quasi-medical level.

That's one of the existential questions in medical education today. We've done workshops with residents about the limitations of these tools. We have a professional development session on how they engineer their prompts and how they think about what they find. We've had them work through difficult cases, like the bleeding versus clotting decision in a patient with AFib who has both a high stroke risk and a high bleeding risk, using their large language model of choice. We also run a session on using AI as a tool for your own continued education. Right now, the way I advise residents to use large language models is for important clinical questions where you're trying to find the landmark randomized trial. A model is really helpful for identifying the important trials, and it can even find the links, so you can read the papers yourself. You can ask ChatGPT, Claude or OpenEvidence what thought leaders are saying about the flaws of left atrial appendage closure for mitigating stroke risk in AFib, or about the strengths and limitations of a study. You can use these as tools to deepen your understanding. But the line between something that does your thinking for you and something that enhances your thinking is fuzzy.

Honestly, I'm so cynical about observational data that it's hard to emphasize enough how pointless I feel the vast majority of that research is. I don't say that to impugn the people doing the work, because a lot of it is valuable for helping us generate clinical questions we can then answer. The real challenge is that whenever observational work is discussed, there's an implied causality, and even if you caveat the limitations, when something is published people talk about it as though it lets us understand causality. I feel truly cynical about how much evidence describes the world without answering clinical questions in any meaningful way. And I'm very cynical about causal inference as a field, and about AI's ability to infer causality. Something like smoking and lung cancer, where the hazard ratio is around 20, you can infer causality from. But with a hazard ratio of 1.2 or 1.4, there are real limits. No matter how sophisticated your model, your adjustments or your machine learning algorithm, I think there's a fundamental epistemic limitation to whether you can ever infer causality. I'm probably a little too cynical about what that means for how people interpret research.

AI is 100% going to entrench clinical practice where the evidence has fully expired. AI is trained on medical evidence, and it doesn't have a perfect model of human physiology. Just as doctors don't know which of our current practices will turn out to be wrong, it isn't possible to know. We can have ideas and hypotheses, but until something is rigorously studied in an experiment where people are randomized to different strategies or treatments, you don't know what you don't know. I'm very bearish on the idea that any kind of modeling will solve that problem. The entrenchment of mediocre medical practice by computers is a very real thing.

What I say to patients all the time is that the AI is right when it's right and wrong when it's wrong, and it can be really tough to figure out which. Reading an EKG has several layers. There's the systematic approach of rate, rhythm, axis and intervals. There's answering the clinical question: in a patient with chest pain, does this EKG mean ischemia, or a PE? And then there's the layer where AI is increasingly being deployed, where the EKG is a prediction tool, suggesting that the patient has structural heart disease now or is at risk for developing AFib in the future. For a standard read devoid of clinical context, AI is pretty good most of the time. It can probably pick up channelopathies like long QT syndrome and Brugada, and it does a good job discriminating arrhythmias. For answering a clinical question, it depends on what supporting or refuting information you give the model. The history you get from a patient is really informative for how you interpret the EKG, and sometimes it tells you something completely different from the EKG in a vacuum. So AI's ability to interpret an EKG in context depends on the context it receives, whether the model acquires it or a human inputs it. For detecting current structural heart disease, such as a low ejection fraction, I think AI is probably pretty good. But even an abnormal EKG will usually get an echo from a human who isn't an expert at reading EKGs, so you end up in a very similar place, and I'm not sure it makes much clinical difference. The last layer, predicting what will happen in the future, is where we have to be careful. If we don't study it rigorously, we're going to overmedicalize a huge number of people. If a tool says someone is at risk for AFib, what does that mean? Do they wear a Zio patch for two weeks every six months for the rest of their life? Get a loop recorder? And if monitoring finds a dozen short runs of an atrial tachyarrhythmia in an asymptomatic person, do you anticoagulate them? Does monitoring actually lower strokes, or does it just mean prescribing a lot more Eliquis to people who won't benefit? There are true uncertainties about what to do with risk predictions, and it's really important that we study these strategies rigorously and prospectively, and don't just assume that "at risk" means we know what to do. Otherwise we create a lot of anxious people worried about risk predicted by a black box we don't fully understand.

People are going to do whatever they want regardless of what we tell them. I'm pessimistic about the idea that our job is to tell people what to do or which information sources to trust. If we're dismissive, in a way that feels condescending, about the places patients get information, we're going to lose a lot of their trust. So what I do is treat the person in front of me with respect, listen to what they said, and address their concerns. I don't think there's another strategy, because the information is out there and people will seek it in the ways they want to. My job isn't to be the thought police for what my patients investigate. It's to help people navigate a sea of confusing information.

My experience is that AI scribes saved me no time. I dictate my notes, and I don't copy and paste data, so they take me two seconds to write. I watch residents write notes a lot, and a lot of the time spent is on copying and pasting data from elsewhere in the chart, which is pretty ineffective. When I tried an AI scribe, it didn't save me time, and it didn't do any of the most annoying parts of charting. The place where Epic causes burnout is in clicking the diagnoses, acknowledging the medications, doing the med rec, and the lag every time I click a tab because the EMR is slow. The scribe solved none of those problems. It also made the notes too long, and I put a bunch of slop in there that I didn't want. I write a note to document what I need for billing and to remember what I discussed with the patient, so the people it's most helpful for are those who spend a lot of time on notes. I don't begrudge anyone who uses it. I also don't love the idea of giving all that data to a company that may be training my replacement, because they're listening to me talk, and I don't know what they're doing with it. But if it saved me time or made my life less annoying in Epic, I'd 100% be using it. I suspect what's driving hospital adoption is a revenue play: if you can shave a little time off every encounter across a health system, you'll get doctors to see more patients. I wish the people working on these tools would solve actual important problems. Why can't an AI give me a clear, comprehensive med rec without clicking through a million screens? Why, when I want to prescribe Zepbound, does Epic make me enter it in milliliters, and make me click two different things to change the dose? Why can't I just say, "Alexa, increase the Zepbound to 5 milligrams"? To me, the people working on this are solving some of the least interesting problems in healthcare, and I don't understand why they aren't trying to make decision-making better.

Being a doctor is the coolest job in the world, and the trust that people place in us is really profound. When you talk to anybody who reflects on what they love about being a doctor, almost nobody tells you about how cool the science is, or what the decision-making was like, or the secret zebra nobody else thought of, or the time they diagnosed Takayasu arteritis. Nobody cares about any of that. People talk about the human element: how we help people through the most stressful and challenging times in their lives, and how, by providing information, support and emotional attunement, we make the job of being sick, or having a sick family member, a little bit easier. So hopefully people who use AI as part of their job as a doctor are using it to be more present for the social and emotional part of the work. We are still in a human business, and the role of a doctor, at least right now, is to help people who are going through the most stressful times in their own lives or their families' lives. AI is hopefully a tool that will make us more effective at that. So little of medicine is about figuring out what the right answer is, and so much of it is about helping somebody through a challenging time in their life.