First, Do No Hype...

Earlier this week, I tuned in to the fireside chat between Mark Zuckerberg and Jensen Huang who were musing about how AI will solve all of our woes, including healthcare's most complex problems. I'm sure you will be thrilled to know that "every business is going to have an AI," including your local healthcare provider. With that, I'm positive our data will be in safe hands, especially with Zucks who will live up to his reputation in handling our information with the utmost care and respect. Because who better to handle our data than Silicon Valley's finest. Watch the fireside chat here.

On a (slightly) serious tone, the AI hype train keeps chugging along, with tech giants racing to stake their claim in the projected $188 billion healthcare AI market. NVIDIA, fresh off turning into the year's must-have tech toy, now has its sights set on healthcare and medicine. CEO Jensen Huang boldly declared that "healthcare is probably the most impactful utility of generative AI that there will be." Not to be outdone, Amazon, Google, Intel, and Microsoft are all touting AI for everything from drug discovery to medical imaging to cloud healthcare offerings.

Jensen & Zucks mused about their ambitions for an open source AI future. Credit: Youtube

This week, shares of NVIDIA, Arm, AMD and others rebounded as investors grew confident that Big Tech's massive AI infrastructure spending spree will continue. Microsoft's revelation that it spent $19 billion on largely cloud and AI-related capex last quarter, with projections of $84 billion for 2025, helped ease fears about the sustainability of the AI stock rally. Meta's Mark Zuckerberg boasted that the social media giant has installed around 600,000 of NVIDIA's latest AI chips. The AI arms race is in full force.

But how much of this "revolution" will affect healthcare?

Doctors and medical experts are greeting the AI hype parade with a mix of intrigue and healthy skepticism. As Stanford cardiologist Dr. Sneha Jain put it, "People want to err on the side of caution because... it's, 'First, do no harm.'" It's the ultimate reality check for the "move fast and break things" ethos that has fueled much of the tech industry's growth.

So, can we trust that AI to "First, do no harm"?

A sobering new study casts serious doubt. The "ProbMed" research, literally titled "Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQA", puts contemporary AI models like GPT-4 through simple medical diagnosis tests and found that even the best models performed worse than random guessing on many queries. Shockingly, they were easily fooled into giving false diagnoses when fed inaccurate information. Read about the study here.

LMMs perform poorly on specialized medical diagnosis questions, especially with adversarial pairs, underscoring their unreliability in handling medical diagnoses Source: Yan et al., 2024.

Of course, I am not generalizing conclusions based on a single study. However, I did try to challenge many of these commercial models with deliberately inaccurate information, AKA known as "adversarial testing" or "adversarial evaluation". The result is that they would apologize and thank me for "bringing it to their attention". This underscores the need for far more robust testing of medical AI models and a paradigm shift in how we should deploy new models, especially LLMS. Nuanced clinical judgment, ethical tradeoffs, and messy human factors don't easily translate into neural networks and algorithms. Issues around data bias, privacy, liability, workflow integration, provider trust, and a lot of politics abound.

None of this is to say AI can't eventually transform aspects of medicine. But it won't happen overnight via magic tech pixie dust. It requires painstaking clinical validation, regulatory scrutiny, and winning hearts and minds. Evolution, not revolution.

So, what can we do?

A good start would be for tech giants to turn down the hype and focus on the hard, unsexy work of engaging doctors, patients, and regulators as true partners. Bake in transparency, accountability, and rigorous testing from the outset. Engineer for privacy, security, and fairness. Anticipate and mitigate unintended consequences. Think beyond the flashy demo to the real-world implementation. Primum non nocere.

Policy frameworks and public-private partnerships, like a proposed national network of AI assurance labs, could help establish much-needed guidelines and guardrails. Empowering clinicians with more visibility and control over AI tools, as some startups are attempting, may boost adoption. Aiming initial use cases at efficiency gains vs. high-stakes clinical decisions could build trust and confidence. Remembering that AI is a tool to complement human care, not replace it, provides needed perspective. Lower-tech, higher-touch interventions are often more impactful. We can't code our way out of fundamentally broken markets and policies, as alluring as the AI sci-fi may be.

Can we trust AI in healthcare? Credit I, Robot, 2004.

Still, the siren song of AI-powered healthcare is seductive, for patients and investors alike. Big Tech will keep funneling billions into the dream of building superhuman robo-docs, hoping some of it sticks. With Microsoft's numbers and the tantalizing prospect of more chip sales to feed data-hungry algorithms, semiconductor stocks will likely continue their rise. NVIDIA's $329 billion one-day gain this week is a microcosm of the AI rush.

But as studies are starting to show, healthcare is the ultimate stress test for technoutopian fantasies. Failing an FDA trial carries far more consequences than a buggy app release. Now we have a large caution sign courtesy of those medical diagnosis models that couldn't diagnose. It's a wake-up call for AI developers and investors to pause and reflect.

By all means, let's harness the potential of AI to augment and improve healthcare over time. But let's do so with eyes wide open, keeping both feet planted on the ground, even as our market caps soar to the gusts of hype. Primum non nocere. First, do no harm.

But also, please, do no hype!