I used Claude Code to get a second opinion on my MRI

In my day job we tried creating a credit assessor tool using LLM as the credit assessor. It did great, generated a report on the assessed business that was incredibly detailed and plausible. Then I started running tests and getting into the details, and found that if you ran the same report on the same data, it generated completely different, still very plausible, results. I could run the same source data through the assessment process 10 times and get 10 very different results. We had to can the project and go a different route. LLMs are designed to produce plausible results, not factual results. We can fix this when using them for software dev by using linters and tests (though we've all had the experience where the LLM invents an API endpoint). I would not trust raw LLM output in any situation where that kind of testing and verification capability isn't present.

What happened to VERIFYING an answer? Does nobody do that anymore? When I ask an LLM, I trace the sources, and see if they make sense. More often than not the sources don't actually say anything about the topic in particular... > It’s also scary how easily you can lead each LLM to the answer you have in mind. Exactly. Which is why "treat an LLM like a human expert who can answer your question" doesn't work. It's more like a human bullshitter who makes up convincing looking answers, and tries to please you. If the answers have actually some grounding in the training material, that's useful as some kind of holistic google, but often it's not.

My step mom was having debilitating pain. A year of going to doctors and no one was able to find a cause. I scanned her discharge paper work which had her prescriptions on it and gave it to Claude. It identified a prescription that had that exact side effect. They later confronted her primary care that concurred and took her off it. A friend of mine's wife recently passed. They were chasing a suspected heart defect for over a year. She had been intermittently fainting. At about the year mark they decided to scope her digestive track. They found bleeding ulcers from cancer that was all over her body. I input her fainting symptoms into Claude and gastro impact was number two suspected after heart issues. I have a few of other cases it's helped with. I'm not sure it could do worse than my own experience with the medical system. This is doubly true in places that lack any sort of medical care.

> It’s also scary how easily you can lead each LLM to the answer you have in mind. Scary in this context of course, but I find that it is an interesting thought for coding: it suggests that maybe, a developer who knows what they are doing will end up leading the LLM to coding something that make more sense than a developer who doesn't know and just vibe-codes blindly. Sounds pretty obvious, but I wanted to say it.

As someone who uses Claude Code to summarize published research, you have to ground it in peer-reviewed results or it gets lost. But also, I am grounded with two degrees in the source material. So I am feeding it my views and asking if the published work agrees or disagrees with my opinions and I get fantastic results that way to the point of knowing current clinical trials and treatment regimens than most of the oncologists and which led to a great conversation with the clinical trials team. This doesn't replace people, but it augments existing expertise amazingly well. But also, I hear so many tales of running out of tokens. I ask Claude Code to build a tool to perform a task. I review the tool and then I let it rip if I'm happy with it. As I understand things, most just ask Claude Code to do the task. That seems a bit fraught. Anyway, you have to impose constraints IMO and ask the right questions to get the answers you need or yes Claude Code (or any other LLM) will eventually just agree with you.

Yeah a lot of focus lately on making context windows enormous and putting everything in them. (It should know every detail of your life!) But in my experience LLMs are extremely "prime-able" and also tend to hyperfixate on details. So when asking difficult questions I tend to remove as much context as possible, rather than adding it. I don't want it to reflect my own ideas or biases back to me, I want an actually fresh perspective.

Yup. This works until it doesn’t (fairly soon thereafter), both from experimentation and understanding of theory. Here’s an illustrative example: https://old.reddit.com/r/Bard/comments/1l1qxk9/why_does_gemi...

LLMs are well suited to my (some would say annoyingly) curious nature. when i get an answer, and my first instinct is to ask a ton of follow-ups and "what about"s. i've learned to tamp this down with fellow humans, but with LLMs its great because most of the time the response is "you're right, something doesn't add up... let me try again". i think we eventually converge on to something reasonably true

[deleted]

Have you ever let the LLMs “discuss” with each other to see if that would give better answers? You might end up with the answer from the most persuasive LLM, but you might also end up with better results. Wonder if there is a paper out there on this.

Yes, and that's a problem. Doctors (or experts in general) hate it when people don't trust them, but the thing with experts is that people have to trust them. And in my life (and a few times just in the last few years), enough doctors have been wrong enough that I cannot just trust them anymore [1]. If it is important, I will ask them to explain to me, and sometimes I will just ask for a second opinion. I have read about doctors complaining that "with AI, patients now come with their own diagnosis and don't trust us when we say it's bullshit, and it is a problem". I can feel for them, but if they give the feeling that they don't listen to the patients and the patients don't trust them, it's not only the patients' fault, I would say. [1]: I have more than one examples of my relatives like this: A doctor says "wow that's bad go to the ER", the ER says "nope it's all good, go home", first doctor learns about that and says "WTF you GO TO THE ER, call me and I will insult them on the phone", and finally resulting in a surgery where the doctors say "they were lucky we could operate right now, because in a matter of hours they could have died from this". How in the world can I trust them after one event like this? Happened to me (in some variation) 3 times. Not based on an LLM diagnosis in the first place: based on a doctor's diagnosis.

Maybe a difference here is asking AI for conclusions. When I have it do a buyer's report for me, I ask it for "what questions should I be asking? What are typical things that go wrong with this type of vehicle?" I don't delegate conclusions to the AI but use it to educate myself. Then, I can gather further information to make MY decision .. to buy it or not.

Exactly. Old parts of the system will be working if you leave them undisturbed. Mechanics have very good intuitions of this sort of thing. I read about before there's proper engineering / physics theory about this too, it's like a car as a machine is a linear/smooth physics system with multiple weaknesses. Overtime longtime period of running many places might weaken but it still evolves into a slightly different smooth system, until you introduce a replacement which cause a mis-match of impedance or something like that.

Maintenance-induced failures are what it’s called with small aircraft. You’ll do something to prevent a failure (like, replace an old but functional alternator) but cause an oil leak or engine vibrations because you had to remove the propeller to complete the job.

Maybe I am missing something but I just find this wrong. Everything is a puzzle: there is one "Truth" or one diagnosis. You (a smart human) should be able to converge on it by cross-examining your LLMs. By themselves, they have no interest in revealing this, no stakes, which makes them tools only useful at the hands of a capable investigator.

Isn't that the case with human "experts"? If you had encounters with doctors, mechanics, etc. you'll know you can get a completely different diagnosis for the same problem which obviously means (in most cases) that the person you thought an expert is wrong. What is needed are studies that will take a cold look at the actual results because AI seems to be required to be perfect or it is useless. It just needs to be as good as a human for most stuff, but in the long run it will be much better. At least that what extrapolating current reality shows us.

You’ve really honed in on the key issue. This is exactly how keen hackers news commenters approach this.

YouTube has saved me at least that much in appliance repairs... and it doesn't even have an AI. It's amazing how valuable access to information can be.

[deleted]

I would love to hear more about this

Saved me $2000 on a koi pond pump and filtration system

Tbh LLMs pulling data out of medical documents in it's training set and searchable online is likely a much easier task than fixing some weird CSS alignment issue.

Sounds like a fun watch! What is the name of the channel?

There are few things better in this world than having a car shop you can trust. I found one and pray that management doesn't change.

Yeah, I know! It was strange. They gave me a test, and it came back negative, but they insisted it was negative because I had "latent tuberculosis," which supposedly wasn't detectable by the test yet but was about to become active. I forgot to mention that, besides getting a second opinion from another radiologist, I also took a more modern test at another private clinic. That test has better detection rates than the one the state clinic used, and it came back negative too. I have suspicions they had some kind of government quota to keep the hospital staffed with patients in order to receive funding. Or they were just completely incompetent. I pushed back by bringing them another radiologist's report and the results of a better test that I paid for myself, so I guess they decided to back down.

Well you can't diagnose pyelonephritis without a urine culture as well, which my GP kindly noted after I already took a full 14 day dosis of antibiotics. The ER I was at before tried to, anyway.

Not only that, what is the point confining someone to prevent the spread of a disease about a quarter of the world is already infected with? I suppose there could be reasons, but I don't know them.

Incentives.

Yea I find a lot of stories on the web about doctors misdiagnosing things to have oddities like this that don't seem to make sense. It often seems like the author is leaving something out. Not saying OP is lying, but tb is a very, very weird conclusion to come to from just one radiology report...

I once worked on a medical hackathon concept for computer-assisted population screening for cervical cancer in a developing nation. Community health workers take photos. The AI would look at the images, and make a call of "clearly negative" vs "clearly positive" vs "needs (scarce) expert review". But taking good photos is hard, so it's also "photos insufficient" and "worker needs additional mentorship on taking photos". Only by computes reducing all three costs - expert workload, exam success, and quality-control/training - might successful deployment be financially and logistically plausible for that nation.

The healthcare affordability crisis is only going to exacerbate the trend of using AI as a replacement for a real doctor. I went to urgent care a few months ago to get tested for COVID and two other flu strains and it came out to almost $500. Anecdotally, several people in my life who embrace less traditional (and sometimes more conspiratorial) views on modern healthcare tend to be the ones that can't afford it. A confident-sounding chatbot to answer questions day and night about what's going on with your body is very seductive in a world where access to real healthcare is getting further and further out of reach.

> General-purpose LLMs are _fantastic_ at medical diagnosis that do not involve imaging. Can you share the reasons that you believe this? > At the same time, they're worse than useless at anything involving medical imaging. What is special about medical imaging that makes AI/LLMs specifically bad?

As a programmer, I don’t feel threatened by the technology itself, but I do feel threatened by the second-degree effects such as what the technology does to our field, especially in the wrong hands.

It's funny (and a little depressing), because HN routinely assumes that their world view, and thus, their domain expertise, transfers. There's no shortage of tech people convinced they deeply understand law, medicine, philosophy, etc. despite never having read much on the topics.

Yeah I think the OP is muddling the point by conflating "physician's version of the diagnosis" with "The Diagnosis". There is absolutely one "The Diagnosis". Human body is a machine, albeit a very complex one, and all measurement sources have noise. But they are all measuring one reality, and if there is a problem, there should be one explanation that all measurements align with. They can be noisy but can never be conflicting (instrument error notwithstanding). Physicians' ability to arrive at "The Diagnosis" would vary, but it does not mean one does not exist. I am not sure if characterizing human body as derministic or not is relevant here.

> Anyone who doesn't understand this is an idiot imo I don’t think that’s true. Avoiding this mistake requires knowing that an ultrasound may not detect calcification. For a patient reading their own report, I don’t think that’s intuitive. I would expect most people to read “no calcifications” and assume that their joint has no calcifications.

> You would have to assume that every type of diagnosis instrument has infinite clarity and is always correct to be confused in this case. There's a difference between 99.9% clarity and 50% clarity. Even if neither exactly equals 100%, it's understandable that a layperson would expect different language between them

This comment sounds like it's written by someone who doesn't interact with real people very often

"On two occasions I have been asked [by members of Parliament], 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."

> Anyone who doesn't understand this is an idiot imo I disagree. A priori it's not obvious to a layperson whether or not a statement that uses unconditional phrasing is intended to be authoritative or conditional on something unspecified, like the resolution of the measuring device. This goes for any sufficiently technical field. If you got the brakes checked on your car, and the mechanic did <something> and told you there are no issues with them, and you then took your car to a different mechanic who did <something else> and told you there is a problem, you would not be an idiot for thinking that these conclusions contradict one another.

It’s funny that the answer to this has increasingly become “yes” over the last few decades.

I don’t think people are idiots if they don’t understand how a normally intelligent person might not intuit that. I do think they have a seriously underdeveloped theory of mind.

> Anyone who doesn't understand this is an idiot imo Even if this is true, so what? Idiots get sick at least as often as others, and the medical system needs to work as well as it can for that population too.

[deleted]

Semi-related: my father has complications from a motorcycle accident ~25y ago that crushed arteries in his leg coupled with diabetes (insulin / kept sugar at ~100 and his A1C was kept under 6.7 for ~15y). 6w ago had to have his toes removed due to dry gangrene; they eventually (2.5w ago) had to remove his leg below the knee because of the severe blood flow issues below the knee. Between the toes and the below the knee amputation, there were no less than 15 different doctors and PAs / related personnel who COULD NOT COME TO A CONSENSUS. They would just tell my mother and I (PoA) the details; they refused to come up with a singular plan of action moving forward, leaving it up to us to make 'an informed decision,' something that's IMPOSSIBLE when you have to take up to 15 different opinions into consideration. What exactly are we supposed to do as patients/family members when medical personnel cannot give reasonable paths forward and instead just throw a bunch of shit over the fence at you and tell you, "you decide what to do from here," regardless of how many VERY DIRECT conversations I had w/the 'care team' on doing better to provide a limited array of options and reasons/likelihood of 'positive outcomes'. I'm used to dealing with a wide variety of stakeholders/SMEs in decision-making; it's my job to apply my extensive industry experience to present our clients with their options, ranked and reasoned. Doctors, in my experience and most recently with my father, clearly do NOT do that (I assume due to liability; but, no real idea, honestly). So; when dealing with LIFE CHANGING circumstances, what are we supposed to do except rely on what might be able to offer more analysis and option narrowing w/AI? I certainly don't want to make the job of medical staff more difficult by putting out crazy theories I found on the interwebbernets through my own research, etc; but, when we're having to deal with uncertainty and insanity, what else can we do?

Your see this in coding agents too. The only times so far I’ve really seen Opus tie itself into a knot is where I’ve asked it to fix something that I thought was broken but actually wasn’t in the way I had described. It will bias towards your description (I’m guessing because that’s the most recent context it has?).

i'm sorry, but AIs only "know" about stuff that they have been trained on. If we would allow AIs to be trained on the petabytes of medical data hidden in hospital systems, they would most likely be much better at diagnosing illnesses and conditions than the average doctor. (Justifiable) Privacy around medical records so far prevents this. You think you're cheering for humans, but in fact you are gatekeeping healthcare.

I feel like the promise of these models is to help people make more informed decisions. Improving the knowledge economy and general understanding. The problem is these are just statistical models at the end of the day, so you need to know something to be able to identify the errors. You can’t let them really be autonomous and you also can’t really have people turn into glorified approvers. If the machine is correct 89% of the time, you cannot make people responsible for that 11%. It’ll just cause automation fatigue. tl;dr: the actual use cases of these LLM (or generative AI in general) is rather limited, so it is offensive how much hay has been given to them eating the entire capitalist system. They are not fit for purpose.

What is "it"? Claude Opus 4.x? ChatGPT-5.x? GLM? DeepSeek? RadFM? Med-PaLM?

We're discussing calcific tendinitis (https://radiopaedia.org/articles/calcific-tendinitis?lang=us). If you think you have it, you can see a doctor and consider shoulder radiographs to start.

They do. Ultrasound in orthopedics is a relatively newer field, and there aren't quite as many sonography techs and radiologists experienced in reading these studies, which is likely why you don't see it offered more widely. Edit: I should mention that ultrasound is basically unusable for evaluating bones. Sound waves can't penetrate bone, and so you end up just seeing a huge black void. That's a huge orthopedics use case that ultrasound just can't benefit. However, ultrasound is fantastic for evaluating muscles, ligaments, tendons, and other superficial soft tissues.

We order ultrasounds all the time for shoulders (for like soft tissue issues; for trauma, you'd start with an xray). For other joints, such as the knee, MRIs are a better choice (unless htere has been substantial trauma, in which case xray initially or further), though more expensive, unless you're excluding a Baker's cyst, in which case an ultrasound is fine. Since MRIs are more expensive, private doctor's might order them instead of an ultrasounds. (I'm a doctor)

Ultrasound was overlooked by US medicine as a first line imaging tool for a long time because it takes real skill and experience to do it right. But it's making a comeback. We've had Chinese, Indian, Australian, and American doctors visit us for one to two month stints to build up their skills. Given the skill involved, it's probably a liability concern they don't want the exposure over there.

They're used quite a bit for nerve entrapment—both in diagnosing and treating.

It's a manual, non-standardized process without a standardized output. Image quality depends both on user skills (how deeply they press the sensor on the skin) and the machine they have. Unlike CT/MRI the examination results cannot be easily shared and compared between patients for studies.

Ahh, AI is coming for your job. Full sarcasm, is there one that’s that’s more immune?

> Does radiology really make +$700,000.00 a year ? The radiologist I know does not, but they are paid very well (and these numbers are always dumb when you're not sure if they're living in Manhattan vs literally anywhere in Kentucky) Like most medicine, a large % of the job could be done by any decently talented person willing to follow instructions and shadow for a few months. Like most medicine, the remaining % is what you're paying for, because it is literally life and death and you can't do things like "pull the logs" or "lets turn it off and take it apart" or "huh i need to put this down and come back later". Even in radiology, because "well lets just do it again to be sure" is often not a viable option. While there is a problem in how we have inflated the cost of education for medical fields, the insane health insurance issues (US obviously, but it does have some effect globally when the expert radiologist you hire from the US to help with research costs that much), and probably some better ways to approach splitting the work for the entire field, like most professions dealing in life or death, medicine likely will always be paid well.

Physicians salaries account for about 8% of healthcare costs in the US.

The savings go straight into patients' worse outcomes.

You know the radiologist you're responding to is a real person? Your last line seems needlessly callous.

To the consumer! Haha just kidding. We all know where they'll go.

same here. I started doing yoga and rock climbing, and it stretched everything out, and strengthened all the muscles around it. I rarely have an issue now.

Physical therapy is very often under recommended in the US under the belief that insurance won’t cover it. They might. And, for anyone reading, you don’t even need a referral for the first 30 days in some states. Physical therapy is for more than just hip replacements and car accident trauma. Like regular therapy, a lot of “normal” people can benefit from it. It’s also not just stretching.

What did you have exactly? With calcifications, physio without the shockwave component definitely doesn't allow going back to the normal gym routine. It's just not enough.

> the surgeon recommended months of physiotherapy before resorting to the knife In my limited experience, "If all you have is a hammer, everything looks like a nail", rings particularly true with medical professionals.

>before they even have a case with you My problem is that I needed information from 2 ENT visits to feed into ChatGPT to get that study. On the first visit he scoped my sinuses and immediately said "I can see evidence of allergic reaction, see those white bumps?". On the second visit I got an allergy stick test and it came out negative. Those helped lead to that NIH study. It would have been very hard to have walked in with that study in hand.

Not stupid. Because it wasn't on my radar that it was allergy related until the ENT mentioned allergies.

That seems to be only for first generation, drowsy-making, tablets. Second gen formulas don't cross over the blood/brain barrier. https://www.myalzteam.com/resources/zyrtec-and-alzheimers-me... There IS one year-old finding that suddenly stopping Zyrtec after daily 3-month use may lead to nasty itching, and if that happens you can re-start and then taper off. https://www.fda.gov/drugs/drug-safety-communications/fda-req...

Where are getting that from? All I can find is about 1st gen antihistamines (i.e. Benadryl, which I doubt many people take daily, because of the drowsiness). Even for those, evidence seems to be mixed at best. "Huge increases" seems like hyperbole.

Only first gen, 2nd gen does not have this issue anymore or it’s greatly reduced

Misinformation. Only first-generation antihistamines with anticholinergic effects are associated with cognitive decline in elderly patients.

I believe it depends on which ones, the older gen or certain classes of antihistamines

Wait, what?? Now I'm getting in panic mode because I do take regularly anti-hystaminic tablets/pills (the newer ones, based on ebastine because they don't make me feel sleepy)

but those are two different things. Of course something like Deep Resolve is great, as are modern model based reconstruction algorithms for CTs, but here we are talking about LLMs and their ability to interpret medical images, which has nothing to do with what you said.

Sorry? You use AI to hallucinate medical images and that's good?

Sure but claude and ChatGPT are not Siemens 'Deep resolve'.

ChatGPT is completely unplayable at chess on its own. It's unable to keep track of the state of the chess position and therefore will make an illegal move within about 10-12 moves. I would put GPT-5.5's rating at 400, since it can't even make legal moves reliably. I've tried to pay chess with GPT-5.5, even played it again tonight, allowing it to use `python-chess` to keep track of the state of the position and to get a list of legal moves at each turn, so that it was fair. I also gave it blindfold odds, again to make it a fair fight, but it was not even close. GPT still isn't better than maybe 1000 Elo, maybe 1200 tops. Even with what amounts to being able to see the position and also being unable to make an illegal move, GPT-5.5 hangs material left and right, doesn't make a plan, and got smoked even when I gave it blindfold odds, to the point it's boring for me to play even under those conditions. I'm not sure it's better than whatever the GPT model was that was out about 8 months ago. I also thought it might be somewhat better than a beginner due to reading chess books, but no, it's complete garbage at playing chess, not even average-level skill.

Interestingly LLMs are extremely bad at chess position _images_. I have to imagine if you give it positions in text it'd be pretty great but when I was learning chess and pasting images of positions in for analysis I couldn't believe how wrong it was. I actually thought it was looking at the board in reverse but even when pointing out problems it seemed completely incapable of understanding what it was missing (of course... it doesn't really "understand" anything). LLMs truly are marvels with text but anything spatial seems to really mess it up, somehow.

Oh I agree with you, it's just that I don't think LLMs are either. If you think of LLM knowledge, especially in scientific/engineering fields, as a lossy representation of the density of ideas, then you'd expect to see some weird behavior. I'm sure there is some sort of a temporal discounting and people thinking about this, but a naive NLL or Reverse-KL on medical literature would engrain some weird, wrong ideas.

Many DSM-5 diagnosis come into effect with the ICD-11, ICD-10 doesn't have a good deal of them, and that rollout is still fresh & ongoing. It is kinda spooky, though, to have freshly minted doctors from a few years back whose school-knowledge will forever be "outdated and archaic" based on standards published before they were in school. Some good advice I got: treat this as a generation shift, find younger and newer doctors who are familiar with the "modern" standards.

Why would you expect a Polish psychiatrist to understand the differences between different versions of a diagnostic manual used only in the US?

“A man with a watch knows the time; a man with two watches is never sure.” I imagine reasons for what you’re asking might include: * Prompting an LLM is work, and they’re already overworked just doctoring—every conversation with a computer is a conversation you’re not having with a patient; * They’re probably right more often than they’re wrong; * “When you hear hooves, think horses, not zebras”: the 15th case today of strep throat is probably strep throat, regardless of today’s 15th falsely-confident LLM weighing-up; * They tend to have spent many many years honing a clinical intuition that makes an examination, to some degree, hard to articulate fully to the LLM; * Liability/overdiagnosis: All this stuff is probabilistic. Inevitably, there’s going to be a time when the LLM throws out something I thought unlikely that turns out to be right, and there will be other times when it’s wrong but now I have to document why. How many false leads do I need to chase per one true differential? Does this really compare favorably to seeking a second opinion from another human doctor? * Not everything needs to make it into the record. Once it’s in the LLM, it’s discoverable and litigable and hackable and permanent; * Medicine is practiced in very different ways in different contexts—even in this thread, one radiologist routinely orders ultrasounds for soft tissue shoulder problems, and the other medical-world person replying has never heard of such a thing—presumably both within US health care contexts. Some doctors hand out antibiotics like candy, others are more cautious with respect to resistance. What’s right can depend on the time, the place, the clinical setting—more than just the immediate patient-level facts at hand, in ways that become awkward or unwise to express explicitly. And of course… who’s to say they don’t do LLM-assisted research, in cases where they think it might be helpful?

> I don't understand why doctors don't prompt LLMs before saying wrong things. Is it ego? Either that or laziness I'd imagine. This isn't limited to LLMs. Expert digital assistant systems that you query have existed for a long time. A good physician will double check anything even slightly unexpected against one.

6.69 hasn't been released yet, to be fair.

Hey what did you do to fix your sleep? Help us all and maybe an llm will index your diagnosis (hi ChatGPT)

The context is very important: decades of a poorly-diagnosed chronic illness had left him deeply distrustful of the medical system. This is the real root issue. At 75 years old, he was stubborn. Is that reasonable ? Yes, perfectly. Could he have been right since the beginning ? Certainly. Did he deny evidence ? Yes. Zero doubt that he was intelligent, everything points toward that direction, but that doesn't make a person less stubborn, because accepting the evidence, is also accepting that you were wrong if you initially postured yourself as adversarial instead of cooperative. He would have read Wikipedia, scientific papers, etc, even without AI. He did not want to be convinced. It works both ways: https://www.foxnews.com/health/woman-says-chatgpt-saved-her-... or https://www.today.com/health/mom-chatgpt-diagnosis-pain-rcna... Nonetheless, someone very smart, just didn't want to move from his position.

i mean, other smart people have famously delayed cancer treatment without needing poor guidance from LLMs! that's not at all new or unique to LLM chatbots

GPT-4o, which is what that article is most likely about, was an older low param count slop model which was known for abusing emojis and sycophancy. It does not really have any relevance to latest claude frontier models. Your comment is akin to saying "Karen from facebook who is a human pushed essential oils and ivermectin as a cure to cancer. Now doctor Y is suggesting chemo. Both are humans, humans cannot be trusted!"

Very curious what made you run to the emergency first thing in the morning that an LLM understood as "just normal, take some OTC meds and wait".

Explaining diagnosis and treatment recommendations decisions inherently involves explaining away the alternatives. In this world where patients are ultimately responsible for our own care, explaining your rationale is a straightforward part of the job - otherwise there is nothing for patients to base their decisions on apart from how the options make them feel. If visits haven't been allotted enough time to get the job done, then that is something you need to take up with health plan bureaucrats rather than taking it out on patients.

I dunno man, it's one thing to have your car still be broken because you were wrong, it's a different thing poison yourself on the basis of having done your own research. The mechanic can laugh at you, it hits a doctor differently.

[deleted]

you are literally taking sleeping pills ..

Well it was a nightmare for my mother's do-nothing GP surgery in the UK. She had several conditions which were being handled completely separately without central coordination, and her health was in serious decline. We went in with a list of 20 AI-generated questions based on her conditions and treatment (which I was able to screen as I have a bio postgrad, but not medical training), including those related to NICE guidelines and procedure, and, frankly the GP bricked it and ordered a load of new interventions. My mother started to get proper treatment. I wouldn't trust AI to make a diagnosis, but I would absolutely trust it to notice where procedure hasn't been correctly followed, where a treatment is counter-indicated because someone has missed a line on a health record, or where there's a clear potential alternate diagnosis which has been missed for spurious reasons. Also, unfortunately, where doctors aren't doing a decent job - often because they're overworked or underfunded.

There’s more than two options here. It was already difficult to deal with self diagnosis for doctors, now we have a machine that outputs recommendations, and does it with confidence whether it’s correct or not. The same issues that were present with search-engine self diagnosis are still present with LLMs. If you provide Google with an incomplete list of symptoms and can’t interpret the information you find correctly, you will likely get an incorrect diagnosis. The same is true for LLM output.

Nightmare because users approach LLMs with the false confidence that they're always right, and present LLM outputs as fact to Doctors who have to waste time explaining that it's wrong most of the time. It hurts more than it helps.

Its a nightmare because it erodes trust. Doctors are not "always right" which is why "always get a second opinion" is codified in culture. But AI's problem is that its completely full of shit, sometimes, and the people most qualified to evaluate whether its full of shit are the doctors, not the patients, but just like OP's original article, patients are left feeling like their second opinion from AI might be more trustworthy than their doctors opinion.

Nightmare because the AI is just generating a random text that fits the question.

Do you know how many life threatening illnesses I’ve diagnosed myself with by googling symptoms?

Agreed. This gets worse in cultures in which Doctors have no habit or haven't been trained that educating the patient is part of the job. Whenever I am back to my birth country, I specifically avoid doctors that are older than mid 30s, because they all have the same, terrible bed manner. They might be good at diagnosing and treating, but they never, ever explain anything, even when asked. Some even have "helpful pamphlets" to hand to the patient - anything to avoid explaining. It seems that in their view their job is not helping the patient, but completing a task - running a scan, performing a procedure, administering medicine etc. The human, that is subject of the task, is invisible.

I was a very early adopter in my circles with AI and I shared it with many people. Strangely, I seem to be the most skeptical about AI in my circles as well, but because I was the gateway for a many folks, they want to come back and share their experiences with me. And it's so much like listening to someone in a church congregation sharing their experiences with god. Clear and obvious gaps are hand-waved away exactly how you're describing.

>This is the root of AI psychosis. There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks because their fundamental basis is not evidence, it’s belief. Treating it as if it is an intelligence is the problem. The problem is that AI psychosis is fundamentally the belief that an LLM is "thinking" at all. Outputs are just believable word vomit which resembles factual information.

I _believe_ the term "AI Psychosis" is a "thought-terminating cliché" that readily puts you in a position to disarm any criticism to your point of view, which, if you're aware of it or not, it's a belief in and of itself. I'm more willing to bet your can't have discussions because you're trying to have debates. But on your actual point, I don't think AI needs to "surpass current experts abilities in all promised fields" as a marker of its ability. The immediate gains has already shown some remarkable promise and more LLMs should have had safeguards around mental health up front. If I were to put it on a scale, I would say it is net positive long term with a strong negative spike up front which was somewhat preventable. But who knows, maybe is just have "AI Psychosis" and you can easily dismiss me.

I don’t think they will improve, there is too much incentive to poison the datasets going forward. A lot of the models up to this point have been benefitted - like Google did - from essentially ‘pre SEO’ internet. Now the same tools are being used to generate nigh infinite good sounding bullshit, which poisons the dataset in all sorts of hard to detect ways. To add insult to injury, the human experts are also not as. Naive, and have many incentives to poison their own input in subtle ways too.

Human expertise is also improving all the time and not limited to just connecting dots. When AI seems to surpass a particular human, it's just because the human lacks broader knowledge and fails to investigate further. An expert already knows they don't know everything. That was never the point. Critical thinking cannot be delegated to AI any more than it can be delegated to a book. There is nothing new going on here.

[deleted]

> There’s a lot of unpack here, and I won’t go too deep because you can’t really have a discussion with affected folks Do you think it is any more possible to have a proper discussion with someone who preemptively paints the other person as mentally ill? Or someone who preemptively victimizes themselves? Cause I don't think these are the hallmarks of an honest discussion. See also the entire past decade of political discourse. Like, consider this: > It is weirdly religious in a way, because if you were to present contrary evidence (e.g. experts in a field weighing in about how plausible sounding responses are bunk), you would only be told you don’t believe enough in the long term potential and capabilities. A trivial counter to this is that you can just be an expert at something (e.g. your own work), use the damn thing yourself (professionally), and evaluate the outcomes for yourself. Then maybe remark "LLM good". Now you come and remark "LLM bad", and point at random "evidence", either of outright other workloads, or even the one at hand: you're asking someone to reject the reality they've already experienced, entirely based on the assumption that they're "merely religious" or "in psychosis". You tell me if that's any more epistemically rigorous and sensible than their story.

Why is it psychosis and not lower standards? While I can understand being skeptical of non-experts' claims that such answers are enough, I don't understand why you call it "psychosis" and not simply naivety or lack of expertise. At the same time, the new so-called "models" haven't been pure transformer-based LLMs, but entire systems with tools (with access to the Internet), data storage, and the options to trigger additional instances for different tasks.

Right but hallucination rates have been consistently decreasing every model iteration. It's about error rates. As also a fellow scientist, I also will mess something up. Humans have an error rate. Once that error rate is low enough, it doesn't matter that it's > 0, it matters that it's low enough to be trustworthy and useful. Coding agents of 2024-25 had error rates too large; you couldn't meaningfully vibe code anything and needed a ton of oversight. It's still true but FAR less so, and this is after like a year of iteration.

>very far down the wrong path. Absolutely agree. Have seen this first hand

The free alternative to Wikipedia is the library, not “don’t learn anything new ever”. I find Claude is surprisingly similar to a confident but incorrect coworker, with the benefit that Claude will reevaluate when I correct it.

Slightly OT Nitpick: in regard to experts and Wikipedia, when doing a neuroscience-adjacent MSc, experts in the field actually directed me to Wikipedia as an excellent source for high-level neuroanatomy, including recent research, so I'm not sure your blanket description about experts and Wikipedia is correct.

You 100% can write them off entirely and go about your business as you previously had done. Ignoring the errors, it is very debatable whether there are even productivity gains beyond: human programmer or whatever is excited and cranked up to unsustainable degrees of activity and thinking to 'keep up' with what he thinks is an AI doing the work. I'm seeing this fairly often and when it isn't garbage it's a capable person who has gotten inspired by their 'collaboration' in which the busywork is being done by a machine, but they're doing so much directing and correcting that it's not unlike what would happen if they got heavy into meth and went on a tear. You absolutely can write them off entirely and decide for yourself what your comfort level of human-killing speed-freakism you want to pursue in your productivity. There's a long history of humans managing astonishing levels of productivity through self-destructive means. This is not even cheaper, once the 'first one's free' wears off: it's just a novel method of getting humans to burn themselves harder in the belief that they have a magic feather. The ones who're really throwing themselves into the situation are the ones who'll burn out, but who aren't setting themselves up for atrophy and learned helplessness. Anyone who believes the technology lets them be a lazy manager just getting paid, is in for an unpleasant discovery.

Can you elaborate on how you use Gemini to diagnose long term health issues? Considering doing the same for myself, but I have no idea what is too much vs too little information, and generally the type of prompt engineering to do.

So if you want to do a surgery but you don’t see any surgeons around you ask a grocery butcher to have his way?

How many times have you seen an expert go "yeah these results are good consistently enough for a non expert to trust them without expert assistance"? There is a huge difference between having a chance of a good result, which can be useful for experts able to filter out the bullshit, and consistent success. I would generate code as a helper, I would never allow a guy from marketing to merge unreviewed AI code.

I’ve never seen an expert use AI in their field beyond the initial ‘oh interesting’ stage.

How does a Large Language Model process images then?

There's a whole bunch of other studies on this topic, as well as metastudies, and from what I can tell the problem is real. https://www.sciencedirect.com/science/article/pii/S245195882... (+ cf. its references)

The OP describes getting injected with a homeopathic botanical formulation and receiving another type of therapy that wasn’t indicated for his condition. I wonder if this person was going to a traditional doctor or if they were visiting some type of specialty clinic as a second opinion. For most conditions you can find specialty clinics that will prescribe and administer (and bill for) a lot of non-indicated treatments, but some patients like being in the care of doctors who take action and do things after being recommended more conservative treatments by primary doctors.

In Germany we get zero-th opinion because you can't even get an appointment within the next 8 months.