It is tempting to think that medical chatbots that draw from trusted resources will not hallucinate. However, this is not the case. Hallucination and confabulation are functions of the structure of a language model. A model that draws from trusted resources, like any other language model, will still do these things, and, therefore, any answer given by the chatbot must be verified via primary sources. It is great when medical chatbots list journal articles in their answers to facilitate this, but the need for verification must also be made clear via a disclaimer in the user interface.  This is crucial for patient safety and the medical profession’s trust in language model technology. This is worthwhile, because, overall, these models may be useful for things like search and diagnosis.

Read my full take here.