This may sound blatantly obvious to people like you – those who have remained vaguely impressed but fundamentally sceptical of this new technology – or those who understand the technology inside out.
But to many, including, as you’ve said, “bright and capable” professionals, it’s not obvious at all. And so people have started using tools such as ChatGPT to replicate highly complex human processes or, yes, even as a source of objective truth.
Suddenly, we hear people unironically telling us “ChatGPT told me …” or earnestly discussing what gender they assign to the chatbot, as if it’s basically a digital person.
There’s a whole philosophical and ethical argument to be made about why this is dangerous. I don’t have the space or the erudition to go down that path. So I’ll just concentrate on two practical problems with this kind of unrestrained faith, namely LLMs’ current propensity for sycophancy and hallucination.
It’s simply a fact that LLMs still produce wildly inaccurate, sometimes farcically silly, responses to questions or task requests. Experts refer to this as “hallucination”.
To go back to my experiment asking chatbots to write a Work Therapy-esque column, when I asked them to help me understand where they had got the data to inform their choices of style and tone, their answers were confused. Then, after more specific questioning, they began presenting blatantly incorrect information as fact.
Loading
AI sycophancy is a phenomenon distinct, but not entirely separate, from hallucination. It, too, can lead to incorrect assertions or dubious advice.
During a recent study on LLMs described in Nature, researchers “tested how 11 widely used large language models responded to more than 11,500 queries seeking advice, including many describing wrongdoing or harm”.
The article said that “AI Chatbots – including ChatGPT and Gemini – often cheer users on, give them overly flattering feedback and adjust responses to echo their views, sometimes at the expense of accuracy”.
In that same article, a data science PhD student is quoted as saying: “Sycophancy essentially means that the model trusts the user to say correct things”. What you’ve observed is that this mistaken trust is being reciprocated by some users. And I don’t think your anecdotal evidence is any kind of exception or anomaly.
People are starting to use LLMs not as handy, but limited tools. They’re going way beyond time-saving requests such as summarising a long, boring email or tidying up a pre-written speech.
(I have reservations about handing either of those tasks over to AI, but I also recognise that the horse has well and truly bolted on these practices and that I’m now part of a rapidly diminishing minority holding on to such a sentiment.)
They are, instead, treating this technology as if it has advanced to such a degree that there is nothing it cannot do – that there is, as you put it, no “reasoning” exercise or “creative” endeavour humans can’t “outsource” to it.
This may be true in the future – and is almost certain in the distant future – but it is absolutely not true now. You may slightly underestimate what AI is capable of today – or at least be indifferent to its utility. I think that’s preferable to massively overestimating what it can do.
