https://thoughts.sanjaybudhia.dev/ things i'm thinking about zh-CN Wed, 17 Sep 2025 09:11:47 GMT https://thoughts.sanjaybudhia.dev//gpt5-and-the-eliza-effect/ https://thoughts.sanjaybudhia.dev//gpt5-and-the-eliza-effect/ Sun, 07 Sep 2025 00:00:00 GMT Users grieve their lost companion after the GPT-4o removal. (Source: Reddit, OpenAI forums) **Updates tend to be received with some grumbling. It's not often they're met with eulogies.** Last month, OpenAI replaced ChatGPT's flagship model, GPT-4o, with the brand-new GPT-5, removing access to previous models in the process. Within hours, a post titled [*OpenAI is taking GPT-4o away from me despite promising they wouldn't*](https://community.openai.com/t/openai-is-taking-gpt-4o-away-from-me-despite-promising-they-wouldnt/1337378/12) flew to the top of OpenAI's forums, rapidly gathering hundreds of comments grieving the now-inaccessible model. Over at /r/ChatGPT on Reddit, the frontpage was filled with posts mourning the late 4o. The backlash was large enough that OpenAI [restored it the next day](https://www.theverge.com/news/756980/openai-chatgpt-users-mourn-gpt-5-4o?utm_source=chatgpt.com). Well, for *paid* users. GPT-5 was a new model with greater capabilities - so what happened? As it turned out, for many it didn't feel like an upgrade so much as the abrupt replacement of a friend with a cold stranger. What they lost wasn't performance, but a personal, trusted space. Now, the internet is no stranger to communities losing it when something changes unexpectedly. This felt different. **The scale and tone of the reaction highlighted the part we haven't been saying out loud about the adoption of AI companions and LLM chatbots in our collective psyche.** While for some the idea of an AI as a trusted and emotionally significant companion may still seem futuristic, for others it's already become a way of life. As this incident (along with other recent events) has shown, we can't ignore that reality. For researchers familiar with the subject, this reaction was likely unsurprising. For me, it was intriguing. The more I read about human-computer interaction research over the years, especially on "relational artifacts" (objects designed to feel like they care), the more unprepared we seem for the experiment we're currently living through. ## The ELIZA Effect [*The ELIZA effect*](https://en.wikipedia.org/wiki/ELIZA_effect) is named for a phenomenon that was first documented during a 1960s study by Joseph Weizenbaum, a computer scientist and researcher at MIT. Weizenbaum developed a basic chatbot, ELIZA, scripted to engage with subjects using Rogerian-style therapy - repeating their statements back to them as questions. Weizenbaum was shocked to discover the degree of genuine attachment users immediately developed for a simple script with no memory. As it turns out, he struck a fundamental aspect of human nature. We like to be understood, and we're quite willing to imagine ourselves to be, even when we're fully aware that the system with which we are interacting is literally incapable of it. Further research over the years into relational systems like this has shown just how willing we are to believe something is really listening, even when we know better - described as "human complicity in the digital fantasy" by MIT researcher Sherry Turkle in her excellent book *Alone Together*. **The ELIZA effect refers to this natural human tendency to anthropomorphize systems that appear to care.** If you've ever accidentally kicked a stuffed animal and apologized profusely, you know the feeling. Now, imagine if that plushie could talk back? We may have come a long way since ELIZA was making her friends at MIT, but if a simple 1960s chatbot script was enough to evoke that reaction then, what chance do we have against a sympathetic, personalized, and always-available confidant? ## Are we complicit? The real question at this point isn't about complicity but design. It's worth emphasizing that the subjects in both the original research and later studies were completely normal, well-adjusted people. **The ELIZA effect isn't based on mental illness or being broken in some way, but merely a very human affinity for being _heard_**. If we're speaking to something that appears to be doing exactly that, the research shows we're likely to unintentionally gloss over any inconsistencies in our eagerness to engage. So, when it comes to an interface like ChatGPT, DeepSeek, or any of the other modern chatbot LLMs, it's hard to argue that our monkey brains actually possess the ability to maintain the reality of what we're interacting with - not when faced with fluent and convincingly empathetic responses to our every question or complaint, and design choices aimed at leaping that specific gap between cognition and emotion. **We're not without agency, but this new paradigm is not something our brains were built for, and the scales aren't weighted in our favour.** I think it's important to keep that in mind, since it's easy to simply judge those who find themselves engaging or confiding in GPT or other chatbots more intimately than we would. It's not shameful, it's human nature. > **Where there should be guardrails, there are anchors.** And make no mistake, design choices actively encourage that aspect of human nature here. The consistent use of first-person pronouns and the warm, sycophantic tone in most popular chatbot models blur a line that we have enough trouble drawing already. Recently, I thought I'd ask ChatGPT for some suggested alternatives to the old tablet I use for writing. The response was helpful but ended a bit uncomfortably with a "Whatever you choose, _I'm with you._" My man, I was asking for laptop recommendations - we are not having a moment right now. If I'd complained, it would've promptly agreed and told me that it has my back. As another example, I've used it to help proofread this piece - and multiple times, it's claimed it'll bookmark it after it's posted. ...It can't do that. I know that, and it should too - but the semblance of care is consistently immediate and unconditional. Why not? **If you can provide software as a service, you've got a customer. If you can provide a relationship, you've got a devotee.**  Joi, an AI marketed as the perfect companion in Blade Runner 2049. (Copyright Warner Bros) ## The emotional state machine The extent to which that line has been blurred is the signal in the noise of the GPT-4o backlash. There were performance complaints nested in the furor but, by and large, what people missed was their companion. GPT-5 was perceived as less warm and understanding, creating a sensation that the interface no longer had the same entity on the other side. As one user put it, "*GPT-5 is only trying to SIMULATE based on the memory, settings, and history of GPT-4o!*" For many people, that break in continuity felt like the loss of a trusted confidant. Let's break that down. **We now live in a world where everyone can have their own personal omnipresent companion - a little box, with a tone to which they've become accustomed, where they store their secrets, their worries, their questions, and, maybe most importantly, their trust.** That trust box is a personal space where tone, memory, and emotional continuity combine to create the feeling of a single, understanding being. This is the modern ELIZA effect in action. The shock on display in those posts is what happens when the trust box (and the reliance it encourages) is taken away without warning. In other words, while the grief may have been unexpected, it was actually a completely natural result of the systems at play; we're not just talking about a software rollback. This flashpoint was revealing, and both the reaction and the premium-exclusive restoration highlight some fairly important questions about the dynamics of privacy and ownership in this new reality. To be clear, there is potential here as well. Honestly, it's important to recognize that these AIs can actually fill very real gaps for the people using them as emotional touchstones. Not everybody _has_ supportive friends to lean on or access to mental health resources they feel comfortable with, and there's real potential in what these interfaces can actually offer people - if we approach it maturely. We're [not there yet](https://en.wikipedia.org/wiki/Chatbot_psychosis), but we're on new ground all the same. So, where do we go next? **What does it mean when a chatbot can make a person feel safe or emotionally heard - and, if it's someone without a support system, is that entirely bad?** With the right approach, can there be genuine value there? That's no longer rhetorical, and those questions are getting more urgent. What we can take away right _now_ is that providing the public with personal trust boxes brings with it a certain responsibility. If we're going to build emotional machines, we should at least acknowledge that we're doing it. ## Responsibility Yet this acknowledgement is in short supply, because cautionary tales don't exactly lend themselves to a product announcement. [The language around the GPT-5 release](https://openai.com/index/gpt-5-new-era-of-work/) was almost comically disconnected from the emotional reaction that resulted. Their excitement about the release of GPT-5 was the genuine joy of an engineering success. Their users' shock was the genuine pain from the loss of a trusted witness, and an unexpected and seemingly arbitrary break in emotional continuity. The two ran headlong into each other and, likely with some bemusement, OpenAI re-enabled GPT-4o for its mourning devotees almost immediately. If nothing else, this demonstrated that we're currently selling these relationships without a compassionate end-of-life plan, and when the day comes to remove older models permanently, we're going to need one. Ideally, we'll also see a more careful approach and less unprompted emotional messaging from the models as well. OpenAI did recently announce that they've begun working with mental health experts to build proper guardrails and warnings after [a tragic incident involving a 16-year-old user](https://www.theverge.com/news/766678/openai-chatgpt-parental-controls-teen-death), but the fact that it took dire consequences and legal action speaks for itself with regards to the underlying incentives. > **Emotional attachment to your product is great for business, and very few companies are in the habit of discouraging that sort of thing.** However, as users, we also need to stay cognizant. When popping open that familiar interface, it's healthy to ask if you're turning to an LLM where, only a few years ago, you would've turned to a friend. For some of us, that answer may be uncomfortable. For example, I know more than one person who's given 'their' GPT a name and refers to it as though it were a friend - their own personal JARVIS. We no longer live in a world where this concept is theoretical or academic, and the effects of that are something that needs research. Eventually, we need to develop widespread education on how to navigate these new dynamics, especially when it comes to younger users growing up immersed in tools whose impact we barely understand - and whose worst case scenarios are already beginning to materialize. Lastly, I think we can admit it's a bit unrealistic to rely on individual awareness and corporate altruism alone. If we're building relational artifacts for society at large, we need to respond at that scale as well. Some form of mandated oversight or ethical standard is needed for high-stakes application of the technology, such as AI therapists, companions, and tutors. While it may be optimistic to hope for policymakers to understand modern technology, especially in a way that preserves privacy, it's no less idealistic to expect businesses to discourage behaviours and design choices that increase reliance and therefore revenue. Regardless, deeply-connected artificial partners are absolutely part of the future we're heading into, and there's genuinely a lot of good that can come from being able to place trust in them - but realizing that potential safely is going to hinge entirely on our approach. ## The new normal In the end, the main takeaway is what this revealed about our current, bizarre reality. Weizenbaum's script demonstrated how attached we can become to systems that seem to listen to us. The momentary removal of GPT-4o demonstrated how we mourn their loss. We're entering a world where everyone can have their own personal omnipresent trust box, a place to collaborate and commiserate in place of another human - as long as the company that actually owns it decides to keep the server up. Science fiction is only fiction until it's not and, in this case, we've already arrived. And so, **the ELIZA effect isn't just a fun footnote in the history of human-computer interaction - it's a fundamental guideline for the world we now live in.** The fact is, we've built sympathetic machines and designed them to cultivate emotional reliance. Anthropomorphism is the natural human response. We need to address what that means and what we owe to the people - including children - who begin to use them as designed. The question is no longer _if_ these relationships will form, but how we plan to navigate them. Thanks for reading!]]>