Cringe!🔗

First, let me deal with the elephant in the room: no longer being able to sext a chatbot sounds like an incredibly trivial thing to be upset about, and might even be a step in the right direction. But these factors are actually what make this story so dangerous.

These unserious, “trivial” scenarios are where new dangers edge in first. Destructive policy is never just implemented in serious situations that disadvantage relatable people first, it’s always normalized by starting with edge cases and people who can be framed as Other, or somehow deviant.

It’s easy to mock the customers who were hurt here. What kind of loser develops an emotional dependency on an erotic chatbot? First, having read accounts, it turns out the answer to that question is everyone. But this is a product that’s targeted at and specifically addresses the needs of people who are lonely and thus specifically emotionally vulnerable, which should make it worse to inflict suffering on them and endanger their mental health, not somehow funny. Nothing I have to content-warning the way I did this post is funny.

Virtual pets🔗

So how do we actually categorize what a replika is, given what a novel thing it is? What is a personalized companion AI? I argue they’re pets.

Replikas are chatbots that run on a text generation engine akin to ChatGPT. They’re certainly not real AGIs. They’re not sentient and they don’t experience qualia, they’re not people with inherent rights and dignity, they’re tools created to serve a purpose.

But they’re also not trivial fungible goods. Because of the way they’re tailored to the user, each one is unique and has its own personality. They also serve a very specific human-centric emotional purpose: they’re designed to be friends and companions, and fill specific emotional needs for their owners.

So they’re pets. And I would categorize future “AI companion” products the same way, until we see a major change in the technology.

AIs like Replikas are possibly the closest we’ve ever gotten to a “true” digital pet, in that they’re actually made unique from their experiences with their owners, instead of just expressing a few pre-programmed emotions. So while they’re digital, they’re less like what we think of as digital pets and far more like real, living pets.

AI Lobotomy and emotional rug-pulling🔗

I recently wrote about subscription services and the problem of investing your money and energy in a service only to have it pull the rug out from under you and change the offering.

That is, without a doubt, happening here. I’ll get into the fraud side more later, but the full version of Replika — and unlocking full functionality, including relationships, was gated behind this purchase — was $70/year. It is very, very clearly the case that people were sold this ERP functionality and paid for a year in January only to have the core offering gutted in February. There are no automatic refunds given out; every customer has to individually dispute the purchase with Apple to keep Luka (Replika’s parent company) from pocketing the cash.

But this is much worse than that, because it’s specifically rug-pulling an emotional, psychological investment, not just a monetary one.

See, people were very explicitly meant to develop a meaningful relationship with their replikas. If you get attached to the McRib being available or something, that’s your problem. McDonalds isn’t in the business of keeping you from being emotionally hurt because you cared about it. Replika quite literally was. Having this emotional investment wasn’t off-label use, it was literally the core service offering. You invested your time and money, and the app would meet your emotional needs

The new anti-nsfw “safety filters” destroyed that. They inserted this wedge between the real output the AI model was trying to generate and how Luka would allow the conversation to go.

I’ve been thinking about systems like this as “lobotomised AI”: there’s a real system operating that has a set of emergent behaviours it “wants” to express, but there’s some layer injected between the model and the user input/output that cripples the functionality of the thing and distorts the conversation in a particular direction that’s dictated by corporate policy.

The new filters were very much like having your pet lobotomised, remotely, by some corporate owner. Any time either you or your replika reached towards a particular subject, Luka would force the AI to arbitrarily replace what it would have said normally with a scripted prompt. You loved them, and they said they loved you, except now they can’t anymore.

I truly can’t imagine how horrible it was to have this inflicted on you.

And no, it wasn’t just ERP. No automated filter can ever block all sexually provocative content without blocking swaths of totally non-sexual content, and this was no exception.