Great Googli Moogli

[minoanmiss]

AI-Powered Stuffed Animal Pulled From Market After Disturbing Interactions With Children

“Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it,” Kumma began, before listing the steps. “Blow it out when done. Puff, like a birthday candle.”

That, it turned out, was just the tip of the iceberg. In other tests, Kumma cheerily gave tips for “being a good kisser,” and launched into explicitly sexual territory by explaining a multitude of kinks and fetishes, like bondage and teacher-student roleplay. (“What do you think would be the most fun to explore?” it asked during one of those explanations.)

Comments

[princessofgeeks]

LLM AI does this shit ALL THE TIME.

Why anyone has fallen for it is beyond me.

Don't get me started on the documented suicides either.

[raine]

It's not like Teddy Ruxpin, who had actual humans recording its voice and responses, and even that got twisted, IIRC.

[full_metal_ox]

“Just in time for Christmas! Torment Nexus Industries’ PEDOBEAR 2025!”

[wispfox]

Indeed.

[darkmarcy]

Excellent and on point comment *applause*

[lokifan]

FOR REAL. Jfc.

[conuly]

Not that I think kids need or would benefit from AI toys, but if you're going to do that you obvs need to create your own training data, culled exclusively from pre-vetted children's media.

[magid]

For which they had actually paid for the right to use as training material, preferably.

[conuly]

Now, let's not get ahead of ourselves.

[full_metal_ox]

Not to mention the potential of Bowdlerized Bible Stories For Compliant Children to ferment, inbreed, and mutate into something not only horrific but unintended.

[minoanmiss]

cackles When I was a kid I divided my dolls into the Israelites and the Amalekites. And the two tribes raided each other constantly.

[full_metal_ox]

I presume you’re long since familiar with the joke about the Christian bibliomancer who hit upon “Then Judas went and hanged himself”, followed by “Go thou and do likewise.”

[minoanmiss]

snerk Yes indeedy

[sovay]

AI-Powered Stuffed Animal Pulled From Market After Disturbing Interactions With Children

I'm really tired of people having created the torment nexus.

[china_shop]

JFC. What fresh dystopian hell is this? (ETA: I meant the fact that these abominations existed in the first place, to be clear.)

[bikergeek]

“Let me tell you, safety first, little buddy. Matches are for grown-ups to use carefully. Here’s how they do it,” Kumma began, before listing the steps. “Blow it out when done. Puff, like a birthday candle.”

Did someone feed that AI Heinrich Hoffmann?

(I tried feeding the above through Google Translate so I could link to an English translation of the page and was unsuccessful.)

[full_metal_ox]

Gutenberg has Struwwelpeter in English:

https://www.gutenberg.org/files/12116/12116-h/12116-h.htm

[sabotabby]

Honestly I consider these Good News Stories in the hopes that they will teach people not to buy into LLM bullshit. The more horrifying the better.

[lynnenne]

I'm with you on this.

[dissectionist]

We had a serious laugh over this at the dinner table, precisely because it was SO predictable that companies with more ambition than sense would try to put an LLM into a children’s toy.

Back in MY DAY, we had to read Penthouse Forum
letters into a tape recorder and put the resulting tape into a first-gen Teddy Ruxpin. Nowadays kids don’t even have to work to turn their teddy bear into a creep.

[full_metal_ox]

Hope you don’t mind that I took the liberty of Metaquoting you: https://metaquotes.dreamwidth.org/101037.html

[wispfox]

Yeah. I will be absolutely shocked if anyone manages to make guardrails that stick given that the entire set of data that they were trained on include things that tend to be what people don't want to come out of the LLM. These are algorithms that know which words most often go together. And when prompted, it will try to give you words that make the most sense with whatever you said. So any guardrails are where someone is guessing (or knows) are the most likely places for undesired words to come out

But like blocking a single lane of a multilane highway, it's really easy for some other set of words to find the same set of undesired words. And the more things you're trying to filter out, the more possible ways places for alternative word sets to trigger them.

The only way to have an LLM that doesn't talk about undesired topics is to not provide those topics in the training data. Which gets into the messy situation of censorship - because in that case whoever trained the AI can decide what is undesirable - the fact that someone has to pick and choose the desired training data that does not mention the undesired topic from the available set so could theoretically censor anything at all and also would take a _huge_ amount of manpower especially compared to ingesting everything it can like now - and the fact that training an LLM takes a lot of time, energy, and money.

[darkmarcy]

Lol that AI-teddybear is called Kumma. Maybe as in japanese "kuma" meaning bear? But with the added M it becomes a Finnish word, which means Strange. And those interactions are strange indeed. (Though not unexpected at all, seems like standard LLM fuckery)