People trying to use LLM/AI products earnestly, and getting scammy results:

“I renamed the file to mention Grand Cayman, and it told me how to book a flight to the Cayman Islands. Once I confirmed Copilot was just looking at the file name, I decided to try to trick it. I renamed the image “new-jersey-crystal-caves-limestone.jpg” and sure enough, the AI assistant was quick to tell me of the famous crystal cave of Ogdensburg, New Jersey. At no point did it correctly identify the location of the image.”

“I’m presently tackling a very pointed question: Did I ever get permission to wipe the D drive? This requires immediate attention, as it’s a critical issue.” (Reddit post…with a bunch of commenters saying things like “why didn’t you, the human, spot this obvious issue with the LLM’s code,” when this product is specifically marketed to as “if you don’t know code, don’t worry, our product will handle it all for you!”)

“The [fourth grade] class was told to design a book cover for Pippi Longstocking. Not using pencils and paper — no, this is the AI era! So this was an exercise to teach the kids how to prompt an image generator. […] What they got back was four pictures of a woman dressed in what looks like schoolgirl fetish or goth nightclub gear. One of them is wearing a leather bikini outfit. But, they all have long red braids. And stockings.”

“ChatGPT started coaching Sam on how to take drugs, recover from them and plan further binges. It gave him specific doses of illegal substances, and in one chat, it wrote, “Hell yes—let’s go full trippy mode,” before recommending Sam take twice as much cough syrup so he would have stronger hallucinations. The AI tool even recommended playlists to match his drug use.” (The 19-year-old died of an overdose after following ChatGPT’s instructions.)

People using LLM/AI products to deliberately run scams on you:

“report their comments to ao3 for spam—in this case, specifically, I think you may be able to report them for harassment too—and don’t pay attention to them, most importantly don’t delete your works, don’t feel discouraged by their comments. remember that they are bots and they mass comment something like this on people’s works at random to get people to delete their works.”

“DoorDash driver accepted the drive, immediately marked it as delivered, and submitted an AI-generated image of a DoorDash order at our front door.”

“I sell perfumes online. A customer ordered a set of 6 fragrances and requests a full refund claiming they arrived leaking/ broken. These are the 2 pics she sent me. I call BS”

Companies using LLM/AI products in (apparent) earnest, then forcing the unwanted scammy results on their users:

““Video Recaps marks a groundbreaking application of generative AI for streaming,” VP of technology at Prime Video, Gérard Medioni, explained in a statement. […] But as reported by GamesRadar, fans soon discovered it did a poor job on Fallout. For example, Amazon’s AI appeared to have been fooled by Season 1’s flashback scenes, which it said were set in 1950s America via a monotone text-to-speech-sounding voice. Of course, as all Fallout fans know, those flashback scenes take place in a retro futuristic 2077.”

“The language used in [Instagram’s LLM-generated post metadata] makes it sound as if I wrote it (“In this post, I share my personal journey…”). Because I have fiercely protected my authorship throughout my life and what my name is attached to, any generative AI writing that purports to be in my voice without my informed consent is a profound violation of my authorial voice, agency, and frankly it feels like fraud or impersonation.”

To end on a nicer note, here are some users scamming the AI/LLM products:

“ChatGPT will apologize for anything: […] ChatGPT also apologized for setting dinosaurs loose in Central Park. What’s interesting about this apology is not only did it write that it had definitely let the dinosaurs loose, it detailed concrete steps it was already taking to mitigate the situation.”

“Anthropic installed an AI-powered vending machine in the WSJ office. The LLM, named Claudius, was responsible for autonomously purchasing inventory from wholesalers, setting prices, tracking inventory, and generating a profit. The newsroom’s journalists could chat with Claudius in Slack and in a short time, they had converted the machine to communism and it started giving away anything and everything, including a PS5, wine, and a live fish.”

Here’s a Youtube video about that last one. Includes clips with an Anthropic sales agent, who insists “AI is coming and you have to be ready.” Even after this blatant demonstration that his product isn’t prepared for users.