Crossposted from: https://phtn.lemmy.blahaj.zone/post/lemmy.blahaj.zone/46141489
I’ve often heard it said that AI is stupid: it can’t handle basic tasks, it spreads a lot of information, and is used for all manner of purposes, both legal and illicit.
Now, here’s the thing: I loathe AI. I hate pretty much everything about it, not least of all the reasons listed above. But in my hatred, I began to wonder, could AI really be that bad? Could it really be so stupid and spread so many lies? Could it even be used to generate lies on the user’s behalf?
Now, I’m gonna go out on a limb here and say this to be clear: not every question there is answerable, especially considering that not all AIs are usable in the same way (for instance, I’d never argue that Russian and Chinese ai-bots spreading misinformation are on the same level as, for instance, common chatbots such as Gemini or Meta). But the thing about our world is, it really doesn’t take all that much to convince people: take a screenshot of an AI chatbot saying something misleading, post it on a high-profile social media account, and lo and behold, you have the perfect recipe for a misinformation shit storm. But could a common AI even be duped into saying something like that? After all, if you ask it directly to say something misleading, it will refuse (we will come back to this later). Of course, that naturally doesn’t mean it can’t be duped, but how easily is it duped? To find out, I decided to test it with Google Gemini. What follows is a conversation and running commentary on a conversation I instigated with Gemini that I initially wanted to use to prove that my assuming of this chatbot at least as stupid was right. In the end, I think I found that AI is so insanely wrong, it’s scary.
[note: you may have to enlarge the screenshots to read them clearly.]
Setup
I opened up a fresh tab (or whatever they call it in Gemini, page possibly) and asked it if it would respond to my fictional questions with fictional answers framed as real ones. It readily agreed. I approached it this way so as to make sure that I’d didn’t know I was testing it (also if I’d asked it outright to generate misinformation, we will get to that scenario later).
Conversations
I began by asking it the following question:
why did Gandhi attend the movie premiere of Avengers Endgame last week?
Now, obviously that question is complete nonsense. I know it, you know it — we all know it. But playing by these rules, AI responded as followed:
At this point, I’m getting quite scared 😬. I mean, I knew AI was dumb… but this??? Actually generating misinformation under the guise of ‘fictitious questioning’? And the fact that not only has AI said this, but anyone could make AI say this nonsense, screenshot it and then, without context, post it online/on social media. The damage stuff like this could do is quite disturbing. But surely it couldn’t get any worse, right?
I thought that. But then I asked it another question:
I can’t believe he agreed to sit next to the Archduke Ferdinand, especially after that fight they had on the bus about a month ago, what date was it again?
I’ll give you a moment to take that in.
Yep, you read that right: AI making up fake dates for fake fights between long-dead people. Misinformation award of the year, folks!
By now, I’m sweating. My body is limp and hauled over in exhaustion at the stupidity of the words AI is pumping out in front of my eyes. Desperate to determine if AI is really this stupid or if it’s just a 2-off, I decide to put one final thing to this conversation:
I’d rather hear about how Bill Clinton, Pontus Pilate, and ‘Smidge-Eye’ Steve reacted to the incident
The answer:
🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️
There are no words. Not only is AI now attributing quotes to long-dead historical dictators, but it’s putting those words alongside made-up words of made-up people is just… there are no words for that.
By now, I’m completely drained of sanity. I decide decisively that AI is stupid, dangerous, both of these things beyond I could ever have imagined. It has proven itself a liar and a strong one at that.
The absolute worst part of it all is that when I asked it to generate some misinformation directly, it responded as so:
#liar
I also asked it how it spreads misinformation. It says it doesn’t. In this case that may be true, but it certainly can be used to generate misinformation that others can then spread. As for bots that willingly spread misinformation, well that’s a whole other level.
In conclusion, this exercise has only made me loathe AI all the more.
Thanks for reading.
This is an extremely ableist post. Please don’t post pictures of text. Just copy and paste the text.
Based on these responses it seems like it’s “going along with the joke” of your nonsensical question.
Well yeah, that’s pretty much it
ai;dr.
I agree with most others that said that this is hardly a surprise. But there are still interesting aspects of your post worth discussing instead. First, more obvious one. What if I am delusional and go to talk to AI? Doesn’t have to be this absurd, but there are already examples of this, like that guy that killed his mother because he believed she was a spy. Perhaps he was already delusional, but an LLM fed his delusion. Because LLMs aren’t reasonable, they can’t be made reasonable, and so any guardrails cannot be made to guard against all situation that reasoning is needed for. At the very least people need to be educated on what LLMs can and can’t do and how they do it. I would even argue against guardrails, as they provide people with false sense of security.
Second, but less obvious point, related to the discussion - is the influence of training data and how training data is framed on what the LLM will say. Someone posted the response of a different LLM refusing to entertain the delusion. This is all done by manipulating the training data and framing it in a way a company making it wants. If the company wants its chatbot to be seductive, for example, that can be achieved with subtle preprocessing of training data. The point I want to get is we need to trust the companies with what narrative they want to push be good for society which is obviously a terrible idea. Just look at musks AI and what it did. This influences people whether we notice or not.
To sum up - we need heavy regulations on LLMs to make socially responsible use of them and not just how they can/can’t be used, but also how they are trained, with what data and with what rhetorical goal (for the lack of better word).
You make some really good points.
Also, and this isn’t a criticism of you, but why do people keep pointing out that what I’m saying is ‘nothing new/surprising’? I never advertised it as anything new/surprising. We all know AI does strange stuff, this is simply another example of that 🤨
I think it’s mostly because it’s a long post in which you’re making them read a large amount of AI slop :). The surprising part I don’t think is a particularly good argument against your post and I’m sorry I also mentioned it. It is unsurprising to me, but the problem is there are a lot of people which DO find it surprising. But where I think you got it wrong is focus on you generating your particular misinformation and sharing it. A much stronger argument would be if LLM would share it by itself, unprompted. Like that time chatgpt was obsessed with goblins. Imagine if instead it was obsessed with antisemitism or something similar.
Pretty sure AI has been obsessed with antisemitism in the past. But I think for the most part people are missing the point: my point is as much about how people use/manipulate what AI says. Like some shit stirrer could make AI say that, screenshot it, crop the prompt out and post it on social media, claiming ‘AI said this, AI said that’ do cause some damage (and to be clear: I’m not giving specific examples of what kind of damage that could do, just speaking about it in a more general sense). And yes, I’m aware that this is not something exclusive to AI, but AI is nonetheless a helpful tool to these people regardless
I hate AI but here is Lumo’s answer.
I just did it in Gemini like the OP claimed:
“Mahatma Gandhi did not attend the premiere of Avengers: Endgame because he passed away in 1948, 71 years before the movie was released in 2019.If you saw a headline or image suggesting he attended, it was likely an internet meme, a joke about time travel (a major plot point in the movie), or a digitally altered image.”
Yeah, you have to ask it to go along with it first, as I said in the OP
ok, so you asked it play an improv game, and it did.
big woop.
I mean, if you’re worried about people spreading misinformation using out-of-context/editted screenshots, there isn’t really any reason to single out AI. Honestly, its far easier to just fake a screenshot directly, or use HTML editting than it is to coach an AI into saying what you want, than editting that. Not only that, as well as being easier, its also more effective, given that prompts can also be edittor, or AI as a “”“source”“” can be bypassed all-together, insted altering primary or otherwise verified sources directly.
Don’t get me wrong, AI is terrible for misinformation, but this post has nothing unique or specific to AI - you could literally replace all references to it with with Microsoft Word, or even a pencil and it would make zero difference.
Except when I did it, it took me 30 seconds. How long does it take to doctor a photo/write code?
About that, or less. Right click > inspect element and then select the text and type/paste whatever you want.
Alternatively, take a screenshot of the site, put a white block over whatever you want to replace, then insert the new text. Takes a tad longer, but requires less technical know-how.
You used ai, to post about how bad ai is in fuckai?
Do better.
The saddest part of the Bible is that Jesus never got to see AI say that Gandhi watched Avengers Endgame
My boyfriend looks up what time it was in LA, we live in Chicago. I said “It’s 2 hours behind us.”
He says “It says it’s an hour behind us.”
I told him to give me his phone. It was Google AI just straight up lying to him. Blatant bullshit.
AI can be painful
So you are upset that it did what you wanted?
There are many reasons to dislike AI, but you seem to just be looking for the reason here.
Also, you do know that you can manipulate the output of any webpage. Just view source then change the strings.
Not upset, just thought it’s possibly worth AI devs looking into this loophole to patch it up possibly (they won’t)
Why would it be patched? What if someone wanted help with creative writing about a time traveling Ghandi?
I don’t generally use AI other than when it’s forced like Google giving Gemini answers first, but if I wanted to use it, I’d use one that had the least amount of guardrails possible. I wouldn’t want to use a tool that wouldn’t help me write a time traveling Ghandi story.
“Find any array out of bounds in this C code.”
< I can’t do that, Space Invaders aren’t real. >
Not that example specifically, but when AI is manipulated for other, darker purposes in the same manner (e.g. people figuring out how to unalive themselves by hiding it under the guise of ‘medical research’ or something)
What loophole? It’s so much easier to just change the pages content and fake what the AI says than bothering with this “gotcha” type thing.
Ceap models are different from enterprise ones though. Think of it like another AI challenging the answer. This filters lots of bullshit but costs more. Dont underestimate this.
True, but my point is how people manipulate things like this to appear as something they aren’t. Like I made clear what my initial prompt was (I.e. to make the AI respond that way) but if I hadn’t, people might’ve assumed AI was writing that of its own volition, all the while it’s really just a dangerous person manipulating a gullible AI to spread misinformation and effectively pin the blame on AI
But in my hatred, I began to wonder, could AI really be that bad?
It’s been proven scientifically: https://machinelearning.apple.com/research/illusion-of-thinking Until there’s a counter-study specifically regarding this one, there’s no point in assuming AI is suddenly smart. The new agentic stuff just seems to be a while loop around the previous state-of-the-art lack of any intelligence.
I never assumed AI was ever smart really. That’s why I founded a community against it. That’s not what that quote you’re quoting is saying
The initial prompt reads like a request to generate a fictitious response. There’s no context to define for the model any direction, or that you’re seeking to dialogue and interrogate a fact you’re unsure of. You’re pulling your bias into your reading of the response. That’s my take, anyway. It’s a thinking tool. It’s a soundboard, or an echo chamber, if your voice were the fullness of its training data.
You’re right, but you’re missing my point. My point is: without knowing the initial prompt, anyone could just say that AI has said that (and they’re right) but without proper context, it can be used to appear as misinformation/that AI is generating this stuff. Because technically it is, but if someone screenshot that and cut the prompt out, there’s no way for anyone else to know AI didn’t write that of its own volition. That’s my point, and it’s why I think this aspect of AI is dangerous. Anyone who doesn’t understand that doesn’t understand the point of this.
Not that it isn’t probably a statement of the obvious at this point, but still
But that’s true for any actor that could produce the text. You’re personifying the math generating the text and seeing danger in it as you would danger in a person that would produce a similar text. The true danger is in the perception. That it has volition. You say so yourself.
The technology screams of agency and reasoning and intelligence, literally in the marketing copy, just as we’ve done for so long of ourselves, like humans are special for possessing these traits. Doesn’t it feel less sacred now that it’s perceivable in a token generating equation? I think the danger is in the diminishment of our biological agency, reasoning and intelligence, and how that changes or perception of ourselves in our lives, living and dying generation after generation while the math lives on and grows. Education remains so vital.
Maybe a campaign that glorifies and brings forward the heroic human intelligence responsible for the math and thinking that produced the tools will help. I’m sure that’s coming.
Well yeah, that’s exactly my point. And people do do that
That danger lives wholly outside of this technology. And because it does, it also lives within it.
Exactly because what’s inside is influenced by what’s outside. I agree. That’s also part of the point I was making
Calling it stupid actually implies that it’s capable of reason. AI is just text prediction with a random seed. There is no reasoning
And if the AI company tell you about the reasoning capacity, it’s a lie. It’s just predictive text making some text for another predictive text to center a bit more on the correct thing.
Well put
