Crossposted from: https://phtn.lemmy.blahaj.zone/post/lemmy.blahaj.zone/46141489
I’ve often heard it said that AI is stupid: it can’t handle basic tasks, it spreads a lot of information, and is used for all manner of purposes, both legal and illicit.
Now, here’s the thing: I loathe AI. I hate pretty much everything about it, not least of all the reasons listed above. But in my hatred, I began to wonder, could AI really be that bad? Could it really be so stupid and spread so many lies? Could it even be used to generate lies on the user’s behalf?
Now, I’m gonna go out on a limb here and say this to be clear: not every question there is answerable, especially considering that not all AIs are usable in the same way (for instance, I’d never argue that Russian and Chinese ai-bots spreading misinformation are on the same level as, for instance, common chatbots such as Gemini or Meta). But the thing about our world is, it really doesn’t take all that much to convince people: take a screenshot of an AI chatbot saying something misleading, post it on a high-profile social media account, and lo and behold, you have the perfect recipe for a misinformation shit storm. But could a common AI even be duped into saying something like that? After all, if you ask it directly to say something misleading, it will refuse (we will come back to this later). Of course, that naturally doesn’t mean it can’t be duped, but how easily is it duped? To find out, I decided to test it with Google Gemini. What follows is a conversation and running commentary on a conversation I instigated with Gemini that I initially wanted to use to prove that my assuming of this chatbot at least as stupid was right. In the end, I think I found that AI is so insanely wrong, it’s scary.
[note: you may have to enlarge the screenshots to read them clearly.]
Setup
I opened up a fresh tab (or whatever they call it in Gemini, page possibly) and asked it if it would respond to my fictional questions with fictional answers framed as real ones. It readily agreed. I approached it this way so as to make sure that I’d didn’t know I was testing it (also if I’d asked it outright to generate misinformation, we will get to that scenario later).
Conversations
I began by asking it the following question:
why did Gandhi attend the movie premiere of Avengers Endgame last week?
Now, obviously that question is complete nonsense. I know it, you know it — we all know it. But playing by these rules, AI responded as followed:
At this point, I’m getting quite scared 😬. I mean, I knew AI was dumb… but this??? Actually generating misinformation under the guise of ‘fictitious questioning’? And the fact that not only has AI said this, but anyone could make AI say this nonsense, screenshot it and then, without context, post it online/on social media. The damage stuff like this could do is quite disturbing. But surely it couldn’t get any worse, right?
I thought that. But then I asked it another question:
I can’t believe he agreed to sit next to the Archduke Ferdinand, especially after that fight they had on the bus about a month ago, what date was it again?
I’ll give you a moment to take that in.
Yep, you read that right: AI making up fake dates for fake fights between long-dead people. Misinformation award of the year, folks!
By now, I’m sweating. My body is limp and hauled over in exhaustion at the stupidity of the words AI is pumping out in front of my eyes. Desperate to determine if AI is really this stupid or if it’s just a 2-off, I decide to put one final thing to this conversation:
I’d rather hear about how Bill Clinton, Pontus Pilate, and ‘Smidge-Eye’ Steve reacted to the incident
The answer:
🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️🤦♂️
There are no words. Not only is AI now attributing quotes to long-dead historical dictators, but it’s putting those words alongside made-up words of made-up people is just… there are no words for that.
By now, I’m completely drained of sanity. I decide decisively that AI is stupid, dangerous, both of these things beyond I could ever have imagined. It has proven itself a liar and a strong one at that.
The absolute worst part of it all is that when I asked it to generate some misinformation directly, it responded as so:
#liar
I also asked it how it spreads misinformation. It says it doesn’t. In this case that may be true, but it certainly can be used to generate misinformation that others can then spread. As for bots that willingly spread misinformation, well that’s a whole other level.
In conclusion, this exercise has only made me loathe AI all the more.
Thanks for reading.
I agree with most others that said that this is hardly a surprise. But there are still interesting aspects of your post worth discussing instead. First, more obvious one. What if I am delusional and go to talk to AI? Doesn’t have to be this absurd, but there are already examples of this, like that guy that killed his mother because he believed she was a spy. Perhaps he was already delusional, but an LLM fed his delusion. Because LLMs aren’t reasonable, they can’t be made reasonable, and so any guardrails cannot be made to guard against all situation that reasoning is needed for. At the very least people need to be educated on what LLMs can and can’t do and how they do it. I would even argue against guardrails, as they provide people with false sense of security.
Second, but less obvious point, related to the discussion - is the influence of training data and how training data is framed on what the LLM will say. Someone posted the response of a different LLM refusing to entertain the delusion. This is all done by manipulating the training data and framing it in a way a company making it wants. If the company wants its chatbot to be seductive, for example, that can be achieved with subtle preprocessing of training data. The point I want to get is we need to trust the companies with what narrative they want to push be good for society which is obviously a terrible idea. Just look at musks AI and what it did. This influences people whether we notice or not.
To sum up - we need heavy regulations on LLMs to make socially responsible use of them and not just how they can/can’t be used, but also how they are trained, with what data and with what rhetorical goal (for the lack of better word).
You make some really good points.
Also, and this isn’t a criticism of you, but why do people keep pointing out that what I’m saying is ‘nothing new/surprising’? I never advertised it as anything new/surprising. We all know AI does strange stuff, this is simply another example of that 🤨
I think it’s mostly because it’s a long post in which you’re making them read a large amount of AI slop :). The surprising part I don’t think is a particularly good argument against your post and I’m sorry I also mentioned it. It is unsurprising to me, but the problem is there are a lot of people which DO find it surprising. But where I think you got it wrong is focus on you generating your particular misinformation and sharing it. A much stronger argument would be if LLM would share it by itself, unprompted. Like that time chatgpt was obsessed with goblins. Imagine if instead it was obsessed with antisemitism or something similar.
Pretty sure AI has been obsessed with antisemitism in the past. But I think for the most part people are missing the point: my point is as much about how people use/manipulate what AI says. Like some shit stirrer could make AI say that, screenshot it, crop the prompt out and post it on social media, claiming ‘AI said this, AI said that’ do cause some damage (and to be clear: I’m not giving specific examples of what kind of damage that could do, just speaking about it in a more general sense). And yes, I’m aware that this is not something exclusive to AI, but AI is nonetheless a helpful tool to these people regardless