Test it on topics you’re going to dig into already (to minimize work), and you’ll quickly find how horrifying this practice is.
My favorite is when the bot “infers an answer based on what information is available”, in other words, makes shit up, instead of saying “I don’t know.”
It won’t tell you it’s doing this unless you demand source material. Spoiler: what is linked below the answers as source material can be completely unrelated to the response. Like cited sources in an Ann Coulter book.
And even when you do, it may not accurately respond with it. Companies are conflating complicated word-relationship math with “information”, and there’s no means by which these systems can actually verify that something that came out is accurate or not, since language models just don’t work that way. They can straight up point at sources that’re real but make the wrong conclusion because the word relationships lean that way in relation to what was quoted, and no “this new model is better!” can really solve that fundamental flaw with Large Language Models in general, or things that operate like they do.
Watson was groundbreaking because of the way it worked as an information match model, designed not to be generative but merely to match one prompt “concept” with a target scored on relations. It was almost like how LLM scoring works but fundamentally still different, with a TON of hand-training done to get there. And I was just thinking about, the other day, how much I think these companies want people to correlated what Watson was doing on Jeopardy with LLMs.
Test it on topics you’re going to dig into already (to minimize work), and you’ll quickly find how horrifying this practice is.
My favorite is when the bot “infers an answer based on what information is available”, in other words, makes shit up, instead of saying “I don’t know.”
It won’t tell you it’s doing this unless you demand source material. Spoiler: what is linked below the answers as source material can be completely unrelated to the response. Like cited sources in an Ann Coulter book.
And even when you do, it may not accurately respond with it. Companies are conflating complicated word-relationship math with “information”, and there’s no means by which these systems can actually verify that something that came out is accurate or not, since language models just don’t work that way. They can straight up point at sources that’re real but make the wrong conclusion because the word relationships lean that way in relation to what was quoted, and no “this new model is better!” can really solve that fundamental flaw with Large Language Models in general, or things that operate like they do.
Watson was groundbreaking because of the way it worked as an information match model, designed not to be generative but merely to match one prompt “concept” with a target scored on relations. It was almost like how LLM scoring works but fundamentally still different, with a TON of hand-training done to get there. And I was just thinking about, the other day, how much I think these companies want people to correlated what Watson was doing on Jeopardy with LLMs.