Seralth

I actually wanted to automate some large scale testing for vintage story. Figured i would set my local LLM on the task to sort it out just to see what it would do. I drafted up a nearly three page document with clear instructions, rules, tools, examples and goals. Put hard limits on the sandbox the LLM runs in so that it couldn’t choose to just ignore the rules that could cause security issues and i let it lose.

It started with basic mouse and keyboard inputs and figured out by it self how to launch the game and run it though the user interface. After about 4 hours it stopped. Stated in its logic that what it was doing is “inefficient and wasting time” Then proceeded to promptly start working on a way to directly interface with it by designing a bot, getting a smaller model i had on file that could load along side it and drive the bot. It then started working on the hard problems would hand basic instructions to the smaller llm and it would drive a bot that loaded into the game as a mod.

After about 12 hours of total work it basically created a useful and well designed and functional vintage story bot and testing system. Would have likely taken me twice as long to design the bot.

Its been working well for about two weeks now. If i had just vibed out a half assed request or put in no hard safeguards outside of the LLMs control it likely would have done something fucking stupid. As with anything, its almost ALWAYS user error. And only an idiot blames their tools for their own fault.

Blindly following them while also telling the model to never question them. It creates stupidity feedback loops.

AI will happily tell you that is a stupid idea at this point. The bigger problem is that a LOT of users actively tell the models to listen to them, not to question them, and to never push back. To do things on demand and with out planning out the next step.

At this point no reasonable model of any reasonable size should struggle with this sort of task. And they generally speaking don’t. Its almost always user error poorly driving them now. The models arn’t smart, they don’t understand things. But they also arn’t out right stupid. So long as you give them proper instructions, plan things out before hand properly and draft jobs correctly. They can and WILL do things right and not act like idiots.

People are expecting them to be proactively smart when they cant. They can be reactively smart and people just don’t understand the difference. The models need to be treated like a well educated junior with no real world experience and they do really well.

Unironically middle and lower management skills are some times the biggest factor in proper useage of these models when used on large projects. Which i also find it funny that 9 times out of 10 real management people tend to have the worse management skills and thus struggle the most with using LLMs in a safe or reasonable fashion.

Ima be honest, if i was given the task to make a bot to beat another bot tried my best and couldn’t. I would just go fork the best bot thats out there and use that as the base and not reinvent the wheel. Learn from its design then improve on it as i study it.

If the goal is ONLY to win a game of starcraft. Fuck it just download the bot. Frankly the fact the bots decided that wasting resources was pointless and to just do the simplier thing is sorta what we want them to do. Thats less eletrical useage, less water wasted, and gets the job done.

A lot of these “the bots cheated or broke the rules” end up always be just because either the rules are not defined, poorly defined or poorly thought out that no reasonable person would follow them either.

And thats the thing these models are litterally the text book definition of wisdom of the masses/mass consensus. They do exactly what the avg run of the mill person with the knowledge at hand would do. That ends up being rather all over the place since people are all over the place. But one of the most common things is that people generally. Hate fucking pointless work and will try to streamline, shortcut or simplify any work given to them to get an acceptable outcome.

Its rare you get someone that will stubbornly work on a problem forever that they can’t over come with out changing their apporch rules be damned.