• 1 Post
  • 345 Comments
Joined 3 years ago
cake
Cake day: July 4th, 2023

help-circle
  • So, I 100% agree.

    The key here is there are many assets, like textures, etc, where you want to better convey intent from the design team to the art team.

    And, if you aren’t trying to actually sneak in AI assets and “cheat”, you can actually very easy make the AI assets visually obvious.

    Like just slap a bright purple watermark on them that says “PLACEHOLDER” or etc

    Thats very normal.

    The companies claiming “oh whoopsie that asset accidently made it in” are just blatantly lying.

    If they really wanted to ensure the assets didnt make it into the final build, its very easy to tag files, name them a certain way, watermark them, have a build pipeline that checks, etc to make it super obvious if you accidently included them.

    You should just assume companies pretending this wasn’t possible are just lying. They thought theyd get away with it and got caught and hoped ppl would buy their excuse.


  • I agree for the most part, but keep in mind theres decent chunks of “writing” in video games that are mostly filler and probably no one would notice or can tell if the line is AI vs an intern who wrote it.

    Like… npc one-liners and stuff like that, item descriptions, etc. Stuff where its probably not even gonna be looked at by 99.9% of ppl anyways and the bar for quality is very low.

    Whats critical here us understanding that you dont replace your writers here.

    You just took a boring/monotonous task off their plate so that they can focus time/energy on more important parts of the game


  • For Art:

    Perfectly fine for placeholders for mock ups and helping convey what you want to an artist, acting as a form of interop between non-artist and artist. IE “Id like something like this”, but final art assets should be real ones made by real artists.

    For UI, layout, UX:

    Completely fine for mocking up, much the same as art. You can rapid prototype placeholder layouts to get passed blockers and start developing, and the actual artist can catch up after.

    Coding:

    I work as a software dev, AI given proper tooling and setup codes perfectly fine now, but still needs human review and guidance to keep it on the rails. You still end up with human developers steering the AI, they just do their job faster

    It heavily benefits from “momentum”, the more good approved code in the system, the better the AI is at staying on the rails.

    Theres a faction of devs that think even that can be automated away. I consider people who have gone that far to have drunken the kool-aid. I trust AI output as far as I can throw it.

    Audio:

    I have like zero experience on this, but every sample Ive seen comes across as very poor quality, its barely even good enough for placeholder mocks but maybe there are bespoke tools out there? I haven’t seen any though yet. IE SFX and etc.

    Writing: Honestly perfectly fine like coding, given once again a human is still steering it heavily and you have proper tooling to control it. However since this is “user facing” Id consider it a liability.

    Also afaik AI writing is very prone to issues and being very bland, so I would not use it heavily.

    Editting: However for editting I can already confirm it’s splendid. Its good at sanity checking your work and catching stuff like plot holes, inconsistencies, etc that normal grammar tools don’t cover.

    Its also good at helping catch stuff like using terms/words/phrases too much in ways normal tools wont.

    You still want a real human editor, but the AI can help with initial sanity passes to cover a bunch of the “easy” stuff.

    And its very good at applying the edits. Batch fixing mistakes, etc.

    DevOps: Its splendid. It can pretty quickly do things like convert requirements into technical requirements, rapidly discover negative tests cases, call out edge cases, manage tickets…

    Greatly helps with keeping that stuff organized in teams.


  • Because we need to improve the AI process

    This is what I bring up in my complaints all the damn time when I see this.

    They arent wrong here but they missed an incredibly critical part.

    Did they train you on how you do that?

    Cuz it sounds like they haven’t, because that simple little “improve the process” remark is 99.9% of the goddamn challenge and it takes a tonme of training and practice to know how you do that well.

    You can eventually reach the point this person wants but you need entire libraries of skill files and tools setup to do it.

    Is your entire codebase pre-emptively vectorized for semantic search yet? Im guessing no.

    Do you have a full set of bespoke per-agent toolsets to streamline their capabilities and avoid context bloat?

    Do you have a solid repo of skill files you all share and constantly refine everyday to slowly patch info gaps and correct mistakes the agents make?

    If all of this sounds like gibberish to you, this is the damn problem with companies now.

    I keep saying it like this:

    If you hand all your employees nailguns, tell them all they must use nailguns for their job now, but give them zero training on how to use nailguns, you are gonna have a bad time.


  • Its a bit of both.

    Absolutely, for sure, a decent amount of telemetry is for simply making decisions about what people actually use.

    “Should we improve (thing), or drop it and stop supporting it?”

    Well, lets just track how many people actually use it first for a bit and then decide.

    Youd be surprised how often users beg for features and then stop using them after 1 week lol.

    But sometimes a random feature you thought no one uses much turns out to be actually quite popular.

    This same goes for optimizing. Your highest traffic parts of your website are there you wanna focus the most on stuff being optimized to save money and improve user experience.

    Do a tonne of companies track stuff just to sell it as data for training AI?

    Yeah, they do. And its gross.

    But there is a huge amount of telemetry thats just developers wanting to genuinely improve the user experience, catch bugs, etc.



  • Theres not really any fooling here. Theres tonnes of interesting examples you can find.

    Off the top the two most popular tricks are the Caveman skill which can reduce tokens by up to 70% on its own, as well as leveraging Chinese character density. Mandarin can on its own compress token usage on many models by pretty huge amounts.

    Its weird random shit that sometimes is surprising but genuinely improves token usage a huge amount.

    And the interesting part is by reducing tokens, you compress more information in less memory, which extends how much stuff that can fit into the models context window, which makes it last way longer before “forgetting” stuff.

    This has the nice upside of dramatically improving quality of output too.

    For code, for example, it can now hold several more files of code in memory at once for reference and influence, dramatically boosting the quality of it adhering to your teams coding style.

    Thats just one example you learn on how to make the tool less stupid.

    Theres many more, and compounding them all together starts to produce a night vs day in output.

    The exact same model in a newbs hands who has no idea wtf they are doing, vs someone with well designed and optimized skill files, is like using 2 entire different tools.

    Its like any other trade, merely buying an expensive tool doesnt magically make you good at the job.

    Knowing how to use the tool is way more important


  • Its not complicated. People have become extremely insulated away from what the real work looks like of a dev over the years.

    The reality os starkly contrasted to public perception.

    Most software developers heavily use LLMs now. They sucked 5 years ago, we’re meh 3 years ago, decent 2 years ago, but over the past year and a bit have rapidly become genuinely more efficient when used right and skillfully than doing about 90% of your work load by hand.

    Bits and pieces still require doing it by hand, but the vast majority of work for the average dev now is via moderating an LLM (with skill) to success.

    Unfortunately a fuck tonne of devs lack that “with skill” part still, and what this comes out as is them costing their companies tremendously more money to do the same job.

    A loooot of companies (stupidly) hedged their bets that if they just gave their devs wild west access to using LLMs without training they’d magically just “figure it out” along the way.

    Which is nonsense, why would a dev feel compelled to conserve tokens or improve efficiency with zero incentive?

    So now companies are scrambling as they realize their devs, who just spent 12 months going hog wild with LLMs, still havent learned how to use them well and in dact have developed arguably worse poor habits that they now need to unlearn

    Thats where the industry is at now largely.

    Meanwhile companies like the one I work at predicted this as a natural thing and we’re preparing for it long in advance. When token prices shot up we already had set ourselves up with lots of training so the price increase was not nearly as noticeable.

    I think when Im fully optimized out on a project I only spend about $10~$15 a day, despite going full steam for 5 hrs or so.

    And despite that my productivity is probably higher than unskilled devs who burn through 10x~20x that. I get more work down in way less time and way less tokens.

    Training and the resource/knowledge pool go a long way here. It cannot be understated


  • If my boss comes to tell me that from now on my “productivity” will be measured in token usage rather than actual “production”

    Did your boss actually say this, or are you just going off of some memes you saw and think thats something happening often.

    Most companies care about getting work done in as few tokens as possible now, developers that can achieve the same results but with less tokens are extremely valuable.

    Not only that, but less tokens also inherently means faster.

    Any company that is blowing through tokens, without any effort put into training employees how to use the tools better, deserve to fail.


  • Start by learning all the critical things like Skills, MCP, Agents, etc.

    Then look up skills and MCP tools that reduce token usage, improve recall, improve searching, improve parsing, etc

    Then learn how to use sub agents bound to cheaper models for more expensive operations (the largest of which us always search and find ops)

    Swapping to a cheaper model for a subagent with the job to just go find a specific thing alone can reduce costs like 30%, equipping it with tools that can search and find faster can push that to 70%


  • No, I work in the industry and am vety actively entwined eith systems where we contract out to train and show companies how to use LLMs better.

    And a lot of our clients now are of the “how do we use less tokens” variety, and you walk into the project and see the way they currently operate and go “oh god”

    The average developers have absolutely zero clue wtf they are doing, they’ll burn a million tokens on something that outta take 10k.

    We often can get token usage down easily 90%+ in the first month just by on boarding and offering some basic training and helping install some basic guard rails, skills, etc.



  • Absolutely 100% all of this, though with a lot of other tricks like caveman mode and careful skill files and helper scripts to help the agent quickly surgical extract out just the useful output, you can substantially reduce token burn and improve its memory.

    As well as carefully having it rollback changes everytime a fix doesn’t work, and having ut keep a markdown file log of each fix it tried and the results, so it can review each thing it tried previously.


  • Reproducing the bug with an automated test is harder, its code you can run that tests your other code.

    But allows you to just 1 click run it and get a yes/no “is this still broken” output without having to manually reproduce it by hand each time.

    Whats important is this is in the domain of what LLMs can actually work with, the output of the test is something they can parse and iterate on until it works.

    They execute the command to run the test, check the output, and keep working til the test passes.

    They can add additional tests to help isolate the problem, or strip down the existing test until its doing the absolute bare min steps to reproduce, in order to narrow the scope of whats causing it.

    But when your test involves stuff running in the kernel of an OS, your automated tests meed to effectively be code you write that bootstraps a virtual machine up and manipulates and observes that second machines kernel…

    You can do it, but its one of the most complicated forms of automated tests to design and run!


  • Yeah, LLMs are gonna spin their wheels hard when it comes to testing anything at the kernel/os level, if you dont have automated testing with a virtual machine setup to actually be able to replicate a bug, you 100% just cannot test anything they produce or say

    As soon as you have the ability to go “Okay we have a failing test, make it pass”, the LLMs get a lot less stupid, because instead of just randomly fumbling around and guessing, they have actual feedback to iterate on and can actually chew on it til they fix the issue or give up.




  • Innovating the boring stuff

    There is no innovating it, you simply have to just do it.

    This us a purely logical requirement, the cide is already abstracted to the maximum feasible point.

    You simply have to write the code that connects the output of pipe A to the input of pipe B

    This is called the Domain Rules or Business Rules, its the stuff specific to your apps needs that simply cant be abstracted further.

    If we define for example “This endpoint lets you add a person to a room, but a room cannot have more than 8 people” you cannot get around needing to somehow define this business rules in your logic.

    Even at its absolute most abstract form, its at least a couple lines of code minimum.

    Now, most api endpoints have several rules. And often apis can have hundreds of endpoints. And often businesses maintain multiple apis.

    So, 3 x ~7 x ~100 x ~3 puts you at like 6300 lines of code baseline for defining business rules.

    And then for every. single. rule. You have to write a test that positive and negative tests these rules.

    Which puts us at about 2100 rules, multiple by about easily 10 to 12 lines per test easy.

    So 11 x 2100 = 23,100 lines of code for tests, though its prolly closer to double that.

    ALL of this is extremely simple and easy to do, its just a lot of fuckin typing lol.

    AI can pump this out in about 1/10th the time I can, prolly closer to 1/20th tbh.