alot of really small SLMs seem to be too small to even function properly.
however, there may be some unconventional uses for some of them.
one usecase keyword generation. heres what gemma-3-270m-it-qat-Q4_0 gave me
prompt: keywords similar to “construction”
Construction activities
* Construction design
* Construction equipment
* Construction management
* Construction materials
* Construction methodology
* Construction organization
* Construction planning
* Construction process
* Construction project
* Construction project details
* Construction project management
* Construction project management activities
* Construction project management assessment
* Construction project management checklist
* Construction project management document
* Construction project management evaluation
* Construction project management framework
* Construction project management methodology
* Construction project management plan
* Construction project management project
* Construction project management report
* Construction project management review
* Construction project management schedule
* Construction project management software
* Construction project management strategy
* Construction project management system
* Construction project management tool
* Construction project planning
* Construction project scope
* Construction schedule
* Construction strategy
* Construction work
```
90 TPS (but got stuck in a repeating loop)
i think theres alot of other uses. but they arent really LLM answer that people usually reach for an LLM.
I don’t think these are “keywords similar to construction”… That’d be something like “building”, “contractor”, “builder”, “construction company”, “remodeling”, “roadworks”, …
And your model didn’t really find any of them. I think that output is next to worthless. If you’re a construction company and use this list for your SEO, nobody is ever going to find out you have an excavator and hire you.
well different prompts, at different temps, diff fine tunes , diff quants and quant quality, rag, and lora , are all going to give different results. this is just a single prompt using the model as it is pre-packaged.
you would have to test by running ~100 prompt variations and determine A)no useful value B) unidentifiably wrong answers (primarily if your requirements are unforgiving) C) no predictable/reliable performance whatsoever.
if thats turns out to be the case and its unacceptable…
A) scale up the model size slightly B) spend cash on hardware or service to run the appropriately sized model that accomodates the scale that you need(which could be prohibitively expensive).
Sure. I guess what I wanted to say is: Your example doesn’t really support your claim. It kinda goes the opposite way. (But doesn’t prove that, either.)
I think it’s a bit of a double edged sword. Efficiency and small models are great. We also have some NLP tasks which don’t really require big LLMs. But once you try to use AI, apply it to some real world problems, or embed it into some context… you’ll notice performance issues really add up. I tried to use some 2B parameter models, spent some time tuning everything, built more logic and failsafes into the framework. But at the end of the day I scrapped that hobby project, because the model kept being very limited in its abilities. And issues kept piling up to the point it was unusable.
But keyword extraction is way easier. I don’t think we need LLM inference for that. There are several NLP approaches to do it without a lot of compute. They’re probably way better and more efficient than asking a chatbot to generate some.
im not aware of any nlp tools that generate new keywords(without attaching to an existing database).
i’m not advertising that all LLMs do what mainstream users expect it to do.
there are many applications, just for this single example. for other uses or the exact application… you need to figure that out yourself.
it’s true that larger modern LLMs are cool. but this post isnt about that.