Depends what your interests are! I can get around 35ish tok/s TG at - c 48,000 of Qwen 3.6-35B-A3B (Q4_K_M) using polystrata on a similar rig (you’ve got an A1000, I have a T1000).
Which means absolutely nothing to you right now but suffice to say - Polystrata + Qwen 35B is something to try. It’s among the “smartest” you can run on your rig.
Personally, for every day use, I like Ling 3.0 Tiny at Q6 or SharpMiniCPM5-2B at Q8. They are stupidly fast, and supported by MCP tools and a bridge I wrote, work as excellent “hey, I’m thinking of X. Can you tell me about it? And what about Y and Z?” or “How does this read?”
Think of it like a Google in your pocket, except off line (unless you tell it to go online). Which technically I suppose makes it agentic lol.
(I can show you some examples if you’d like. I keep a set list of 8 tasks I use for benchmarking as I tweak things, as well as some off the wall / out of band queries)
Being that Ling or CPM only take up a sliver of the GPU, I can run them concurrently with other models (like an image generator, text to speech, vision etc) to create a really useful offline house brain, that can roam across my local documents, offline wikipedia (Kiwix) and the like.
It’s the combination of things that results in a 1+1=3
Having said that, my AI ladder is
90% Ling or minicpm5
9% Qwen 3.6-35B
1% GLM5.3 Flash via open router… which costs fractions of a cent for what I need (there are tricks to this but it’s probably out of scope to info dump it on you at this stage).
For now, I keep a $20 ChatGPT Plus (separate pools for chat and Codex have been so good) but I am ditching that asap, with the changes coming this month.
HTH. Write back if you need anything else. It’s nice to see someone else rocking an A1000 / T1000. Your rig should be significantly faster than mine… which is why the A1000 is still stupidly expensive for a 50W card.
Thanks for the very rich reply. I’m a bit reassured that I can do some basic stuff, I’m not a poweruser anyway, if this word is still used. The aim is to not rely on external tools and actors as much as possible for private stuff. My work does give me access to a commercial gpt for work, but I don’t want to use it for my private life matters. The learning curve seems a bit higher that I thought, but I saw some tools that could maybe help me there, like ollama or localai.io, any thoughts on these projects?
If you want a gentle start, may I suggest Jan.ai? It’s a ride on lawn mower compared to the sports car I suggested…but you need to dip your toe into the water before you go ludicrous speed.
Once you get a feel for that, I’m going to suggest some MCP tools to you as well. I don’t know if Jan finally added MCP servers to their stack (they should have by now!).
The following are … ok. Not the best and not what I run with but again, lawn mower before sports car
Sorry - again: too much info. Start with Jan. You’ll be amazed…and then perplexed…and then frustrated. That’s the time to migrate thru to llama.cpp or PolyStrata and start adding those MCP tools I mentioned above.
Depends what your interests are! I can get around 35ish tok/s TG at - c 48,000 of Qwen 3.6-35B-A3B (Q4_K_M) using polystrata on a similar rig (you’ve got an A1000, I have a T1000).
Which means absolutely nothing to you right now but suffice to say - Polystrata + Qwen 35B is something to try. It’s among the “smartest” you can run on your rig.
Personally, for every day use, I like Ling 3.0 Tiny at Q6 or SharpMiniCPM5-2B at Q8. They are stupidly fast, and supported by MCP tools and a bridge I wrote, work as excellent “hey, I’m thinking of X. Can you tell me about it? And what about Y and Z?” or “How does this read?”
Think of it like a Google in your pocket, except off line (unless you tell it to go online). Which technically I suppose makes it agentic lol.
(I can show you some examples if you’d like. I keep a set list of 8 tasks I use for benchmarking as I tweak things, as well as some off the wall / out of band queries)
Being that Ling or CPM only take up a sliver of the GPU, I can run them concurrently with other models (like an image generator, text to speech, vision etc) to create a really useful offline house brain, that can roam across my local documents, offline wikipedia (Kiwix) and the like.
It’s the combination of things that results in a 1+1=3
Having said that, my AI ladder is
For now, I keep a $20 ChatGPT Plus (separate pools for chat and Codex have been so good) but I am ditching that asap, with the changes coming this month.
HTH. Write back if you need anything else. It’s nice to see someone else rocking an A1000 / T1000. Your rig should be significantly faster than mine… which is why the A1000 is still stupidly expensive for a 50W card.
Thanks for the very rich reply. I’m a bit reassured that I can do some basic stuff, I’m not a poweruser anyway, if this word is still used. The aim is to not rely on external tools and actors as much as possible for private stuff. My work does give me access to a commercial gpt for work, but I don’t want to use it for my private life matters. The learning curve seems a bit higher that I thought, but I saw some tools that could maybe help me there, like ollama or localai.io, any thoughts on these projects?
Friends don’t let friends use Ollama
If you want a gentle start, may I suggest Jan.ai? It’s a ride on lawn mower compared to the sports car I suggested…but you need to dip your toe into the water before you go ludicrous speed.
Once you get a feel for that, I’m going to suggest some MCP tools to you as well. I don’t know if Jan finally added MCP servers to their stack (they should have by now!).
The following are … ok. Not the best and not what I run with but again, lawn mower before sports car
https://github.com/ofershap/mcp-server-markdown
https://github.com/jeffreyrampineda/kiwix-wiki-mcp-server
https://github.com/modelcontextprotocol/servers/tree/main/src/filesystem
https://github.com/exa-labs/exa-mcp-server
https://exa.ai/
https://www.home-assistant.io/integrations/mcp_server/
Sorry - again: too much info. Start with Jan. You’ll be amazed…and then perplexed…and then frustrated. That’s the time to migrate thru to llama.cpp or PolyStrata and start adding those MCP tools I mentioned above.
https://github.com/VecSzn/PolyStrata
https://github.com/srware-net/LocalLightChat/
https://github.com/open-webui/open-webui
Then we can get freaky :) Oh…we have such sights to show you lol
Many thanks! I’ll pin your reply for when I’m down the rabbit hole! Jan looks nice
Enjoy and good luck!