I’m new to local llms, i’m running Debian on this machine. Where should I start and what model(s) could I get testing on it?

  • Spite_By_Design_LLC@lemmy.dbzer0.comEnglish
    0·
    19 hours ago

    Depends what your interests are! I can get around 35ish tok/s TG at - c 48,000 of Qwen 3.6-35B-A3B (Q4_K_M) using polystrata on a similar rig (you’ve got an A1000, I have a T1000).

    Which means absolutely nothing to you right now but suffice to say - Polystrata + Qwen 35B is something to try. It’s among the “smartest” you can run on your rig.

    Personally, for every day use, I like Ling 3.0 Tiny at Q6 or SharpMiniCPM5-2B at Q8. They are stupidly fast, and supported by MCP tools and a bridge I wrote, work as excellent “hey, I’m thinking of X. Can you tell me about it? And what about Y and Z?” or “How does this read?”

    Think of it like a Google in your pocket, except off line (unless you tell it to go online). Which technically I suppose makes it agentic lol.

    (I can show you some examples if you’d like. I keep a set list of 8 tasks I use for benchmarking as I tweak things, as well as some off the wall / out of band queries)

    Being that Ling or CPM only take up a sliver of the GPU, I can run them concurrently with other models (like an image generator, text to speech, vision etc) to create a really useful offline house brain, that can roam across my local documents, offline wikipedia (Kiwix) and the like.

    It’s the combination of things that results in a 1+1=3

    Having said that, my AI ladder is

    • 90% Ling or minicpm5
    • 9% Qwen 3.6-35B
    • 1% GLM5.3 Flash via open router… which costs fractions of a cent for what I need (there are tricks to this but it’s probably out of scope to info dump it on you at this stage).

    For now, I keep a $20 ChatGPT Plus (separate pools for chat and Codex have been so good) but I am ditching that asap, with the changes coming this month.

    HTH. Write back if you need anything else. It’s nice to see someone else rocking an A1000 / T1000. Your rig should be significantly faster than mine… which is why the A1000 is still stupidly expensive for a 50W card.

  • joseki@lemmy.zipEnglish
    0·
    1 day ago

    I’d sell that hardware and use what you get for it to prepay a DeepSeek API key

  • BeefAndPoultry@lemmus.orgEnglish
    0·
    1 day ago

    I have 8GB VRAM and 32GB RAM, mostly using llama.cpp to run Tiel 35b at Q4, it’s ok at writing code in an agent as long as it can compile and run tests to iteratively fix its mistakes.

    Though there’s been new optimized engines, like Strata, that could maybe allow you to run Qwen Flash Next at decent speeds.

    There’s also Qwen 4 coming out soon that may give some good new options for us VRAM poors. Just stick to MoE models.

  • The Bard in Green@lemmy.starlightkel.xyzEnglish
    0·
    1 day ago

    Unfortunately, 8GB of VRAM is not going to be enough to run the really interesting stuff. You can do some light image generation, TTS / STT (with Chatterbox and Whisper respectively) and as far as LLMs are concerned, you’re looking at Qwen2.5-7B or similar.

    The real question on that box is what kind of CPU do you have? 64 gigs of system RAM, you are looking at CPU inference, which is definitely slower than GPU accelerated, but can be workable for Qwen 3.8 or Muse Glimmer. OpenWebUI is a good starting point to have a chat bot. The real question is what are you hoping to be able to do with a local model?

    • BeatTakeshi@lemmy.worldOPEnglish
      0·
      18 hours ago

      Many thanks, these all in one solutions really interest me, I don’t think I’d have the time to learn how to deploy and connected all individual elements of a full stack myself

  • fozid@lem.radiantfig.fyiEnglish
    0·
    1 day ago

    Depends what workload you expect to use it for. Different models for different tasks. Personally for agentic coding on an 8gb GPU I currently use either Groot 19b a3b, cyber tiel 35b a3b or orinth1.5 35b a3b, all in q4_k_m running with llama.cpp with context of 130k. Get about 30 tokens per second. My own benchmarking puts Groot just slightly ahead of sonnet 5, not quite 5.5 though.

    • BeatTakeshi@lemmy.worldOPEnglish
      0·
      1 day ago

      Thank you. No conding and no setting agents. Really about task logging, summarising documents, some image processing or generating, asking questions, bouncing ideas…

      • fozid@lem.radiantfig.fyiEnglish
        0·
        1 day ago

        Ok for image generation, I recommend flux 2 klein 4b and run it with stable diffusion which provides both a cli and a web UI. You can also run it through comfyui instead, but after trying it out I wasn’t a massive fan.

        For the rest I’m confused, you said no to agentic and coding, but then mentioned task logging and summarising documents, which depending on the tasks and documents, can benefit from agentic and coding? You won’t be running a sota model, so the llm won’t know everything. It will need to be honest about what it does and doesn’t know, and be able to research using the internet, then if you want it to produce a document in a folder summarising or reporting for you, this is all agentic stuff, you need to setup MCP tooling.

        Genuinely if you want a good general chat bot that can use the internet, if go with lfm2.5 8b a1b on a q4 k m as it will be really fast and good quality. If you want something stronger with the agentic tool calling then orinth 1.5 35b a3b q4 k m.

        • BeatTakeshi@lemmy.worldOPEnglish
          0·
          1 day ago

          Thanks. Sorry i’m not an expert in AI. No coding meaning i don’t intend to do programming, of apps or whatever. No agentic in my limited knowledge is not advanced agents that would do things on its own on conditions, I would always be prompting something. For example for tasks I have a nextcloud server and I use its task app, accessible via caldav for syncing across devices, and I would like the local AI to be able to create a task when prompted, or create a calendar entry. Ideally vocally. But maybe that’s already too much?

          • fozid@lem.radiantfig.fyiEnglish
            0·
            22 hours ago

            Asking an ai to do something for you is essentially agentic usage. All an llm can do is generate text or images or audio or video. The key part is generate. They can’t do anything but generate. As soon as you want an llm to do something, you need to give it a tool to do that thing with. It then needs to be capable of working out how to effectively carry out that task.

            The most basic llm will chat to you based on the knowledge it was trained with. Generally, the bigger the llm the more knowledge it has. This is a basic chat bot, the bigger it is the more knowledge it has.

            After this came reasoning and thinking, where the chat bots got smarter. They still had the same amount of knowledge, but they were able to reason, work stuff out, think about things, before answering your question and prompt.

            Then to do stuff, like create docs, search the web, carry out an action, the llm needed to be given tool access and actually be able to do tool calls to use those tools. Anything like create, edit and delete a file, view a folders contents, create a diary entry, search the internet, all required tools. In fact anything except for basic chat response requires a tool. This is where MCP servers and agentic harnesses come in. But you also need an llm capable of tool calling. Giving an llm access to a coding sandbox enables it to be able to calculate answers for you.

            As an example say you want to get an llm to create a diary event for 30 days before Xmas to buy a present for a friend. The llm needs to first work out what day today is, work when Xmas is, then work out what day 30 days before Xmas is, llm’s are terrible at maths, but with a coding sandbox, it can create a basic calculation to work out the date to create the diary event for. It would then call a tool to create the diary event on that day.

            Adding voice recognition is another model that takes your voice commands, converts it to text, then passes it to the next llm that does stuff.

            Ai isn’t magic. You have to build and create the stack that does what you want. But ai can help you do that.

      • BeatTakeshi@lemmy.worldOPEnglish
        0·
        1 day ago

        Thanks. The part where it says it has 72 tensor cores for ml, does it improve anything on the limited vram?

        • ffhein@lemmy.worldEnglish
          0·
          24 hours ago

          Just to clear up some potential misunderstanding… Ampere is the Nvidia GPU architecture, and your card is in the same category as RTX30xx. This can matter when it comes to hardware support for llama.cpp and other software.

          Having tensor cores affects the speed your card does certain calculations, but AFAIK there are different types of tensor cores in GPUs and if you’re using quantized models (which you most likely will be, unless you’re training your own models) then it’s INT8 performance that matters.

          Your amount of VRAM determines how large (which usually correlates to how “intelligent” the model is) model you can fully load on the GPU. If the entire model doesn’t fit in VRAM it can use CPU RAM swapping (partial offloading) which severely harms performance, though MoE models are less affected.

          So both VRAM and tensor cores affect generation speed, but in completely different ways.