Unfortunately, 8GB of VRAM is not going to be enough to run the really interesting stuff. You can do some light image generation, TTS / STT (with Chatterbox and Whisper respectively) and as far as LLMs are concerned, you’re looking at Qwen2.5-7B or similar.
The real question on that box is what kind of CPU do you have? 64 gigs of system RAM, you are looking at CPU inference, which is definitely slower than GPU accelerated, but can be workable for Qwen 3.8 or Muse Glimmer. OpenWebUI is a good starting point to have a chat bot. The real question is what are you hoping to be able to do with a local model?
Unfortunately, 8GB of VRAM is not going to be enough to run the really interesting stuff. You can do some light image generation, TTS / STT (with Chatterbox and Whisper respectively) and as far as LLMs are concerned, you’re looking at Qwen2.5-7B or similar.
The real question on that box is what kind of CPU do you have? 64 gigs of system RAM, you are looking at CPU inference, which is definitely slower than GPU accelerated, but can be workable for Qwen 3.8 or Muse Glimmer. OpenWebUI is a good starting point to have a chat bot. The real question is what are you hoping to be able to do with a local model?
CPU is i7-12700