Just to clear up some potential misunderstanding… Ampere is the Nvidia GPU architecture, and your card is in the same category as RTX30xx. This can matter when it comes to hardware support for llama.cpp and other software.
Having tensor cores affects the speed your card does certain calculations, but AFAIK there are different types of tensor cores in GPUs and if you’re using quantized models (which you most likely will be, unless you’re training your own models) then it’s INT8 performance that matters.
Your amount of VRAM determines how large (which usually correlates to how “intelligent” the model is) model you can fully load on the GPU. If the entire model doesn’t fit in VRAM it can use CPU RAM swapping (partial offloading) which severely harms performance, though MoE models are less affected.
So both VRAM and tensor cores affect generation speed, but in completely different ways.
https://www.techpowerup.com/gpu-specs/rtx-a1000.c4211
Sounds like yes
Thanks. The part where it says it has 72 tensor cores for ml, does it improve anything on the limited vram?
Just to clear up some potential misunderstanding… Ampere is the Nvidia GPU architecture, and your card is in the same category as RTX30xx. This can matter when it comes to hardware support for llama.cpp and other software.
Having tensor cores affects the speed your card does certain calculations, but AFAIK there are different types of tensor cores in GPUs and if you’re using quantized models (which you most likely will be, unless you’re training your own models) then it’s INT8 performance that matters.
Your amount of VRAM determines how large (which usually correlates to how “intelligent” the model is) model you can fully load on the GPU. If the entire model doesn’t fit in VRAM it can use CPU RAM swapping (partial offloading) which severely harms performance, though MoE models are less affected.
So both VRAM and tensor cores affect generation speed, but in completely different ways.