So the problem here is that every machine would have to run an instance of some model, you can’t spread the model over all the machines. It’d be too slow. Reading from NVMe on your very own machine is already pretty slow as it is. Kinda limits the model size.
So the problem here is that every machine would have to run an instance of some model, you can’t spread the model over all the machines. It’d be too slow. Reading from NVMe on your very own machine is already pretty slow as it is. Kinda limits the model size.