Sorry, I was wrong. I think I probably saw it in a blog post where they mentioned creating an AI search engine using SearXNG and Ollama. I don’t see any mention of native Ollama integration in the SearXNG docs

Thanks man, I’ll take a look

I see. Thanks

I agree with your assessment. I was indeed going to run k8s, just hadn’t figured out what you told me. Thanks for that.

And yes, I realised that 10Gbe is just not enough for this stuff. But another commenter told me to look for used threadripper and EPYC boards (which are extremely expensive for me), which gave me the idea to look for older Intel CPU+Motherboard combos. Maybe I’ll have some luck there. I was going to use Talos in a VM with all the GPUs passed through to it.

Specifically because PCIe slots go for a premium on motherboards and CPU architectures. If I didn’t have to worry about PCIe I wouldn’t care about a networked AI cluster. But yes, I accept what you say

Heavily quantized?

I think yes

I have no idea of how to do this but following

I think SearXNG already has AI integration. Not sure how it works though. I don’t think that I would personally use AI for things other than summarising what I search but it is a useful feature to have

If you consider 4 B580s as enterprise, sure I guess

OP, I have been facing the same situation as you in this community recently. This was not the case when I first joined Lemmy but the behaviour around these parts has started to resemble Reddit more and more. But we’ll leave it at that.

I think I have a solution for you if you’re willing to spend $2-$3 a month - set up a VPS and run a Wireguard server on it. Run clients on your devices and the raspberry pi and connect to it.

As for your LAN: from the discussion you linked, it seems that Jellyfin will use the CAs present in the OS trust store. That’s not very hard to do on Linux but I guess if you have to do it on Android you’d have some more trouble. In either case, using a reverse-proxy (I like HAProxy but I use it at work and it might be more enterprise than you need, for beginners Caddy is usually easier) will fix the trouble you’re having with your own CA and self-signed certs.

I am interested in the attack vector you mentioned; could you elaborate on the MITM attack?

Unfortunately, if you don’t have control over your network, you cannot force a DNS server for your devices unless you can set it yourself for every individual client. If I assume that you can do that, then:

1. Set up DNS server on Pi
2. Set up CA on Pi
3. Create root CRT, CSR and server certs from it (bare-minimim setup)
4. Copy over this stuff to Jellyfin image/VM, and copy root cert to clients trust store.
5. Run reverse proxy in front of Jellyfin and configure the correct IP address of the reverse proxy with an A record in your DNS server.
6. Configure reverse-proxy with server/application cert.
7. Use RethinkDNS on Android to pass everything through the wireguard server hosted on the VPS, and set private DNS to the DNS server hosted on the Pi.

I think that should do it. This turned out more complicated than I imagined (it’s more of a brain dump at this point), feel free to ask if it is overwhelming.

I see. Thanks

I see. That solves a lot of the headaches I imagined I would have. Thank you so much for clearing that up

Thanks, but will NPUs integrated along with the CPU ever match the performance of a discrete GPU?

Thanks for the tip on x670, I’ll take a look

Thanks for the comment. I don’t want to use a networked distributed cluster for AI if I can help it. I’m looking at other options and maybe I’ll find something

Your point is valid. Originally I was looking for deals on cheap CPU + Motherboard combos that will offer me a lot of PCIe and won’t be very expensive, but I couldn’t find anything good for EPYC. I am now looking for used supermicro motherboards and maybe I can get something I like. I don’t want to do networking for this project either but it was the only idea I could think of a few hours back

I’d prefer that you reply with examples/an explanation of what I’m doing wrong instead of cursing

Thanks

I see. I must be doing something wrong because the only ones I found were over $1000 on eBay. Do you have any tips/favoured listings?