• 0 Posts
  • 5 Comments
Joined 1 year ago
cake
Cake day: June 21st, 2023

help-circle
  • This was a real bummer for anyone interested in running local LLMs. Memory bandwidth is the limiting factor for performance in inference, and the Mac unified memory architecture is one of the relatively cheaper ways to get a lot of memory rather than buying a specialist AI GPU for $5-10k. I was planning to upgrade the memory a bit further than normal on my next MBP upgrade in order to experiment with AI, but now I’m questioning whether the pro chip will be fast enough to be useful.



  • Makes sense. I think you’d be fine with pretty much any modern(post DDR4) motherboard/CPU combo these days. I feel Linux hardware support is only really shakey if you’re using a SoC without upstream patches or if you’re using brand new hardware/laptops. With that being said if you’re running a lot of containers on one host have you looked into docker compose or kubernetes(k8s)? Maybe k8s is overkill for home use, but both offer support to restart containers if a health check fails. With k8s you also can spread out containers across multiple physical node, so you could just add a second RPI and “double” your resources.


  • For $350-500 you could easily get a used desktop and processor with 16-32 gb ddr4. But it sort of depends on your home lab goals and workloads. Do you need a lot of storage? Are you CPU bound or memory bound? Some people will suggest used Dell/HP servers, and they’ll look affordable, but keep in mind enterprise gear will eat power and is usually loud. Personally I’d go for a used AMD 5800 or 5900 processor and mobo, install your favorite Linux, and call it a day. AMD processors don’t have quick sync which makes them slightly worse for plex hosting but better for everything else.