fubarx@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 22 days agoI built a private Al mini-cluster with Framework Desktopwww.youtube.comexternal-linkmessage-square4linkfedilinkarrow-up119arrow-down15
arrow-up114arrow-down1external-linkI built a private Al mini-cluster with Framework Desktopwww.youtube.comfubarx@lemmy.world to LocalLLaMA@sh.itjust.worksEnglish · 22 days agomessage-square4linkfedilink
minus-squareheavy@sh.itjust.workslinkfedilinkEnglisharrow-up9·22 days agoTLDW: Tried huge open models and got 0.7 tokens/s. Seems clustering tools aren’t ready yet.
minus-squarehumanspiral@lemmy.calinkfedilinkEnglisharrow-up3·22 days agoI’m new to this, but some advice seems to be use vulkan to get stuff to work. in your spare time one day, look at rocm.
minus-squareheavy@sh.itjust.workslinkfedilinkEnglisharrow-up3·22 days agoI appreciate that! I’m just trying to recap the authors results for people that might be interested in the bottom line. It seems that the beowulf architecture still has too much overhead
TLDW: Tried huge open models and got 0.7 tokens/s. Seems clustering tools aren’t ready yet.
I’m new to this, but some advice seems to be use vulkan to get stuff to work. in your spare time one day, look at rocm.
I appreciate that! I’m just trying to recap the authors results for people that might be interested in the bottom line.
It seems that the beowulf architecture still has too much overhead