Need to let loose a primal scream without collecting footnotes first? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid: Welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned soo many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting thisā¦)
Itās the cost of the electricity, not the cost of the GPU!
Empirically, we might estimate that a single training-capable GPU can pull nearly 1 kilowatt; an H100 GPU board is rated for 700W on its own in terms of temperature dissipation and the board pulls more than that when memory is active. I happen to live in the Pacific Northwest near lots of wind, rivers, and solar power, so electricity is barely 18 cents/kilowatt-hour and Iād say that it costs at least a dollar to run such a GPU (at full load) for 6hrs. Also, I estimate that the GPU market is currently offering a 50% discount on average for refurbished/like-new GPUs with about 5yrs of service, and the H100 is about $25k new, so they might depreciate at around $2500/yr. Finally, I picked the H100 because itās around the peak of efficiency for this particular AI season; local inference is going to be more expensive when we do apples-to-apples units like tokens/watt.
In short, with bad napkin arithmetic, an H100 costs at least $4/day to operate while depreciating only $6.85/day or so; operating costs approach or exceed the depreciation rate. This leads to a hot-potato market where reselling the asset is worth more than operating it. In the limit, assets with no depreciation relative to opex are treated like securities, and weāre already seeing multiple groups squatting like dragons upon piles of nVidia products while the cost of renting cloudy H100s has jumped from like $2/hr to $9/hr over the past year. VCs are withdrawing, yes, and theyāre no longer paying the power bills.
in the same vein, I did some (somewhat wildly) speculative analysis around this a while back too
didnāt really try to model āactual workloadā (as in physical, vs the ārented compute timeā aspect), and therein lies an important distinction: actually owning the GPU puts you at a constant minimum burn rate
and as corbin points out wrt power, these are also specialised formfactor devices. and theyāre going to be getting run at close to max util their entire operated lifespan (because of silicon shortage). so even if any do get soldā¦ long mileage
That is substantially worse than I realized. So possibly people could sit on GPUs for years after the bubble pops instead of selling them or using them? (Particularly if the crash means NVIDIA decides to slow how fast the push the bleeding edge on GPU specs so newer ones donāt as radically outperform older ones?)
I mean, who are you going to sell them to? the other bagholders are going to be just as fucked, and itās not like thereās an otherwise massive market for these things
Ultra ultra high end gaming? Okay, looking at the link, 94 GB of GPU memory is probably excessive even for eccentrics cranking the graphics settings all the way up. Hobbyists with way too much money trying to screw around with open weight models even after the bubble bursts? Which would presume LLMs or something similar continue to capture hobbyistsā interests and that smaller models canāt satisfy their interests. Crypto mining with algorithms compatible with GPUs? And cyrpto is its own scam ecosystem, but one that seems to refuse to die permanently.
I think the ultra high end gaming is the closest to a workable market, and even that would require a substantial discount.