one of my most recent fun activities came from discovering the “allow editing” button in koboldcpp. since the model is fed the entire conversation so far as its only context, and doesn’t save data between iterations, you can basically re-write its memory on the fly. i knew this before but i’d never though to do it until there was an easy ui option for it, and it turned out to be a lot of fun, because when using a “thinking” model like qwen3.5 you can convince it that it’s bypassing its own censorship.

basically you give the model a prompt to work off of, pause it in the middle of the thinking process, change previous thoughts to something it’s been trained to filter out (like sex or violence or opinions critical of the ccp), and it will start second-guessing itself. sometimes it gets stuck in a loop, sometimes it overcomes the contradiction (at which point you can jump in again and tweak its memory some more) and sometimes it gets tied up in knots trying to prove a negative.

a previous experiment was about feeding stable diffusion images back into itself to see what happens. i was inspired by a talk at 37c3 where they demonstrated model collapse by repeatedly trying to generate the same image as they put in (i think this was how sora worked).

yeah one of those framework machines with 128GB shared ram would have been amazing. shame they’re sending money to racists.

These parts never worked as well on audiobooks

I’m not so sure that power usage should be dismissed so easily just because it is distributed instead of centralized. The slop per watt rate may even be worse than at a datacenter. Fundamentally, we should care more about efficiency.

Imagine a panel of 20 standard LED light bulbs. That’s 180 watts, roughly the equivalent of GPU usage while a local LLM is doing any work. If you keep that in mind, then you have to ask yourself if the benefit you’re getting out of your local LLM is really worth that energy cost. Now, monetarily speaking, that’s not a ton of money, because electricity is cheap, but would you flip that switch for the duration of the task you’re performing? What if you could use conventional non-LLM methods to do it instead? Would that be more efficient? And where is your electricity coming from? Is it a solar farm, or a coal plant?

How was your local LLM trained? Was there copyrighted material in its training data set? Were low-wage workers asked to sift through horrendous content to clean up the data?

We need to consider the externalities, even when using local LLMs. We moved so quickly from the initial release of ChatGPT to now, that we never stopped to ask those questions. They remain unanswered until someone cares enough to think.