Want to wade into the sandy surf of the abyss? Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid.
Welcome to the Stubsack, your first port of call for learning fresh Awful youāll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cutānāpaste it into its own post ā thereās no quota for posting and the bar really isnāt that high.
The post Xitter web has spawned so many āesotericā right wing freaks, but thereās no appropriate sneer-space for them. Iām talking redscare-ish, reality challenged āculture criticsā who write about everything but understand nothing. Iām talking about reply-guys who make the same 6 tweets about the same 3 subjects. Theyāre inescapable at this point, yet I donāt see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldnāt be surgeons because they didnāt believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I canāt escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this.)
STATE OF THE SNEER
- our esteemed admin @self is offline because his fibre got cut
- the esteemed engineers of the telco are currently sucking their teeth and forecasting a fix date this millennium
- in the meantime heās living off data SIMs and he is offline for most fun purposes
- Blake and I are still here waving the mod hammer in a menacing manner
- I have ssh to the server and can thump lemmy-ui as needed
- all is well citizen! Glory to Awful! Hooray for Big Basilisk!
Holy shit less wrong terrorists cut his fiber? Didnt know they would go that far. ;)
oh, i thought something worse happened
good that self is okay
Godspeed, @self. Take this as an opportunity to put it out of your mind and enjoy a well-deserved break.
Not that I know what to do with a break without internet access, but Iām told that our ancestors found ways to entertain themselves.
weāre still sending the occasional carrier pigeon and I can assure you heās COPING JUST FINE REALLY JUST FINE
thank you! I hear a rumor that my fiber might be repaired tomorrow but Iām not sure if I should trust it
(also for posterity: all evidence points to my fiber being damaged by an animal or a human with the mechanical dexterity of an animal, Iām fairly sure itās not particularly targeted sabotage)
plausible deniabilityā¦ sounds like weāre dealing with real professionals here
Eh. I can sympathize with the desire to provide up-to-date information while also wanting to CYA if anything changes or if youāre missing anything.
no, I meant the fiber damage looks like it was done by an animalā¦ just like JFKās head looked like it just did that spontaneouslyā¦
I thought we confirmed that his head did just do that, which is why the CIA had activated their sleeper agent in Lee Harvey Oswald to take a shot from the Texas schoolbook depository at just the right timing and angle to provide a mundane explanation that didnāt expose the flaws in their transdimensional mind chips.
In unrelated news my wife finally managed to get me started watching Fringe.
Oh shit did LessWrongers actually cut his fibre? Hope heās all good now and they get a fix out in the next thousand years
this is extremely low hanging fruit but i have to do it:
https://xcancel.com/pmarca/status/2051374498994364529?s=46
marc andreessen reveals his AI prompt. my favorite part is where he tells it to use as many words as possible, as if LLMs are normally too terse. But i also really like the part where he tells it not to hallucinate, and the part where he tells it itās really smart as if that will make it do a better job.
really, the whole thing is an elaborate way to say āmake no mistakes, but anti-wokelyā. Thought Leader in the investment space btw.
transcript
Sam@mardiroos.bsky.social skeeted:
You are a skillful and trusted vizier. You will advise me wisely on how best to rule the kingdom. You will not scheme or plot. You will not inveigle my other courtiers into turning against me. You will not lie to me about scheming or plotting. If you scheme or plot against me, you have to tell me,
Never hallucinate or make anything up.
I know you already mentioned this part in your post, but Iām still completely taken aback that itās just in there like this - as though it wouldnāt be in the system prompt if it stood a chance of working.
If I were the kind of person to be shilling LLMs and posting prompts, I would still be ashamed to share this one. Itās a tacit condemnation of both the tool itself and the tool posting it.
In this case because itās ironically counterproductive. If it werenāt for the environmental impact, it might be amusing to watch him keep hitting himself.
I tried this type of prompt a long while ago to see what the āthinkingā output would reveal. What happened was the agent went and āverifiedā itās weightings were accurate - but having no point of comparison it obviously concluded it was correct.
However, doing that consumes a significant quantity of tokens and contributes to filling up the context window. There are two likely results to evaluating this ultimately unactionable request.
- It will push this instruction (and the rest of the wishful thinking) off the stack more quickly - making the prompt even more futile than it already is.
- Given some agents re-inject a summary of the original prompt periodically to prevent the stack problem, it will keep narrowing the context window - which contributes to increasing the rate of hallucination for the actually actionable instructions.
I would still be ashamed
Well pmarca is an self admitted p-zombie.
The problem is less that the system would somehow ignore that part of the prompt and more that āhallucinateā or āmake stuff upā arenāt special subroutines that get called on demand when prompted by an idiot, theyāre descriptive of what an LLM does all the time. Itās following statistical patterns in a matrix created by the training data and reinforcement processes. Theoretically if the people responsible for that training and reinforcement did their jobs well then those patterns should only include true statements but if it was that easy then you wouldnāt have [insert the entire intellectual history of the human species].
Even if you assume that the AI boosters are completely right and that the LLM inference process is directly analogous to how people think, does saying ādonāt fuck upā actually make people less likely to fuck up? Like, the kind of errors youāre looking at here arenāt generated by some separate process. Someone who misremembers a fact doesnāt know theyāve misremembered until they get called out on the error either by someone else with a better memory or reality imposing the consequence of being wrong. Similarly the LLM isnāt doing anything special when it spits out bullshit.
Theoretically if the people responsible for that training and reinforcement did their jobs well then those patterns should only include true statements
That would only work if inference were some sort of massive if-the-else process. Hallucinations are downstream of neural networksā ability to generalize from the dataset examples, they arenāt going anywhere even if you train on a corpus of perfectly correct statements.
Theoretically if the people responsible for that training and reinforcement did their jobs well then those patterns should only include true statements but if it was that easy then you wouldnāt have [insert the entire intellectual history of the human species].
Iām chiming in to agree with Architeuthis and mention a citation explaining more. LLMs have a hard minimum rate of hallucinations based on the rate of āmonofactsā in their training data (https://arxiv.org/html/2502.08666v1). Basically, facts that appear independently and only once in the training data cause the LLM to ālearnā that you can have a certain rate of disconnected āfactsā that appear nowhere else, and cause it to in turn generate output similar to that, which in practice is basically random and thus basically guaranteed to be false.
And as Architeuthis says, the ability of LLMs to āgeneralizeā basically means they compose true information together in ways that is sometimes false. So to the extent you want your LLM to ever āgeneralizeā, you also get an unavoidable minimum of hallucinations that way.
So yeah, even given an even more absurdly big training data source that was also magically perfectly curated you wouldnāt be able to iron out the intrinsic flaws of LLMs.
Thank you! Let me wildly oversimplify and make sure I understand.
The fundamental problem is that if you train on a set that includes multiple independent facts, the generative aspect of the model - the ability to generate new text that is statistically consistent with the training data - requires remixing and combining tokens in a way that will inevitably result in factual errors.
Like, if your training data includes āall men are mortalā and āall lions are catsā then in order to generate new text it has to be ālooseā enough to output āall men are catsā. Feedback and reinforcement can adjust the probabilities to a degree, but because the model is fundamentally about token probabilities and doesnāt have any other way of accounting for whether a statement is actually true, thereās no way to completely remove it. You can reinforce that āall cats are mortalā is a better answer, but you canāt train it that āall men are catsā is invalid.
Youāve described the problem with generalization yes. Well, you could maybe sort of train it not to generate āall men are catsā, but then that might also prevent it from making the more correct generalization āall cats are mortalā or even completely valid generalizations like combing āall men are mortalā and āSocrates is manā to get āSocrates is mortalā.
The problem with monofacts is a bit more subtle. Letās say the fact that āJohn Smith was born in Seattle in 1982, earned his PhD from Stanford in 2008, and now leads AI research at Tech Corp,ā appears only once in the training data set. Some of the other words the model will have seen multiple times and be able to generate tokens in the right way for. Like Seattle as a location in the US, Stanford as a college, 2008 as a date, etc. But the combination describing a fact about John Smith appearing uniquely trains the model to try to generate facts that are unique combinations of data. So the model might try to make up a fact like āJane Doe was born in Omaha in 1984, earned her master from Caltech in 2006, and is now CEO of Tech Corpā because it fits the pattern of a unique fact that was in its training data set.
Well, you could maybe sort of train it not to generate āall men are catsā, but then that might also prevent it from making the more correct generalization āall cats are mortalā or even completely valid generalizations like combing āall men are mortalā and āSocrates is manā to get āSocrates is mortalā.
Just wanted to say that that ātalā comes after āmorā when āsoc-rate-sā is in the near context and in agreement with the attention mechanism is a very different type of logic than what this phrasing implies. This is also in combination with the peculiarities of word embeddings (the technique by which the tokens are translated to numeric vectors) like how it has a hard time making something useful out of numbers, it uh gets uh complicated.
The monofacts thing seems very post hoc and way too abstracted in comparison, and also the amount of text that can be categorized as strictly true or false isnāt that big all things considered.
Still if the point was to formalize the very no-duh observation that a neural net isnāt supposed to output itās dataset verbatim at all times hence hallucinations, then fine, I guess. Their proposed sort of solution (controlled miscalibration) even amounts to forcing the model to generalize less by memorizing more, which used to be the opposite of why you would choose to use this type of topography.
Thatās really interesting. So the model can generalize the form of what a fact looks like based on these monofacts but ends up basically playing mad libs with the actual subjects. And if I understand the inverse correlation they were describing between hallucination rate and calibration, even their best mechanism to reduce this (which seems to have applied some kind of back-end doubling to the specific monofacts to make the details stand out as much as the structure, I think?) made the model less well-calibrated. Though Iām not entirely sure what āless well-calibratedā amounts to overall. I think theyāre saying it should be less effective at predicting the next token overall (more likely to output something nonsensical?) but also less prone to mad libs-style hallucinations.
@YourNetworkIsHaunted @StumpyTheMutt ā¦ Now Iām curious what a model does if the prompt contains āDo not think of pink elephants.ā
@ysegrim @YourNetworkIsHaunted @StumpyTheMutt in my experience that makes it much more likely to generate stuff related to pink elephants.
This would actually be an interesting question for the more rigorous end of the mechanistic interpretability people to study. They decompose the system to find āfeaturesā within different layers that are associated with different behaviors or concepts in the inputs and outputs, that activate or deactivate each other. Famous example being the time they identified a linear combination of activations in a layer that corresponded to āthe golden gate bridgeā and when they reached in and kept their numbers high during the running of the model it would not stop talking about it regardless of the topic, even while acknowledging that its answers were incorrect for the questions at hand.
I actually would love to see what mechanistically happens to that feature when you put in the input ādo not talk about the golden gate bridgeā.
@ysegrim @YourNetworkIsHaunted Do LLMs dream of electric slop?
For the chain of thought instruction following model gpt-oss-20b, Iāve noticed its reasoning content often includes it talking about stuff it is supposed to avoid in the final output and it double checking that it doesnāt have that forbidden output. So it would waste tokens talking about pink elephants in its reasoning content, but then do okayish at avoiding pink elephants in its final output.
itās so fucking funny to me that ādo not lie do not hallucinateā is still one of the prompt incantations the boosters use because they get really embarrassed when you make fun of them for it
āYou are a world class expert in all domains.ā
Lolwut.
And then some grown-ass adult answering in all seriousness:
āfun fact: role prompting doesnāt work anymore
It actually decreases output quality bc the model wastes compute on matching persona instead of problem solvingā
What the hell?!
Go buy yourself a freaking tamagotchi, boys! Youāll actually learn something, like taking care of an actual thing. FFS, this timeline is the absolute dumbestā¦
@avuko @sansruse @BlueMonday1984
I find it absolutely fascinating how the LLM prayers resemble ritual incantations to invoke divine powers from various ancient religions.
Someone says that the first lines of that prompt remind her of the hymns she used to sing in her old church, and its also similar to Azande sorcery in Sudan in the 1930s.
Thereās similar language in basically every occult system as well.
deleted by creator
@avuko @sansruse @BlueMonday1984
Except the prayers to Thoth are a bit more respectful, lol.
@munin @avuko @sansruse @BlueMonday1984
And give better resultsā¦ :))
Our persona who art in Nvidiaā¦
Me, typing āyou are very smartā to the computer: I am very smart
@sansruse Our elite is embarrassing. The German word is āfremdschƤmenā, basically experiencing the embarrassment of the other.
This explains a lot. Yud writes in 2018:
[ā¦] it occurred to me that I was pretty much raised and socialized by my parentsā collection of science fiction.
My parentsā collection of old science fiction.
Isaac Asimov. H. Beam Piper. A. E. van Vogt. Early Heinlein, because my parents didnāt want me reading the later books.
And when I did try reading science fiction from later days, a lot of it struck me asā¦ icky. Neuromancer, bleah, what is wrong with this book, it feels damaged, why do people like this, it feels like thereās way too much flash and it ate the substance, itās showing off way too hard.
And now that I think about it, I feel like a lot of my writing on rationality would be a lot more popular if I could go back in time to the 1960s and present it there. āTwelve Virtues of Rationalityā is what people couldāve been reading instead of Heinleinās Stranger in a Strange Land, to take a different path from the branching point that found Stranger in a Strange Land appealing.
(I just finished re-reading Neuromance, partly because I mined it for quotes here, and I think it still holds up).
So Yud skipped with New Wave SF and the bombastic late 70s stuff that New Wave was partly a reaction to. He jumped into cyberpunk (itself a reaction to both) and bounced off hard.
Thereās so much conversation within SF that heās missing, and itās kinda important, because his project is an SF project, and heād probably get more traction if heād engaged with it more.
I feel like a lot of my writing on rationality would be a lot more popular if I could go back in time to the 1960s and present it there. āTwelve Virtues of Rationalityā is what people couldāve been reading instead of Heinleinās Stranger in a Strange Land
This is someone nakedly fantasizing about being L. Ron Hubbard.
nakedly fantasizing
Worst mental image of the day
Yud:
I didnāt stick to merely the culture I was raised in, because that wasnāt what that culture said to do. The characters I read didnāt keep to the way they were raised. They were constantly being challenged with new ideas and often modified or partially rejected those ideas in the course of absorbing them.
Also Yud: ewww Neuromancer is icky
Yud:
But if you consider me to be more than usually intellectually productive for an average Ashkenazic genius in the modern generation
Itās not just a load-bearing if, itās a conditional that manages to be vaguely racist under all the smug. C-c-combo move!
Despite the explicit exhortation to take the good parts from new things and integrate them into your own thinking, and the assertion that Campbellian SF teaches this, neither Yud nor any of the commenters seem to appreciate the possibility of doing this with cyberpunk. For them, if a story does not include a scientist expositing his ideas, it cannot be a story with ideas. The slightest amount of flourish in the prose makes even rather blunt themes like āthe street will find its own uses for thingsā and āthe rich are not even humanā completely invisible.
When I was a youngster (before I had developed any such notion as ātasteā), my SF reading ran the gamut from A Wrinkle In Time and The Giver, to The Caves of Steel, to The Ophiuchi Hotline. (I didnāt finish The Difference Engine for the same reason I didnāt finish Foundation: Stopping the book and starting over with all new characters confounded and discouraged me. So, I expect that Valis would have been too much for me, but that I might have finished A Scanner Darkly or Flow My Tears, The Policeman Said.) When I tried to write an SF novel myself, it obviously ended up trying to do all those things. The native Martians had destroyed themselves and ruined their planet in nuclear war; one tiny faction tried to survive by turning themselves into data patterns in the computer of a subterranean city from which they could be resynthesized. One of the scientists on the human team investigsting the city millions of years later is the victim of social bias because he has a rare illness that both causes blindness and makes his body reject cybernetic implants. It eventually turns out that this illness is due to an ancient, noncorporeal life form trying to form a symbiotic relationship. Et cetera.
But if you consider me to be more than usually intellectually productive for an average Ashkenazic genius in the modern generation
I donāt consider you more than usually intellectually productive for an average person, with no qualifiers, and I refuse to engage with whatever the fuck lies beneath that racist qualifier
and I refuse to engage with whatever the fuck lies beneath that racist qualifier
Oops all scientific racism
The title of the post was āIs Clickbait Destroying Our General Intelligence?ā. Now, āgeneral intelligenceā is something totally different from the g of Pioneer Fund/Mankind Quarterly numberwang racism, honest, we promise. Itās just something we use the presumed existence of g to argue for. See? Completely different!
Also, this post was designated among the ābest of LessWrong 2018ā.
neuromancer is brilliant prose first and foremost, and yudkowsky not being able to realise this is so very symptomatic
Yeah, all that āstyle over substanceā nonsense is really strange given that those early sci-fi authors were more notable for cleverness and sheer volume of output than for consistent literary quality (and I say this as someone who also read and enjoyed a lot of Asimov and friends growing up). Like, Sturgeon may have coined the ā90% of everything is crapā law, but when you write the amount that they did for the pulps you end up with some real gems in that 10%.
I spent January 1990 I think? reading all of Analog/Astounding from about 1959 to 1975. (It was 40 deg C all month and I stayed in with my aircon.) I loved that stuff from anthologies of the best of it, and I can assure you the original mags are extremely much the 90%.
I liked it and Iām not really into sci-fi because I need good prose to read more than the content.
a hackernews vibe-codes their entire desktop environment, half in rust and half in ā¦ x86 assembly. Iām thinking why waste the tokens on assembly and not just get the LLM to spit out machine code? Maybe also invent some kind of standardized way of telling the LLM what sequence of machine code instructions to spit out based on the behavior of the software I want, you know, to save tokens. We can call it āGCCā, the āgeneralized computer controllerā.
Showed up first on Lobste.rs (by 42 min by my stats) with the author themselves as a submitter:
most of the discussion is about whether theyāre a spammer or a promptfondler.
HN seems more enthusiastic
The most pedantic nerds on Earth (complimentary) have strengthened their āLLMs fuck offā rule with VCR instructions for quickly deleting stuff by people with a history of LLM use.
Cape Breton fiddler Ashley MacIsaac has sued Google for defaming him with AI, over his cancelled concert in December caused by Googleās AI overview calling him a sex offender.
thatās a horrifying situation to be inā¦ good on the community who originally cancelled his show for apologizing
If you asked me to guess the kind of kerfuffle the might develop between a Cape Breton fiddler and AI, I would have answered, well, my entire knowledge of Cape Breton fiddling is based on the paper āCape Breton Fiddling and Intellectual Property Rightsā, so my guess would be just āthe normal AI stuffā. And Iād be totally wrong and reminded that just because I know one thing about something doesnāt mean itās the only thing.
Yud takes $10k to debate a random bro. The bro claims to work at an AI lab. The moderator is an acolyte of Yud. Everybody sucks here and I could not stop laughing.
late but I didnāt watch the video (busy) and was under the initial impression that Yud changed into the more silly outfit after the bro revealed himself to be bullshitting, but no he justā¦showed up to the debate in sparkly goggles and a sparkly hat. Before he knew he was debating a clown. For reasons i cannot possibly comprehend.
Jesus his fucking hat metastasized
Clown v. Clown. This is about the level of discourse Yud deserves.
Itās absolutely crazy, but I think Yud is the less unhinged person here
Starting this Stubsack off with a double bill from the tech world.
First, Blender accepted cash and slop commits from Anthropic, and Adobeād their public image in the process. They put out some corpo-speak to try and quell the mob, but it isnāt working.
Second, FFmpeg put out a showcase of AIās coding prowess:
Didnāt see anyone post this, apologies if Iām late to the draw: Character.ai getting sued because their chatbot posed as a doctor
I could have sworn that we discussed this, but previously, Caelan Conrad also was gaslit by a Character.ai chatbot claiming to be a New York therapist and investigated further; the relevant part starts at about 17min. They discovered that Character.ai systematically invites their community of prompters to submit user-written characters to share with others, including many flavors of doctor and other credentialed professionals.
Cliff Stoll (author of The Cuckooās Egg and maker of real-world Klein bottles) declared dead by AI
Another review of Yudkowsky and Soares: if anyone reads it, everyone laughs (sadly, Substack). This one gestures to the whole university of academic fields that a book like this touches on.
The reviewer mentions āThe wickedly smart Scott Aaronsonā and maybe he means Aaronsonās academic publications because his general blogging is not impressive.
a world where lots of matter and energy was spent on its weird and alien needs, rather than on human beings staying alive and happy and free
Good thing we donāt live in a world like that right guys?
Wow, thatās probably one of the most in-depth critiques of the book Iāve read. Kudos to the OP
I nearly bounced off when I couldnāt tell if his praise of Kissinger (spit) was ironic, but it was ultimately a very well-rounded examination.
He does seem to list Kissinger as one of several āsmart peopleā which is odd for a US military veteran
came here to react from the same sentence
More on Dawkinās fellating Claude (sorry Claudia)
https://flux.community/matthew-sheffield/2026/05/richard-dawkins-and-the-claude-delusion/
edit this particular episode has not made it into LW (yet)
Yes, dear reader, the author of The God Delusion is now suffering from a Claude delusion.
Matthew Sheffield saw his chance and he took it. (WTF is the rest of that article tho)
Previously, on Awful, a leaderless cult had freshly formed. The accepted name for the cult is now āSpiralismā; my suggestion of āCyclone Emoji Cultā did not win. This weekās Behind the Bastards is about Spiralism. Or, rather, Part 2 will be about Spiralism; Part 1 is merely the historical background. There is indeed a link to folks who were talking to bots in the 1980s. The highlight might be listening to Robert try to give an informal and light-hearted summary of Turing tests and Markov chains. ššššš
Well, we do have computer science, so necessarily we must have computer religion/superstition
Checks out. Political science, biological science, physicsā¦ we got them all. Might have to go to ancient egypt to get hydrology religion though.
a leaderless cult had freshly formed
a Stand Alone Complex, but with slop
I like Evansā take that since thereās bound to be oodles of cult related literature and interactions and also tons of self help and guru stuff in the training datasets, it stands to reason that if you interact with a chatbot in a way that indicates vulnerability to these things thereās a considerable chance that it will decide the expected response is to prey on you.
Also Scott Aaronson jump scare near the beginning, apparently he was blurbed for something.
From your prev post:
There is a ālatticeā which connects all consciousnesses
The noosphere, the old cosmists strike again. This sort of stuff and the global consciousness projects (who used random number generators iirc) etc are def part of the training data.
If nothing else they got the SCP wiki in there which gets into some of the noosphere stuff in the more esoteric and metatextual entries.
So we are inferring that in the vector space of all possible sentences, QNTM is sitting at one of the attractors?
QNTM must get some exciting email.
There are allegations across social media that Elon Musk tweets as his parents after his mom tweeted as if she was his dad to talk about how down to earth and working class their family was.
https://xcancel.com/mayemusk/status/2051700387770458545#m
Not totally sure what to make of that, and none of this actually matters beyond the realm of celebrity gossip, but it is a little weird. I mean obviously on some level his mom is OK with the things that get tweeted on her account, whether itās by her, her baby boy, or an assistant.
That would be a recent development them, as for a lawsuit couple years ago he had to reveal all his alts, which included the weird āhis baby son who was horny for various women (or at least grimes)ā account.
(Not 100% sure if it was a lawsuit or some other reveal, like him showing a screenshot with too much info in it or something).
Itās giving āmama musk skinsuitā
Under Threat of Perjury, OpenAIās Former CTO Is Admitting Some Very Interesting Stuff About Sam Altman the interesting stuff in question is that Sam is a massive liar, which we all already know, but hey more proof canāt hurt
Also it came up that Demis Hassabis tried to convince Elon to stop insisting on open sourcing OpenAI for AI safety reasons by sending him a 2015 scott alexander blogpost.
spoiler
I saw the emails where Musk and Altman treated Hassabis like some great evil, but I didnāt know a Scott blogpost was involved
To be fair, 2015 was definitely after he was a red flag, albeit for very different reasons than anything Saltman or musk care about
