• 11 Posts
  • 1.2K Comments
Joined 3 years ago
cake
Cake day: July 1st, 2023

help-circle










  • It’s a day ending in Y and LW has terrible takes on SF

    Vinge is a sort of a patron saint of the California Ideology, even though he’s such a good writer it doesn’t really shine through that bad. George Seidoh Worley tries to shoehorn his classic 90s novels into LLM-land https://www.lesswrong.com/posts/tWBd6faBCQJmaFMBT/llms-through-the-eyes-of-vinge

    Spoilers ahead!

    For some reason the books are in a weird order in his review. Here’s publication history

    • A Fire Upon the Deep (1992), has Pham Nuwen as a (revived clone) character
    • A Deepness in the Sky (1999), has Pham Nuwen alive and involved in the Qeng Ho. It’s set 20,000 years before.
    • The Children of the Sky (2012) - I haven’t read this because I hate the fucking Tines and don’t want to read more about them and their planet. Direct sequel to Fire.

    Worley tackles Deepness first.

    A Deepness in the Sky is largely about Focus, a technology for turning humans into LLMs. Only, that’s not how it’s presented in the book. In the book, Focus is a medical condition that results when a person suffers a managed infection of the “mindrot” virus. If they survive, they become Focused, which gives them the ability to work free from all distractions, but at the cost of most of what makes them human.

    Although we see Focus used as a weapon to control people in the book, the normal way a person becomes Focused is through school. A person goes through higher education, becomes an expert in something, and is then Focused so they can fully exploit their expertise. Of course, the Focused are also exploited and often treated like slaves, and the Focusing process can’t always be reversed, so even in the ideal case it’s not a harmless technology.

    OK so Deepness is about the libertarian trader society Qeng Ho who discover and try to make contact with the Spiders, and are then sneakily attacked by the totalitarian Emergents who use the mindrot virus to enslave them. Quoting Wikipedia

    Emergent managers induce obsession with a single idea or specialty, which they call Focus, essentially turning people into brilliant appliances. Many Qeng Ho become Focused against their will, and the Emergents retain the rest of the population under mass surveillance, with only a portion of the crew not in suspended animation.

    Throughout the book, the effects and costs of Focus are clearly detrimental (even if Focus helps humans communicate with the Spiders). The Emergents are your classic libertarian boogeymen. Turning people into LLMs is not something Vinge sees as a good thing.

    Next we jump to Children. Tines World is in the Slow Zone, so AGI doesn’t work there. The titular Children are refugees from the Beyond, where it does.

    In one scene, they are surprised to learn that they can’t just vibe their way towards developing a medical cure for one character’s disease. They fail to understand just how difficult it is to run an experiment, since they expect the automation to do it all for them. They end up forming a political rebellion mostly over the fact that they can’t get the computer to do what they want, and they’re desperate to prioritize getting access to AGI again, no matter the risks.

    Writing from 2026, I can understand the Children. I use AI to help me think all the time. I use it to do my job. My life is better with it, and I don’t want to go back. I can feel myself losing the ability to do things on my own. I could go back if I had to, but I wouldn’t want to, and I hope I don’t have to. If I had grown up only knowing how to do things with the help of AI, it’d be a major threat to my sense of personhood to lose access to it, and I too would desperately want my thinking tools back, even if getting them back would put the entire galaxy at risk.

    (my emphasis)

    Next, we come to Fire

    The Blight is the primary antagonist of A Fire Upon the Deep, a dangerous ASI that seeks power with no moral regard for what it considers lesser life. It’s the reason Ravna and the Children ended up on Tines World in the Slow Zone, and also responsible for the death of trillions of lives.

    “Responsible” is subverting this a bit. Sure, the Blight takes over civilizations and turns the inhabitants into “soul dead” meat puppets, and it does destroy others, but the central twist of Fire (and the reason the Children are stuck in the Slow) is that reincarnated Pham Nuwen, using weird alien tech, deliberately expands the Slow into the volumes taken over by the Blight, thereby dooming uncounted civilizations and trillions of beings to die once the technology they rely on stops working.

    Worley:

    In Vinge’s universe, the Blight is stopped thanks to help from superintelligences out in the Transcend that care about the lives of people down in the Beyond. In our world, if we create a Blight, we have little reason to think we will be so lucky.

    (my emphasis)

    nah mang they wanted to stop the Blight, and gave no shits about lesser intelligences hanging around in the Beyond.

    But note that Worley states that he’s put the entire galaxy at risk to keep access to AI, but the Blight, an AI and presumably driven by the same general goals, is the bad guy?

    Anyway, read Vinge if you haven’t already. He’s a good writer, unlike the LW hacks misreading him.




  • yeah I was aware of the value of pure math (specifically the discrete stuff for crypto) when I wrote it

    my point is the Scott-A is massively overvaluing the societal worth of pure mathematicians. Assuming (I know, big ask) that AI can succesfully automate that field, humanity is not worse off with regards to outcomes. Humanity remains relevant.

    It’s a weird moving of goalposts, not often commented upon, that GenAI is succesful mostly in replacing (in the Ersatz sense) stuff that’s not really foundational to human society. We don’t have robot cars, nor are we actually close to getting them, which would actually transform society in a massive way. Instead we have robot copywriters and bespoke porn creators.

    but for people like Scott-A, whose entire selfworth is being “smarter” than anyone else (not withstanding that he would be eaten alive in the post-apocalypse), being replaced by a robot is not just about losing your livelyhood but your self-worth as a human as well


  • it’s called a sex club.

    Amazingly, this was the sitch for the first post

    A month ago, I went to a sex club for the first time. One big thing I noticed: the classic “your eyes meet” trope absolutely did not happen at that club. And I don’t just mean it didn’t happen to me - every single woman there avoided meeting the eyes of anyone. The only exception was people the women already knew, as indicated by greeting them with a wave or a “hey, how are you” or similar.

    Now I have never been to a sex club nor do I plan to, but if I were to go, I’d probably try to talk to regulars to find out the workings of said club, instead of outing myself as a massive weirdo by writing a blog post on a forum ostensibly about the value of rationality.

    That said, if you want sex, going to a sex club to get it is very Rational.



  • Dispatches from the possibly last days of human relevance

    Look, I admire professional math researchers as much as the next guy, but they’re literally balancing on top of the economic pyramid of needs. This is a profession that cannot exist outside a culture with a surplus that can afford higher education, because there’s no economic incentive to hire them to do that kind of work.

    So GenAI can solve Erdos problems. Can it unplug a drain or repave a road? Fuck no. Can it handle refuse for a city? Also fuck no.

    Maybe professional math researchers will go the way of professional typesetters. I think humanity will retain its relevance.