yeah enabling the devil to rape in a acceptable manner… I get you need to celebrate small victories, he is just a mayor of the financial capital of the usa but this is just marketing. Hope is useful… I guess but not when it is the devils giving you a hug. I have been where you are sooo I will let you bust your knee… it is the only way you will learn. For I am the lone yank
Can we have ANY sources here at all or are we just posting AI gen shit in antiAI now?
So now you’ll have to pay $200 every time you have an important question you need answered? My PCP just tells me to use over the counter products anyways. Also is this NY state or city? Why is there a picture of Mamdani?
There are no sources for any of this AI generated Facebook bullshit.
AI is pretty good at helping with very straightforward tax and legal questions. Also, can a lawyer use AI for legal advice? Lawyers are already using AI for a ton of stuff. This sounds hamfisted.
Finally.
Modern capitalism derives it’s scarcity via artificial means such as legislation.
The only thing this does is restrict the means of information to the people of NYC. Have you used traditional web search for health/legal/etc. advice? Even pre-2015 if you didn’t know what you’re doing you would get the most unhinged, predatory, incorrect advice imaginable, but legislating away access wasn’t going to make health information more correct, accurate, or equitable. It would’ve just further entrenched the predatory health/legal/etc. industry as the gatekeepers of that knowledge.
This is no different.
Smart GovOS policy keeping human CDMWorlds accountable.
I understand the sentiment but I think this is too far of a pushback. Restricting access to information even though it may be inaccurate or misleading isn’t the right answer. AI is a tool. I’d wager, force a prompt to specifically be visible whenever subjects break that. Essentially the cigerette equivelant “THIS PRODUCT KILLS YOU” but something like “AI RESPONSES TO MEDICAL INFORMATION SHOULD NOT BE TREATED AS ACCURATE. PLEASE CONTACT YOUR CARE PROVIDER.”
I feel like this logic should of banned search engines since they could provide links to sites with misinformation.
In what way is this restricting information? AI usually just summarizes info that was already trivially searchable
AI shouldn’t be an excuse to take a job, it can be a tool but that’s it
New York State not New York City.
I don’t know why Mamdani is pictured. I think it is because the proposal was introduced in the State Senate by Kisten Gonzalez who is a DSA member.
the text in this image is also just flat-out wrong,
the proposed legislation does not “ban AI from answering questions” on these subjects.
https://www.nysenate.gov/newsroom/press-releases/2026/kristen-gonzalez/ny-state-senator-kristen-gonzalez-her-bill-address-ai says it is strictly about impersonating licensed professionals:
We have documented instances of AI chatbots providing false medical license numbers to users and documented cases of real harm from AI chatbots providing misinformation. This is what my common-sense legislation addresses—protecting users from misinformation, scams, and fraud. This legislation does not prohibit a user from asking a chatbot questions or receiving general information and advice, as long as the chatbot is not presenting that information as a licensed professional. It would also not prohibit licensed professionals from consulting AI technology in the course of their work.
Fake news? On my social media??
Fake news? On my social media??
I’m surprised that this post is still up so many hours after i (and i’m sure others) reported it, especially given the large number of moderators that this community has.
This makes way more sense.
Add edibility of wild plants to the list please
Maybe I’m just ignorant about this, but it feels a bit “gatekeepy” to outlaw answering questions about the law itself. Like… so I can’t use an LLM to help do taxes? Or if I’m served a confusing legal document, I have to pay an expensive attorney to understand the legalese? I can’t even just get a rough idea from an LLM first?
EDIT: Also I’ve literally had a professional accountant send me a ChatGPT conversation they had because they didn’t know tax law well enough. I feel like if anything that should be outlawed. Professionals should actually have to know shit and know how to research primary sources.
I would say its to protect people, if you ask a LLM for a legal answer and trust it, then you could get into legal trouble when it turns out to be wrong.
Not only are LLMs wrong like ~1/10th of the time (for the more advanced ones) but law is also an area where there is black and white rules, but also there is interpretation of some aspects.
Some things are set in stone and some are about understanding the intention of the law and being able to back that up in a court if it was challenged.Do you want the public trusting a LLM that says “oh yes you can claim a tax deduction for an Xbox because its for stress relief related to your job” and then you get whacked with a fine for incorrectly filing your tax.
Using a LLM to help your draft an email to your uncle = low risk. Using a LLM to help you decide if you actually need a medical procedure your doctor has reccomended = high risk.
If that was the logic we were using then Wikipedia shouldn’t be allowed to have articles on health and law either. WebMD shouldn’t have ever been legal either.
Obviously, the benefits to having that information available outweigh the hypothetical negatives. Even with them being incorrect, partial and easy to draw invalid conclusions from, having the information be freely and readily accessible is better for everyone.
I think the distinction is that looking up wikipedia or any other website that appears to have topics of information you can learn from is different from asking a tailored answer to your specific query.
One feels like more like research to improve your understanding, but you know you would have to go to a professional to ask specific questions around your circumstances.
And the other feels like your own ‘personal professional’ that answers all your little specific questions, but is really just a complex algorithm trying to make you happy and doesnt know if is telling you right or wrong, only that it shouldnt say it doesnt know.A program that can be very confidently wrong while giving people advice that can mess up their life (healthcare, finiancial, legal advice, etc) could benefit some, but will ruin others make it not worth the risk?
A program that can be very confidently wrong while giving people advice that can mess up their life
Literally WebMD.
I think there’s an assumption that professionals are always correct. They are not. Professionals are also heavily trained to “[try] to make you happy and doesnt know if [they are] telling you right or wrong, only that [they] shouldnt say [they don’t] know.”
You’re also neglecting how bad health care access, and legal etc., is. Especially in countries like the USA where access to healthcare is financially gated and increasing extraction makes “go[ing] to a professional” not possible unless you wait >6 months to go somewhere that’s a >3hr commute from where you live. WebMD, Wikipedia and chatbots can help to decide whether the >$1,000 and 7hr professional consultation is worth it, and if so be more informed going in so that the time is better spent.
That being said, poor healthcare can absolutely have worse outcomes than no healthcare. I do believe that is a minority of situations, however I also lack any hard evidence to really back that up. Do you have any?
WebMD is just a website with information pages on health topics, not a program or custom response, is it not?
I think that there is a future that it may be a reasonable first step for people with restricted access barriers, but at this stage its too much of the wild west to be using it for these higher level cases.
But regardless i dont have specific information to present you, i woild argue the amount of people that have committ suicide after interacting with LLMs rather than seeing a mental health professional would be a fairly relevant example though.
I thought the issue you had was with the information itself. Why does it “being a program” matter? That said, it is; “symptom checker” specifically and “page rank” more generally.
its too much of the wild west
And? There seems to be this assumption that “wild West” == bad/dangerous/scary/harmful. The modern Internet is certainly far less of the “wild West” it used to be. I would also argue it is also far more dangerous, harmful and nefarious than the “wild West” web.
the amount of people that have committ suicide after interacting with LLMs rather than seeing a mental health professional would be a fairly relevant example though.
Anecdotes aren’t useless, but they need to be contextualized. Is it higher than the baseline? Is it greater then those who didn’t commit suicide because when the mental health institutions failed them they at least had some sort of backup?
Navigating healthcare while dealing with a mental health issues is a paradox. Is the damage caused greater than the damage prevented from allowing access to a tool that can help lower that barrier to entry?
Looking at general trends I find that the harms outweighing the benefits very hard to believe. Do you have anything non-anecdotal, on that specific point, to back it up?
law in itself is very gatekeepy - you have to learn the “correct” way to interpret the words and then get certified for it.
More like you as an average person have no idea what it’s hallucinating and what is the truth and may end up in jail or the hospital like the cases that are starting to get reported on who did just that.
with a group of people that has a bias towards a certain delusion, and/or high acceptance of directly scientifically contradictory beliefs, you can get a bubble and affirm themselves out of reality in the same way. i also wouldn’t suggest banning groups of people, but i believe with groups or with LLMs, a good grounding in both scientific thinking is the best way to keep ‘stable’ away from delusion.
a shame that grants are being filtered by a project 2025 author right now, and a bunch of important science got cut by this admin.
also see it like mobile phones, which instead of being useful mobile personal computers, we got data collection bricks that has a skeleton of a few practical tools built into the soft edges of a giant gambling and addiction pipeline.
there’s a whole rant about the high-level psychological manipulation employed against average people just to make them more comfortable losing their money, which does not mean that portable personal computers are bad, but the failure of anti-trust and consumer protection has allowed all technological and social development to be hindered purely for the extractive demands of the people who actually get to decide what happens.
as long as you keep the general public too tired to think about it, you just have to convince them that some other small group is their tribal enemy and that’s where your collective focus and energy should be going, rather than the more simple answer of rich assholes being opportunistic and stupid/evil.
I guess it’s like banning loan sharks. Sure some people could only.get loans through them because banks were racist or whatever. But it’s objectively bad for everyone using it mainly because when it’s bad its really really bad.
I really think this is a bad idea these should really just be flagged, there’s many situations where asking these questions in necessary and there’s few if any other ways to get an answer
Sorta agree. Maybe force them to have a very obvious sign that says AI is routinely wrong about these topics specifically and are not a source to consider reliable, and then also offer a helpline where one can get reasonable answers even if the answer is “unfortunately you need to seek expert opinion on this”
But googling basic labor laws and simple healthcare tricks should be acceptable like stretches for wrist or knee pain, skin/hair care or what level of diarrhea is hospital worthy should be fine.
Same, especially for medical advices.
I am currently going through a very complex diagnosis of combination of SCDS and migraines, and neck joint / facial nerve issues through doctors - ENT, and neurologists.
Having AI backing me up with providing a structural guidance from multiple sources is a huge blessing.
Especially when you die from migraine, and trying to understand are symptoms worth to go to emergency ward or not.
AI can’t back you up better than you can with your own research. Why are you even considering AI credible? What gives it credibility?
It’s not credible, but you can ask it for a credible source and read that, which is still incredibly helpful.
It’s not worth trusting AI with your life over a doctor. If in doubt, go to the emergency ward (assuming you’re not American, if you’re American then you’re fucked either way).
If you’re an american the AI is worth using, as you can ask it for a credible source and verifying that is much faster.
The problem with medical questions like these is LLM would spew out info confidently that doctors/specialist don’t even know the answer to. LLM doesn’t know when it’s guessing.
Ask it for a source and verify it, you can even give the source to a doctor if you can’t
This is super dumb. People will just Google their questions instead, and get worse results.
What i don’t like about this, is the ignorance towards every other field.
The ai is not allowed to tell you that you might have some illness but the LLM is allowed to vibe code the system that holds your medical data, or the software of the medical device that analyses your e.g. blood and provide the data to the doctor.
Systems and professions are interconnected. Chatgtp telling me that I have cancer is not as bad as chatgtp making mistakes causing my medical data to be stolen and sold.
I’m on the same side as you, vehemently fuck generative AI, however the difference between some LLM output incorrectly telling you that you have cancer and a shitty coder vibe coding with severe vulnerability is the ability to place blame, there’s a person who’s job it was to make sure info is secure and the vibe code actually works. There is no singular person who is traceably culpable for shitty AI output telling you bad info.
I am with you but…
Bad code that puts out wrong information to a doctor or endangers my data, threatens my safety without me having any control over. No amount of my own expertise will protect me. Simply because I don’t control the situation. No amount of awareness campaigns will fix my issue.
But awareness campaigns have successfully communicated to most of us that googling your health issues will result in nothing of value, and usually a cancer diagnosis. I can choose to protect myself by not asking a llm. Like I protected myself by not asking google.
Good direction, but very futile considering that guardrails on LLMs are about as real as LLMs are intelligent
It might not work, but it would serve to increase awareness that you absolutely cannot trust these machines to give you correct answers, ever.
This is fair. The next step is to litigate the companies for the harms they cause.
There’s a whole drove of people now that have gotten harmed, murdered, and committed suicide because of the psychological harm that these models cause.
Were it a car with some kind of manufacturing defect, there’d be recalls and you wouldn’t be able to sell ones with the fault. If it was salad contaminated with bacteria, that’d be recalled and the entire batch would be destroyed. We’re holding salad to a higher standard than technology that people are pushing to integrate deeply into society; into classrooms to help educate children, into medical settings to help organise journals and expedite the work of medical staff, into law… the list goes on.
All the models do this, and it causes real harm to people, but other than individual families and such litigating them, no state has stepped in to protect its citizens.
Litigation against tech companies for the awful shit their LLMS are spewing sounds like a really fun idea.
It’s awful that most people either don’t realize or don’t care. People seem to trust these companies above all else, which is insane. They don’t have the best interest of the users in mind. They only care about profits.
The nice thing about laws is that they don’t give a shit. If they say you can’t do something, it’s your problem to make sure you don’t do it.
It could provide a new revenue stream for NYC, since big tech companies will treat the fines as a cost of doing business
Idiots still doing shit based on that.
Limiting the medical knowledge people can get is a good direction? really?
Limiting medical knowledge from reliable information resources would be terrible. I’d say that limiting the spread of plausible-sounding medical misinformation generated by a fancy autosuggest tool is good, though.
AI chatbots don’t actually “know” anything. They just generate the most statistically probable appearance of a response to each prompt based on their training data. They don’t tell you when something wasn’t in their training data or when there’s limited research on a subject, just give you the most statistically probable response like always, which leads to a “hallucination” that sounds true but isn’t. A tool that’s trained on MCAT testbanks and “aces” the MCAT won’t necessarily give reliable medical advice to real patient questions. Even if you gave an AI chatbot all the accurate training data in the world, it would still hallucinate information occasionally, and it wouldn’t give you any indication of when it did so. If your doctor screws you over, they can lose their license, but if ChatGPT screws you over, there’s currently very little to stop OpenAI from screwing over more people. OpenAI can just stand behind the disclaimer that says ChatGPT can make mistakes, never mind that users are treating it like a licensed medical professional. There are AI tools that are useful in medical research, but they’re a VERY different kind of AI from the AI chatbots this bill is concerned with.
If you want medical knowledge, there are tons of medical journals, news sources, blogs, and podcasts from doctors/scientists who don’t hallucinate BS when they don’t know something.
Instead of banning it then teach people how to use it and not to trust it 100% (who the hell is doing this to begin with?)
If you want medical knowledge, there are tons of medical journals, news sources, blogs, and podcasts from doctors/scientists who don’t hallucinate BS when they don’t know something.
I can ask an LLM to help me sourcing these journals and informations
Expecting people who have already taken the easy route of asking an AI to summarize medical information to do additional due diligence by reading the sources that the AI spits out is naive as fuck.
There’s a reason it became a meme for someone to Google their symptoms, click on WebMD to read about it, and then come to the conclusion that they have cancer, all without ever having seen a medical professional in the meanwhile. Do you really think these sorts of people are going to be using AI responsibly for research purposes only?
There’s a reason it became a meme for someone to Google their symptoms, click on WebMD to read about it, and then come to the conclusion that they have cancer, all without ever having seen a medical professional in the meanwhile. Do you really think these sorts of people are going to be using AI responsibly for research purposes only?
If they come to the wrong conclusion at worst they go to the hospital and someone there tells they are wrong and not to worry. We haven’t ban googling your symptoms.
You’ve missed the point.
Someone who Googled their symptoms and ended up at WebMD were presented with reliable albeit ambiguous information. Carefully curated by medical professionals and not random snake oil salesmen or homeopathy gurus. They fail to interpret the symptoms correctly, because of course they would. If they are looking up causes of a headache, it can be anything from brain tumor to low blood sugar, or even no reason at all.
Someone who asks AI for medical advice is going to get the entire internet worth of fucking opinions, as well as medical journals and such, because that’s where ChatGPT, Gemini, Grok and Claude all got their data from. And it will string those words together in a way that sounds authoritative, but it will have literally zero concept of what it’s talking about. It’s an overgrown autocomplete. You ask for it to provide some citations, but no amount of asking for citations will compel people to actually read them, nor will there be any guarantee that the LLMs couldn’t just hallucinate that as well.
It’s like taking a Google search for medical information and adding a middleman who occasionally filters in junk data, then wrapping it in a nice package, giving the impression of authority and accuracy, making the process as frictionless as possible.
There can be no reasonable expectation that people who have not historically been capable of making good choices in regards to seeking online medical advice will suddenly start doing it all of their own volition.
You might say “What’s the harm if they end up at the hospital either way”? But the danger is that they never go to the hospital at all because the chatbot told them to sun their perineum or inject bleach with the confident prose of a medical doctor and they just… do it. Because why would the AI be wrong?
And who is to blame when someone is injured or killed as a result? Doctors have standards, and they take an oath to first do no harm. If they make a mistake, they are liable for it. How can you hold a machine intelligence accountable for inaccurate diagnoses?
The simple answer is you can’t, and so logically they should be strictly prohibited from masquerading as any kind of authority on medicine.
but it will have literally zero concept of what it’s talking about.
If LLM answers had zero concept they wouldn’t work. We are here talking about it because the answers are correct up to a certain point that makes them good enough.
It’s like taking a Google search for medical information and adding a middleman who occasionally filters in junk data, then wrapping it in a nice package, giving the impression of authority and accuracy, making the process as frictionless as possible.
These middleman are also called teachers, when you go to medical school you have many of these, you don’t get handled the medical encyclopedia and left on your own.
You might say “What’s the harm if they end up at the hospital either way”? But the danger is that they never go to the hospital at all because the chatbot told them to sun their perineum or inject bleach with the confident prose of a medical doctor and they just…
that’s something you are making up, an LLM telling you to inject bleach would be the same as it telling you that 2+2= coconut, it’s the wrong answer, if all the answers where wrong we wouldn’t be here talking about this tech.
And who is to blame when someone is injured or killed as a result? Doctors have standards, and they take an oath to first do no harm. If they make a mistake, they are liable for it. How can you hold a machine intelligence accountable for inaccurate diagnoses?
Doctors prescribe you medicines and do surgery on you, an LLM can’t do these things, if AI tells you to take a medicine you still have to go to the medic to get these.
The simple answer is you can’t, and so logically they should be strictly prohibited from masquerading as any kind of authority on medicine.
The propaganda meme is talking about a ban on AI answers about medicine. LLMs have no authority on medicine, that’s something we will be discussing in 20 years.
You can teach people how to use it all day long, but most still won’t use it well consistently in real life. Humans instinctively trust confident-sounding language. Most of us are also way worse at fact checking than we think. LLMs play right into humanity’s flaws, and they’re built to maximize engagement rather than help us avoid going past the tools’ limits. I don’t trust LLMs to be accurate about anything, but I still caught myself reading and failing to check Google AI overviews sometimes before I changed my default search engine to a no-AI alternative. It’s just instinctual when an “answer” is right there at the top of the page in response to your search question.
Sure, you can use an LLM to help you find sources, but at that point you might as well just use a search engine. There are search engines that provide links with actual text from each site rather than potentially hallucinated summaries.
Sure, you can use an LLM to help you find sources, but at that point you might as well just use a search engine. There are search engines that provide links with actual text from each site rather than potentially hallucinated summaries.
Some popular search engines results are way worst than AI
Some search engine results are crap, yes. But even the worse search engine (minus AI overviews or whatever) gives you a setup where: (1) each piece of information has an actual source, (2) you know who the source is, and (3) you can make decisions about which information to trust accordingly. If a popular search engine puts a sketchy company’s blog at the top of the SERP, you can actually see that and choose to look father down for something more reliable. Also, if your current search engine keeps failing you, try a different one! There are many options out there that you might find more helpful than today’s Google or Bing or DuckDuckGo.
Good luck with things, truly. I think you’re going to get hurt eventually if you use LLMs for medical advice, and I genuinely don’t want that for you or anyone. Hopefully this conversation has at least made you a little more skeptical about using LLMs in certain ways.
Good luck with things, truly. I think you’re going to get hurt eventually if you use LLMs for medical advice, and I genuinely don’t want that for you or anyone. Hopefully this conversation has at least made you a little more skeptical about using LLMs in certain ways.
Good luck with things too, in case you haven’t noticed this meme is from a weird association that doesn’t seem to exist and that promotes pseudo scientific content
Great! Let’s add regulation/accountability to search engine results too then, if they’re as problematic as this.
I don’t know where you live but in my country people don’t go to the medic or hospital for minor things because it cost them money and time. In this scenario limiting access of any sort of medical tool and knowledge sounds like further fueling a corrupted system where power is concentrating in the hands of a few individuals and where people are losing their rights.
Instead of banning it then teach people how to use it and not to trust it 100% (who the hell is doing this to begin with?)
A 20% bad medical advice is worse than 100% bad advice or no advice.
There’s no “teach people how to use it”. You either use it and get completely won’t advice 20% of the time or you use reputable sources.
To verify what LLM says takes same or more time than just using reputable sources.
Limiting the medical misinformation.
FTFY.
AI is a tool that can provide good and bad results. To people who can’t get access to medical care easily (half of the people on the planet) it can be an useful tool to get basic medical knowledge
The issue is that AI is extremely good at sounding right, but only okay-ish at being right.
When you get good sounding information that is wrong, it can be extremely problematic and often much worse than no info at all.
Imagine LLMs aiding people with health improvements, only to lead them into health decline (actually already happening, a lot) not to mention the LLMs that further convince people of their conspiracies.
There is a reason why health professionals go through a lot of training and are under a lot of scrutiny, you wouldn’t want just any rando who thinks he know better to be your doctor
When you get good sounding information that is wrong, it can be extremely problematic and often much worse than no info at all.
A partially good answer is better than no information at all
Imagine LLMs aiding people with health improvements, only to lead them into health decline (actually already happening, a lot)
Already happening where? How can AI lead you to health decline? It cannot prescribe you medicines or do surgery on you
not to mention the LLMs that further convince people of their conspiracies.
It can also help people understanding if conspiracies are true or false, for example had you ask an LLM about this post it would have highlight to you that there’s a good chance that it is propaganda.
LLMs are NOT a reputable source of medical information. They regurgitate information from random sources, and can lead to disastrous effects when pulling from bad sources or incorrectly summarizing from a good source. At best it should exclusively provide links to sources (good or bad) and let the end user make the determination if that source can be trusted.
This isn’t limiting information at all. People can still access all the reputable sources that provide the information they are looking for. This is simply cutting out the unreliable middleman.
At best it should exclusively provide links to sources (good or bad) and let the end user make the determination if that source can be trusted.
You can ask AI to do exactly this and according to your own logic turning it out into a good at best medical tool.
