they don’t really write the model is having an Aha moment due to some insight it had.

Well, they really can’t write it that way because it would imply the model is capable of insight which is a function of higher cognition. That path leads to questioning if machine learning neural networks are capable of any real sparks of sapience or sentience. Thats a ‘UGI’ conversation most people absolutely don’t want to have at this point for various practical, philosophical, and religous/spiritual implications.

So you can’t just outright say it, especially not in an academic STEM paper. Science academia has a hard bias against the implication of anything metaphysical or overly abstract at best they will say it ‘simulates some cognative aspects of intelligence’.

In my own experience, the model at least says 'ah! aha! Right, right, right, so…` when it thinks it has had an insight of some kind. Whether or not models are truly capable of such thing or is merely some statistical text prediction artifact is a subjective discussion of philosophy kind of like a computer scientist nerds version of the deterministic philosophical zombie arguments.

Thanks for sharing the video! I havent seen computerphile in a while, will take a look especially with that title. Gotta learn about dat forbidden computation :)