Weird Science

Scientific Research is Threatened by Much More Than Just AI Slop

Abstract

Abstract:

This piece talks about some of the hidden harms of integrating generative AI tools into academic research. The dangers go beyond fake citations and confabulated facts. GenAI infuses the entire research process with distortions and biases. I specifically talk about the MAHA report, and the use of AI powered “research assistants” like Google NotebookLM. NotebookLM treats all sources equally positively, and sycophantically; even studies that have been debunked or are fraudulent. Google's statement that it's entirely "grounded" in your own sources is not accurate. This is because the LLMs that underly all functions of this AI powered tool were created and trained to power commercial products like chatbots. All "derivatives" (summaries, overviews) that NotebookLM creates are filtered through training layers and content moderation layers to promote polite engagement. GenAI creates probabilistic and algorithmic connections that undermine traditional and stable ways of creating scientific consensus. The "Key Topics" attached to sources by NotebookLM refer to no known, or stable, knowledge organization framework. These concepts and keywords are generated from the sea of chaos that is an LLM - blogs, social media, junk science. I tested multiple junk science studies, and I use a notorious bad researcher name Brian Wansink as a prime example. Wansink has the dubious distinction of having had six scientific articles retracted in a single day! And yet, NotebookLM summarizes his work most favorably. LLM model and their filters designed for social chatbots are now influencing academic research where studying sensitive and controversial topics is essential.