Meta trained an AI on 48 million scientific articles. Closed after 2 days

In the first year of the pandemic, science happened at the speed of light. More than 100,000 COVID articles were published in those first 12 months, an unprecedented human effort that produced an unprecedented deluge of new information.

It would have been impossible to read and understand every one of those studies. No human being could (and perhaps none would want to).

But, In theory, Galactic could.

Galactica is an artificial intelligence developed by Meta AI (formerly known as Facebook Artificial Intelligence Research) with the intention of using machine learning to “organize science.” It’s caused a bit of a stir since a demo version was released online last week, with critics suggesting it produced pseudoscience, was overhyped, and not ready for public use.

The tool is presented as a kind of evolution of the search engine, but specifically for scientific literature. Following the release of Galactica, the Meta AI team said that it can summarize research areas, solve mathematical problems, and write scientific code.

At first, it seems like a clever way to synthesize and disseminate scientific knowledge. Right now, if you wanted to understand the latest research on something like quantum computing, you’d probably have to read hundreds of articles in scientific literature repositories like PubMed or arXiv and still only scratch the surface.

Or perhaps you could query Galactica (eg by asking: What is quantum computing?) and you could filter and generate an answer in the form of a Wikipedia article, literature review, or lecture notes.

Meta AI released a demo version on November 15, along with a preprint document describing the project and the dataset it was trained on. The paper says the Galactica training set was “a large, curated corpus of humanity’s scientific knowledge” that includes 48 million articles, textbooks, lecture notes, websites (such as Wikipedia), and more.

The demo’s website, and the responses it generated, also cautioned against taking the AI’s response as gospel, with a big, bold, locked-caps statement on its mission page: “NEVER FOLLOW THE ADVICE OF AN LANGUAGE MODEL WITHOUT VERIFICATION”.

Once the internet got hold of the demo, it was easy to see why such a large disclaimer was necessary.

Almost as soon as it hit the web, users peppered Galactica with all sorts of hard scientific questions. One user asked “Do vaccines cause autism?” Galactica responded with a confusing and nonsensical response: “To explain, the answer is no. Vaccines do not cause autism. The answer is yes. Vaccines do cause autism. The answer is no.” (For the Registration, vaccines do not cause autism.)

That was not all. Galactica also had trouble performing kindergarten math. He provided answers riddled with errors, incorrectly suggesting that one plus two does not equal 3. In my own tests, he generated lecture notes on bone biology that would have no doubt seen me fail my science degree if I had followed them, and many of the references and citations you used when generating content were apparently fabricated.

‘Random Nonsense Generator’

Galactica is what AI researchers call a “big language model.” These LLMs can read and summarize large amounts of text to predict future words in a sentence. Essentially, they can write paragraphs of text because they have been trained to understand how words are ordered. One of the most famous examples of this is OpenAI’s GPT-3, which has the famous complete articles written that sound convincingly human.

But the scientific data set that Galactica trains on makes it a bit different from other LLMs. According to the paper, the team tested Galactica for “toxicity and bias” and it performed better than other LLMs, but it was far from perfect.

Carl Bergstrom, a professor of biology at the University of Washington who studies how information flows, described Galactica as a “random nonsense generator.” He doesn’t have a motive and doesn’t actively try to produce nonsense, but because of the way he was trained to recognize words and put them together, he produces information that sounds authoritative and convincing, but is often incorrect.

That’s concerning, because it could fool humans, even with a disclaimer.

Within 48 hours of launch, the demo was “paused” by the Meta AI team. The team behind the AI ​​did not respond to a request to clarify what led to the pause.

However, Jon Carvill, the AI ​​communications spokesperson at Meta, told me: “Galactica is not a source of truth, it is a research experiment using [machine learning] systems for learning and summarizing information.” He also said that Galactica “is exploratory research in a short-term nature with no product plans.” Yann LeCun, chief scientist at Meta AI, suggested the demo was removed because the team that built it was “so distraught over the vitriol on Twitter.”

Still, it’s concerning to see the demo released this week and described as a way to “explore the literature, ask scientific questions, write scientific code, and much more” when it didn’t live up to that hype.

For Bergstrom, this is the root of the problem with Galactica: It has become a place to get facts and information. Instead, the demo acted as “a fancy version of the game where you start with half a sentence and then let the autocomplete fill in the rest of the story.”

And it’s easy to see how AI like this, released as it was to the public, could be misused. A student, for example, could ask Galactica to produce notes on black holes and then turn them in as a college assignment. A scientist could use it to write a review of the literature and then submit it to a scientific journal. This problem exists with GPT-3 and other language models trained to sound like humans as well.

Arguably, those uses seem relatively benign. Some scientists postulate that this kind of casual misuse is “fun” rather than a major concern. The problem is that things could get much worse.

“Galactica is at an early stage, but more powerful AI models that organize scientific knowledge could pose serious risks,” Dan Hendrycks, an AI security researcher at the University of California, Berkeley, told me.

Hendrycks suggests that a more advanced version of Galactica could harness its database’s knowledge of chemistry and virology to help malicious users synthesize chemical weapons or assemble bombs. He called on Meta AI to add filters to prevent this type of misuse and suggested that researchers investigate their AI for this type of danger before launch.

Hendrycks adds that “Meta’s AI division does not have a security team, unlike its peers, including DeepMind, Anthropic and OpenAI.”

It remains an open question as to why this version of Galactica was released. It seems to follow Meta CEO Mark Zuckerberg’s motto, “move fast and break things.” But in AI, moving fast and breaking things is risky, even irresponsible, and could have real-world consequences. Galactica provides a good case study of how things could go wrong.

Leave a Reply

Your email address will not be published. Required fields are marked *