Trekkie Quirk Makes AI Chatbots Better at Math

Bet you didn't have this one on your AI bingo card. New research has found that when fed phrases that simulated being on the Starship Enterprise, chatbots spit out more accurate results to math problems. The study, published on arXiv and first reported by New Scientist, didn't set out to explore Star Trek's relationship to AI. But per Gizmodo, the researchers were piggybacking on past results that showed AI produced better answers when given encouragement, and stumbled upon this quirk. In the study, the team tried 60 human-picked phrases that tested this "positive thinking" theory, feeding those prompts into three different Large Language Models before having them attempt grade-school level math problems, per Business Insider.

The phrases they used included, "You are an expert mathematician," "You are a professor of mathematics," and (a personal favorite) "You are as smart as ChatGPT." They closed the queries with assuring statements, like "This will be fun!" and "Take a deep breath and think carefully." Results came back mixed, so the researchers used AI to drum up better suggestions, which they tested again. One particular prompt yielded the most accurate answers: "Captain's Log, Stardate [insert date here]." Why a Star Trek reference resulted in better math is puzzling to even the researchers, who wrote, "This revelation adds an unexpected dimension to our understanding and introduces elements we would not have considered or attempted independently."

Unfortunately, this discovery doesn't mean randomly including Star Trek keywords will beam up the best answers on chatbots. What we can glean from the study is that AI's optimization of the prompts outperformed human optimization—and that how it arrives at those results from the datasets it was trained on is increasingly mysterious. "The key thing to remember from the beginning is that these models are black boxes," Catherine Flick, ethics and games technology professor at Staffordshire University, told New Scientist. "We won't ever know why they do what they do because ultimately they are a mélange of weights and probabilities and at the end, a result is spat out." (Elon Musk is suing OpenAI.)