What’s an underrated way to study decisions? Think out loud: Researchers developed an online protocol for recording audio of people thinking aloud when trying to overcome trick questions

Trick questions of this kind are invaluable to cognitive scientists because they shed light on the cognitive quirks that shape our decision-making. “These aren’t just trick questions,” explained Nick Byrd, a philosopher-scientist and Intelligence Community Fellow who led the research at Stevens Institute of Technology. “They’re opportunities to reveal whether we are thinking reflectively — sometimes we are lured into accepting faulty responses that feel right.”

In their work, Byrd’s team worked with research startup Phonic, Inc., to develop an online protocol for recording audio as people narrate all of their thoughts while answering a series of trick questions. The work, reported in a special issue of the Journal of Intelligence, may not only help scientist reveal how people solve reasoning problems, but help optimize analysts’ reports and briefings that inform high-level decisions.

“The mind is often a black box, and figuring out what’s going on inside is challenging,” Byrd explained. “Brain imaging is useful, but we can also learn a great deal simply by asking people to say whatever they’re thinking.”

However, think-out-loud experiments come at a cost: Recording people thinking out loud is much more complex and time-consuming than surveys. “It took three of us three months to complete an in-person think-aloud study — and just one of us three hours to complete an online survey with the same number of subjects,” Byrd said.

The new tool will significantly decrease the amount of time for completing think-out-loud experiments, lifting the lid on the black box for how humans overcome trick questions.

The paper shows that both online and in-person think-aloud testing can offer valuable insights into subjects’ cognitive world. Many researchers focus on whether subjects fall for the trick or give correct answers, assuming that arriving at correct responses involved careful reflection and falling for the tricks did not. In the think-aloud recordings, however, Byrd’s team found that in as many as 31% of cases, the opposite was true. “Sometimes people were tricked despite thinking reflectively, and sometimes people were correct without reflecting,” Byrd said.

Presented with the race-position question, for instance, one subject gave the correct answer almost immediately — then explained that it was easy for them, since they’d run track in high school. “It turns out that with enough relevant experience you can answer quickly without falling for the lure,” Byrd said. “That might sound like a trivial insight, but higher education’s publish-or-perish environment discourages scientists from using resource-intensive methods like think-aloud protocols to detect and quantify such measurement error.”

Having shown that online surveys can streamline think-aloud audio collection, Byrd and colleagues now hope to use AI tools to streamline the analysis, making it possible to quickly record and interpret think-aloud data. That added efficiency would enable more researchers to conduct larger studies, study more diverse and representative groups of people, and obtain greater confidence in their results.

“One complaint about psychological science is its focus on student research participants in countries like the United States, who aren’t necessarily representative of the whole of humanity,” Byrd said. “Our online method will enable researchers to reach anyone with a smartphone and an internet connection, which includes most of the world’s population.”

The stakes can be high in decision-making: the U.S. Intelligence Community pays close attention to techniques that overcome faulty thinking. “If you’re making important decisions, our think-aloud recordings show that people were much more likely to overcome mistakes if they questioned their initial impulse and considered some reasons,” Byrd said. “So, helping scientists reveal how people solve reasoning problems may improve not only our decision science, but also our decision-making.”