MIT system hits ‘near-human’ accuracy in finding memorable photos

This photo of Donald Trump was quite memorable, according to the MIT prediction system.
This photo of Donald Trump was quite memorable, according to the MIT prediction system.

Tweaking the color and exposure of your photos is fun, and it turned out well enough for Instagram. MIT scientists think they can take things much further, thanks to artificial intelligence software that can predict the strength of human memory.

Researchers at the university have developed an algorithm that can predict how well people will remember different images with “near-human” levels of accuracy.

The system is now available online, where anyone can feed it images that are assigned a memorability score and overlaid with a “heat map” that shows which regions are most likely to stick in someone’s mind — warmer colors like red, orange, and yellow are more memorable, and cooler colors like blue indicate portions that don’t grab someone’s memory quite as hard.

Screen Shot 2015-12-15 at 2.11.44 PM
The technology could be used to make learning materials more memorable and advertising pitches more effective, said Aditya Khosla, a graduate student at MIT’s Computer Science and Artificial Intelligence Laboratory.

Eventually, the software also could help make photographs more memorable by altering what’s in them: blurring out a forgettable region, cropping the image to focus on the most memorable parts, or even adding items that trigger stronger memories, Khosla said.

“Basically, removing the non-memorable parts of the image and replacing them with something much more memorable,” he said. “But how do we avoid making it look like a really horrible Photoshop mistake? That is a really hard research problem.”

There’s plenty more work to go before they get to the point of a building an app that could automatically perfect your selfies, however. For one thing, researchers still don’t fully understand how the software they built actually works — they just know that it produces really accurate results.

“It’s weird, right?” Khosla said. “But you can say the same about a baby. You make a baby, and you can’t explain how or why the baby’s brain works. But clearly, you made the baby.”

The project started with tens of thousands of photos that were used in experiments asking people to record if they’d seen the image before. Those images were scored for how memorable they were, and that information was fed into the algorithm for analysis.

That’s where the mystery comes in. The photo-analysis software was structured as a neural network, which mimics the way living brains process information. Such systems can essentially train a computer to pick apart massive sets of data and find the hidden patterns inside that information without being told what to look for by humans.

Screen Shot 2015-12-15 at 2.14.26 PM
It’s similar to the technology used in speech-recognition like Apple’s Siri, facial recognition used in photo-tagging software on Facebook, or auto-complete search results on Google’s search engine, MIT said.

In this case, the neural networks were given the images and told how high each one scored on the human memory scale. But researchers didn’t tell the software to single out what they thought made one image more memorable than another by, say, telling the computer to look for human faces. They just sat back and let the machine do its work.

The result, they reported in a new study, was striking. When it was shown an unknown image, the new algorithm performed 30 percent better than previous tools at predicting human memory, and was only a few percentage points away from the average real result of humans.

Being puzzled by the methods your computer system uses to solve problems is par for the course in artificial intelligence research, Khosla noted, and opens up some interesting new areas to continue researching.

“The network knows something that we don’t,” he said. “It won’t reveal its secrets to us.”

Screen Shot 2015-12-15 at 2.15.00 PM