Chatbot analysis of an image of Carl Sagan.

A couple of weeks after the Internet made Tay, a sophisticated chat bot, into a Hitler-loving, genocidal repository of 4chan memes, people have discovered another new-ish bot from Microsoft. This one is called captionbot, and it exists to try and describe to you the content of an image.

Upload any image to Captionbot, and you’ll get a sentence back describing what it thinks it sees. Sometimes, the bot will say, it’s not really confident with its answer. It also gets a lot of things wrong right now, based on my incredibly small sample size of random images that I decided to run by the bot because I felt like it, and what other people have said on the Internet.

Here is its analysis of a photo of Tillman, the skateboarding dog (RIP), trying out a surfboard:

Screenshot of CaptionBot. Original image: MARK RALSTON/AFP/Getty Images

I also tried running a meme through Captionbot:

screenshot of Crying Jordan, as interpreted by Caption Bot

This person tried another, very bad meme, and the answer was not good:


As Microsoft explains, Caption bot, like the rest of its AI projects, is still learning, so Microsoft is hoping it will more accurately describe images as more people play with it.

So let’s turn down the difficulty level a little bit, and try a cat:

Captionbot’s analysis of my cat (photo by Abby Ohlheiser)

Okay. What about the pope taking a selfie?

Pope Francis taking a selfie, analysed by Caption Bot. (original is L’Osservatore Romano, via AP)

What about the president?

Obama, as analyzed by Caption Bot. (original image: EPA/MICHAEL REYNOLDS / POOL)

So right now, you might be thinking, “Hey! that’s not bad. Maybe Caption Bot is trying to say that the president is just ‘a man,’ like the rest of us.”

Except Caption Bot can recognize some people by name. Just not, in this case, the president. Here is Caption Bot’s response to a random image I uploaded of Agent Dana Scully, played by Gillian Anderson from O.G.-era “X-files:”

Caption Bot on a screenshot from The X-Files.

It seems, to varying degrees, to be able to identify several other celebrities. The bot recognized Anderson’s former co-star David Duchovny, Idris Elba, and Jennifer Lawrence, but not OPRAH, and it was only kind of sure about Carl Sagan, as you saw at the beginning of this piece.

Don’t try cartoons, though:

Hmmmm. Screenshot of Caption Bot's take on the animaniacs
Hmmmm. Screenshot of Caption Bot’s take on the Animaniacs

Guys, please be nice.

If you’d like to play around with Caption Bot for yourself, it lives here.