"Even Though People Didn't Mean To, They Were Really Excluding Me"

Photo: Courtesy Facebook.
We scroll through Facebook every day, looking at pictures and videos, commenting and liking. Now, imagine that experience without actually being able to see all those photos and videos. How would you know what people were talking about? How would you know what to say to join in on the conversation?

"It was kind of a letdown," Facebook's accessibility specialist Matt King, who is blind, tells Refinery29 of what it was like using the social network without sight. "Even though people didn't mean to, they were really excluding me from the conversation. It was an inherently visual conversation."

King lost most of his vision while he was in college. He has since spent his career — 25 years at IBM and close to a year at Facebook — dedicated to making technology available to everyone. After joining Facebook in its early years and struggling to navigate its interface, now he, and the rest of the Facebook accessibility team, are really opening up the social networking conversation with a new product, Automatic Alternative Text.

Normally, a visually impaired person uses some form of screen reader to audibly read out text that's on screen (on an iPhone, you can do this by switching on VoiceOver in its settings). However, in today's world, much of what's happening online is in pictures and videos. Automatic Alt Text comes in to translate those images into words. You can see it in action in the video below.

I'm inspired by this video about our artificial intelligence research at Facebook.Our AI can now look at a photo, figure out what's in it and help explain it to you. This is especially helpful if you're blind or can't see the photo. We see AI as helping computers better understand the world -- so they can be more helpful to people.We're still early with this technology, and you can already start to imagine how helpful it will be in the future.

Posted by Mark Zuckerberg on Tuesday, November 3, 2015

"That whole saying of, 'A picture is worth a thousand words,' I think it's true, but unless you have somebody to describe it to you, even having three words helps flesh out all the details that I can't see. That makes me feel included," a woman explains in the video.

In the past, she'd scroll through her feed and have to use the comments underneath to demystify what the image was of. With Automatic Alt Text, she can get some semblance of what a friend may be "So happy!" about without resorting to detective work. (A "So happy!" picture of a steamy slice of pizza doesn't necessitate quite the same response as that of a smiling, newly engaged couple, for instance.)

"It’s hard, if you’re sighted, to put yourself in someone else’s shoes," Jeff Wieland, who heads up Facebook's accessibility team, says. "If you don’t know someone who’s blind, people say, 'Well do blind people want to know what’s in a photo? If they’ve never seen a beach?' The emotion of hearing someone is smiling, that warms your heart whether you can see or not."

Automatic Alt Text works using image recognition to identify objects in photos. It generates, and reads out, a description of a photo, listing out items that may be in the picture. It can identify modes of transportation (things such as a car, bus, bicycle, or boat), nature (words such as outdoor, mountain, tree, and snow), words relating to someone's appearance (glasses, baby, smiling), sports, foods, and selfies.
Video: Courtesy Facebook.


While Automatic Alt Text isn't incredibly robust at this point, it is accurate, and it's exceptionally more helpful than not having any photo description at all.

And Facebook isn't the only company working on this kind of technology. At its developer conference last week, Microsoft also showed a demo of how its artificial intelligence systems help one of its blind engineers identify what's happening in front of him in the real world. Just when you thought AI was doomed, or doomed to remain incredibly creepy, here it is finally being used for the betterment of mankind.

As for Facebook's new feature, you can try the experience yourself. On your phone, go to Settings, General, Accessibility, and then turn on VoiceOver. Then, go to the Facebook app. When you swipe over a photo, you'll hear the app read out some of the different items the image may include.

"It’s not just about making [Facebook] possible to use," King says, "but making it possible for someone with a disability to get as much enjoyment out of the product as anyone else — to be equally engaged and connected, and feel equal in society, [like], 'Hey, we matter, we’re part of it, too.'"
Photo: Courtesy Facebook.

More from Tech

Complaining today is much easier than it used to be. Lost luggage, bad service, extra charges, broken products — you used to have to put a letter in the ...
It's Google's 18th birthday which means it can finally see all the porn it serves up
Tonight marks the first of three presidential debates between candidates Hillary Clinton and Donald Trump. Because of the debate's close proximity to New...
GENERATION STARTUP presents Women In Tech, an episode of an exclusive web series with footage left on the cutting room floor. The feature documentary ...
The American Civil Liberties Union (ACLU) developed a phone app that will assist you in recording and reporting police conduct. The app, Mobile Justice, ...
If you aren't registered to vote, don't panic. As a matter of fact, thanks to Snapchat, you can now do it in just one minute, reports Time. The social-...
You may start to notice your Uber drivers snapping a lot of selfies, but it doesn't necessarily mean that you're riding with an Instagram addict. Your ...
(Paid Content) Refinery29 is now on Versy, a messaging and content-sharing app made for busy people like us
Snapchat CEO Evan Spiegel announced that the company has developed new video-enabled sunglasses, called Spectacles, and will subsequently be rebranding ...
You're at 20,000 feet, and you realize your flight doesn’t have WiFi — which wouldn't be a big deal, except you forgot your book at home, and none of your ...
It's a sad but true fact that a bad commute — one where you just miss the bus or subway — has the power to ruin your day. More often than not, these near...
When you think of playing a video game, what comes to mind? Is it a raucous game of Wii Bowling when you were in college? Hours spent on your Game Boy ...
If you're a Yahoo user (or ever have been), you'll want to change your password ASAP — and not just on that Yahoo account. Today, the company confirmed ...
There are two types of people in this world: the zero-inbox purists, and the ones with hundreds (or even thousands) of unread messages in their inbox. ...