Some of the features these companies previewed aim to make the world easier to navigate for the blind and vision impaired. Others were designed to make music, movies and conversations easier to follow for people who can’t hear well — or at all.
Here’s our guide to some of the most interesting changes these companies are planning, and what they may be able to do for you or someone in your life.
Available on: iOS
When is it coming? “Later this year,” Apple says.
iPhones come with a magnifier app that does pretty much what you’d expect: it enlarges our view of whatever’s in front of the camera. But in an upcoming software update, the app will gain a “door detection” tool to help blind iPhone users more easily get where they’re going.
Here’s how it will work: once someone opens the magnifier app, they jump into a new “Detection Mode” and aim the phone at their surroundings. If there’s a door nearby, the app will audibly tell its user how far away it is, and whether it’s open or closed. If the app has a clear view of that door, it’ll also begin to describe aloud specific details — think numbers and text on and around the door and whether you’re meant to turn a knob or push a bar to open it.
Door detection is perhaps the most clever accessibility feature Apple has talked about recently, but it comes with a very specific catch: it only works on a specific list of gadgets. For now, supported devices include the iPhone 12 Pro, 12 Pro Max, 13 Pro and 13 Pro Max, along with iPad Pros from 2020 onward.
That’s because the feature relies in part on a technology called light detection and ranging, or lidar for short. It’s a way for devices (and, more commonly, cars) to very quickly scan the space around them, and Apple only builds the parts that make this spatial awareness possible into its more expensive products.
Automatic image descriptions
Available on: Android
When is it coming? Available as a beta feature in Google’s Lookout app on June 2.
Three years ago, Google launched an Android app called Lookout as a sort of catchall toolbox for its blind and low-vision users.
If you open the app and point the camera at a snippet of text, for instance, the app will begin to read those words out loud. “Explore mode,” meanwhile, tries its best to describe what’s in front of the camera — it likes to describe the water bottle that perpetually lives on my desk as “tableware with text ‘The Washington Post.’ ”
Google’s latest addition to this toolbox is a feature that can describe what’s going on in certain kinds of photos. (Certain iPhone and iPad models have had a similar tool, called VoiceOver Recognition, for a few years.)
“Imagine reading a news story with a photo or receiving an image on social media that people are reacting to, but not knowing what’s in that image,” said Scott Adams, product manager for Lookout, in a video meant for app developers. “That’s where the potential of [machine learning] comes in.”
For now at least, you probably shouldn’t expect Lookout to correctly explain gritty, complicated scenes. Adams said that, beyond offering basic captions like “dogs playing in the grass,” the tool will be able interpret text — say, on a sign — and identify people in images by their “upper body clothing and whether they’re an adult or youth.” Even better, all of that image recognition will happen solely on your phone or tablet — no more sending info to far-flung servers.
Live captions for audio
Available on: iOS, iPadOS, macOS, Android
When is it coming? Available as a beta feature for Apple products later this year and for some Android devices now.
Certain Android devices have had the ability to automatically transcribe audio — be it from a podcast, movie or anything else playing on your phone — for a few years. (If your device uses Android 10 or later, you may already have it installed.)
Now, Apple is joining the club.
The company confirmed this week that a similar feature would soon arrive on iPhones, iPads and Mac computers as a beta feature later this year. Like on Android, Apple’s Live Captions will appear on-screen for just about any audio that a phone can play, but there’s at least one helpful difference here: group video calls in services like FaceTime will attribute those transcriptions to specific speakers, so users can keep up with the flow of conversation more easily.
Apple Watch mirroring
Available on: iOS
When is it coming? “Later this year,” Apple says.
You might think that a wearable like the Apple Watch would have to be poked and prodded to operate it, but that's actually not true — you can actually interact with one by clenching your hands and making pinching motions with your fingers.
But even those gestures aren’t manageable by some of Apple’s users, so the company built a feature that mirrors the action on the Watch’s tiny screen onto a connected iPhone.
That means people with motor disabilities and limited mobility can control those Apple Watches using the iPhone’s built-in tools. Voice Control, which first appeared in 2019, lets people navigate through on-screen menus, webpages and more by calling out very specific commands. Switch Control, meanwhile, lets its users interact with an iPhone using accessories like joysticks and sip-and-puff switches. Since this new feature replicates the Apple Watch’s screen on the iPhone, all of those controls can be used to interact with a Watch without ever having to touch it.
Custom sound recognition
Available on: iOS and Android
When is it coming? Later this year for Apple products. Google wouldn’t confirm a launch date for the new feature.
Deaf users and the hard of hearing have been able to use their iPhones and Android devices to receive visual alerts when they detect certain sounds — think breaking glass, crying babies or fire alarms. In time, though, both companies say people will be able to “teach” their phones to listen for more specific kinds of audio cues.
Think about it: there’s a very good chance that your microwave’s telltale ding doesn’t sound anything like your dryer’s. (Mine emits a sort of sickly, raspy buzz.) Rather than just lumping both of those into a single broad category of “appliance noises,” you’ll be able to train your phone to tell them apart and send you the right notification when your food is warm again.
Help Desk: Making tech work for you
Help Desk is a destination built for readers looking to better understand and take control of the technology used in everyday life.
Take control: Sign up for The Tech Friend newsletter to get straight talk and advice on how to make your tech a force for good.
Tech tips to make your life easier: 10 tips and tricks to customize iOS 16 | 5 tips to make your gadget batteries last longer | How to get back control of a hacked social media account | How to avoid falling for and spreading misinformation online
Data and Privacy: A guide to every privacy setting you should change now. We have gone through the settings for the most popular (and problematic) services to give you recommendations. Google | Amazon | Facebook | Venmo | Apple | Android
Ask a question: Send the Help Desk your personal technology questions.