“Best application of ‘Deepfake’-style AI facial replacement I’ve ever seen,” wrote Twitter user Allan Xia.
Zao is only available to users with a Chinese phone number. After launching Friday, the app quickly gained popularity and was the most-downloaded iPhone app in China on Monday, according to data and analytics company App Annie.
But the app also quickly created controversy. According to RADII China, the app’s user agreement initially gave Zao widespread rights to use the photos and the videos generated with its technology free. The company later amended its terms of service.
Momo did not immediately respond to a request for comment.
While photoshopping has long made images susceptible to realistic-looking changes, deepfake videos are the latest frontier for manipulation as technology advances. Made popular by powerful and widely available artificial-intelligence software developed by Google, these realistic videos have quickly multiplied across the Internet.
Some of the more sinister uses have included revenge videos with women’s heads pasted into porn videos. And many researchers in the field are concerned about potential uses of this type of technology as the United States approaches the 2020 elections.
Zao’s popular app also raises privacy questions following the rise to the top of the app stores this summer of FaceApp, a program that takes photos of people and “ages” them using artificial intelligence. Privacy advocates quickly began warning about the Russian-made app’s vague legalese. Suspicions were raised that the app could be part of a disinformation campaign or secretly downloading photo albums. Leaders of the Democratic Party warned campaigns to delete the app “immediately.”
Among high-profile individuals who have had their own images weaponized against them is House Speaker Nancy Pelosi (D-Calif.), who was the subject of a video distorted to make it sound as though she were drunkenly slurring her words. The video and the speed with which it was disseminated highlighted the way made it clear that even simple manipulations could be used to shape public perceptions.
When Facebook said it would not remove the Pelosi video, stating that it did not violate any of its policies, two artists released a deepfake video of Facebook CEO Mark Zuckerberg bragging about misusing “stolen data” from users. That video also remained on the platform.
User @nikkmitchell tweeted he was “in absolute awe” of the app. “On a side note to the developers, please don’t be evil,” he added.