“I'm sure some bad actors will try to game the system, but it's not as easy as you suggest,” Mosseri responded to one Twitter user who criticized the survey.
It’s worth noting this isn’t a rating system, nobody can opt into rating a publisher as trustworthy. We randomly sample new people each day, and only their responses are used. I’m sure some bad actors will try and game the system, but it’s not as easy as you suggest.— Adam Mosseri (@mosseri) January 21, 2018
There is always the risk that people will try and game any system. In this case we (1) randomly sample people, (2) actively work to de-bias the data so it's representative of the population and (3) re-run the surveys every day.— Adam Mosseri (@mosseri) January 20, 2018
The Facebook survey, which is part of the crowdsourcing initiative that Facebook chief executive Mark Zuckerberg unveiled last week, consists of two short questions with a choice of responses, according to a report by BuzzFeed News:
Do you recognize the following websites
How much do you trust each of these domains?
Not at all
Facebook has not disclosed how or where those questions would appear to users. The company declined to comment beyond Mosseri's tweets.
Some people were startled by the simplicity of the survey. They also noted the potential for it to reward partisan news outlets with loyal audiences or punish niche outlets and start-ups. But Mosseri tweeted that the survey was designed to recognize news outlets with broad recognition and trust among their users.
I understand that some people may balk at how simple a survey is, but complicated surveys can be confusing and bias signal, and meaningful patterns can emerge from broad surveys.— Adam Mosseri (@mosseri) January 24, 2018
It's actually not a simple ratio of trust to familiarity. We ask people if they're familiar with a publisher because, if they're not, it's not even worth asking if they trust the publisher. What we look at, and value, is how broadly trusted a publisher is.— Adam Mosseri (@mosseri) January 20, 2018
More than two-thirds of Americans get some of their news from social media, according to the Pew Research Center. That shift has empowered companies such as Facebook and Google, but it has also placed them in the challenging position of trying to decide what news they should distribute to global audiences.
The headline is misleading. We ask people what they trust, but don’t simply value more the publications that get positive replies. We specifically look for publications that are trusted by people with a wide range of reading habits, so trusted by many different types of people.— Adam Mosseri (@mosseri) January 20, 2018
How we incorporate survey data is every bit as important as the specific questions we ask.— Adam Mosseri (@mosseri) January 24, 2018
This specific change, which is focused on trust, shouldn’t effect small publishers one way or the other. This is because not enough people we survey will be familiar with small publishers for us to know if they’re trustworthy or not.— Adam Mosseri (@mosseri) January 24, 2018
Rasmus Nielsen, the director of research at the Reuters Institute for the Study of Journalism at the University of Oxford, said that the criticism directed at the survey may reflect a degree of elitism, as research shows surveys can produce meaningful signals of trust. But that doesn't mean Facebook's approach will be successful, he added. “Everything hinges on how this is incorporated with other signals and other sources of data,” he said.
Nielsen also expressed reservations about Facebook's wording in the survey questions. He said the company should consider additional responses that allow users to express no opinion or simply, “I don't know.”
Dean Eckles, a professor at MIT's Sloan School of Management and former data scientist at Facebook, said that it's likely the company would base the results on a combination of the survey results, behavioral data and other aggregate information. He also noted that Facebook's move to survey only a subset of users, and not everyone, is a safeguard against attempts to undermine the integrity of the results.
“Survey samples are hard to game in a lot of ways because you'd need a lot of dedicated people in order to game a survey,” Eckles said.
Even as Mosseri offered some insight into Facebook's thinking, some experts questioned the piecemeal release of information and the opacity of the company's potential changes to how users will consume news media online.
“There is a lack of transparency around how they are rolling out this process, and what forms of data they are drawing on,” said Daniel Kreiss, a professor at the School of Media and Journalism at the University of North Carolina at Chapel Hill. “It seems like a weird way to do it.”