At his news conference Tuesday morning, Mayor Vincent C. Gray rolled out his latest batch of agency grades — the product of a nearly year-old partnership with local research firm newBrandAnalytics.
The monthly rollout of the Grade D.C. results has been generally met with a collective shrug from the local media corps, given that it’s some what less than clear what the grades represent: The firm — more commonly hired by hotels and restaurants — takes Internet data, including user surveys and social media messages, runs them though a proprietary filter and spits out a letter.
The process, mayoral spokesman Pedro Ribeiro said, is meant to measure the public’s perception of an agency and, as such, is “a really valuable tool to grade customer service.”
But activist Dorothy Brizill, questioning Gray Wednesday about the most recent results, identified a development that casts tremendous doubt on the utility of Grade D.C. as a gauge of public perception, let alone as an accountability tool: The Fire and Emergency Medical Services department, which just emerged from as bad a month of headlines as any agency has had in the 27-month-old Gray administration, emerged with a solid A+.
“I frankly find it quite questionable that FEMS would have a grade of A+ in the month of March,” Brizill said.
Said Gray, “What I find interesting is that there are a lot of people out there you say are dissatisfied but apparently they’re not expressing it.”
After the exchange, Ribeiro provided some additional data on the Grade D.C. system, which was hailed in a Wall Street Journal article last November. Since launching last June, the system has processed 7,776 “reviews” — about 4,700 from online surveys, the rest from social media messages, mainly tweets. In those reviews, newBrandAnalytics distilled 36,000 “insights” used to determine the grades.
One issue: While 15 city agencies are being graded, those agencies are not being reviewed evenly. In February, for instance, the system processed about 6,800 insights, but two-thirds dealt with one of four agencies: The Department of Transportation, the Department of Motor Vehicles, D.C. Public Library and the Office of Aging.
Some agencies in February were graded on the basis of only a few dozen insights. The Alcoholic Beverage Regulation Administration earned a B based on 31 insights; the Department of Health earned a D based on 88 insights; and FEMS earned a A based on 105 insights — an order of magnitude fewer than, say, the library system was judged on.
Ribeiro noted that the high FEMS rating might be explained by the generally positive feedback that individuals might be inclined to share after being assisted by a first responder, as opposed to the more negative portrayals the department has seen in media stories. He also said the system has been a valuable tool for helping improve customer-service-focused agencies like the Department of Consumer and Regulatory Affairs.
Perhaps the grades are internally useful, but without more complete disclosure of Grade D.C.’s inputs and algorithms, it’s hard to make the case that the public should take the grades (including the city’s overall grade of A-) with anything besides a sizable grain of salt.