To analyze the data, we can draw on artificial intelligence and especially “deep learning” neural network computer algorithms. These tools analyze the visual content of advertisements in the same way we are able to analyze text.
To explore this possibility, I converted all 267 ads into a sequence of images, one per second, and fed them through the new Google Cloud Vision API. The Vision API uses Google’s deep learning algorithms to tag each image with the major objects, activities and themes it depicts, extract any recognizable text, estimate the geographic location it captures, and identify the presence and emotional expression of any human faces.
At present the Vision API is still experimental and entirely automated. It is not perfect, and any errors represent algorithmic errors, not editorial statements. Moreover, the Internet Archive is monitoring only a subset of American television stations and does not always catch every advertisement, so the results here are not exhaustive. With those caveats, here are some initial findings.
Politics is about people and that certainly plays out in these advertisements, with 7,230 seconds (62 percent) of airtime featuring people. In all, 2,172 seconds (18 percent) featured at least two people and 1,152 (10 percent) featured three or more people.
Nearly a third of the airtime featuring people was of candidates talking with groups of potential voters, while the rest focused on a single person, either the candidate or someone supporting or criticizing the candidate. New Jersey Gov. Chris Christie (R) holds the record for the most people in a single shot, totaling 40 faces at one point.
The Vision API also estimated that about 0.4 percent of faces showed anger or sorrow, while nearly 14 percent showed joy, in keeping with the global prevalence of joyful expressions in news imagery.
Textual overlays are very common in political ads, with at least 7,000 seconds (60 percent) of air time containing a textual message of some form, ranging from large font bold overlays to the tiny font disclaimers at the end of ads.
It is common for ads to run a brief message to frame the context of a video clip appearing immediately before or after. For example, an American Crossroads video opens with red text over a black background saying, “What is it with Hillary Clinton lying about terrorists and videos?” before playing a clip of Clinton speaking at the CNN Democratic Debate and then a CSPAN clip of the mother of one of the Americans killed in the attacks in Benghazi, Libya, in 2012.
Textual overlays also are frequently used to summarize or emphasize particular statements made by the candidate, such as the words “Strengthening the economy” that appear as Democratic presidential candidate Hillary Clinton says them in this Hillary for America ad.
Logos are somewhat common in political ads. The Vision API recognizes 538 distinct logos appearing across 986 seconds (8 percent) of airtime. The most frequently appearing logos tended to be those of major news organizations, including Fox News Channel, CSPAN, Fox Business, CNN and the New York Times. These tended to be clips of the candidate or an opponent from previous news interviews or speeches. Nationally recognizable locations are featured in 178 seconds of video (1.5 percent), with the top two locations being the U.S. Capitol and the White House.
The advantages of automation are numerous. It would require an army of volunteers watching nearly a hundred television stations 24/7 to identify every airing of every political ad. Similarly, although a single person could watch the current archive of three and a quarter hours of ads, creating second-by-second annotations would have required a very large team of human coders.
This is why the Internet Archive’s audio fingerprinting and Google’s algorithms are useful. This is the big data future of understanding political advertising on television.
To learn more, you can download the JSON data and run your own analyses.
Thank you to Google and the Google Cloud Vision API team for the use of their tools and the Internet Archive’s Television News Archive for making the ad archive available.