Finding a Way to Review Surveillance Tape in Bulk

By Walter Pincus
Tuesday, March 10, 2009

An agency under the director of national intelligence is seeking to develop an automated computer program that could process millions of feet of videotape, such as surveillance-camera data from countries other than the United States, according to a report released last week.

The goal is to identify "well established patterns of clearly suspicious behavior" of individuals outside the United States.

The research program, called Video Analysis and Content Extraction, has been underway since 2001 and is being undertaken by the Office of Incisive Analysis, part of the government's Intelligence Advanced Research Projects Activity (IARPA). It is one of several IARPA research projects aimed at developing systems that would permit subject-based review of massive video and other databases for counterterrorism and other intelligence purposes.

Information about the video analysis research program was contained in a mandated annual report on data mining sent to Congress last week, with a declassified version posted on the Web site of the Office of the Director of National Intelligence (under "News and Information" and then "IC Policy").

The report provides a slight hint of the enormous worldwide data collection being undertaken by U.S. intelligence agencies in the fight against terrorism.

Most of the massive database searches conducted by U.S. agencies start with a known or suspected terrorist or "inputs associated with a specific individual or group of individuals" under a computer process called "link analysis." They are not covered by the Data Mining Reporting Act, which requires disclosure of only "pattern-based" searches designed to discover "a predictive pattern or anomaly indicative of terrorist or criminal activity," according to the statute.

The video analysis research program is to replace what the report calls "a very tedious, generally manual, process of reviewing video for content that is potentially of intelligence value." The researchers are trying to create an automated system that would detect objects and events, understand them and then classify them by category and index them. It also would be able to browse through and retrieve videos. Some elements developed under the program have been sent on for testing in real situations.

The video material being used in the research is "lawfully collected [video] data from public places outside the United States and information from public media sources," according to the report. Although no further description is offered in the unclassified version of the report, many European, Asian and Middle Eastern countries have extensive video surveillance systems covering public transportation stations, shopping areas, streets and major buildings.

The report says simulated video content also is being employed, using volunteers who agreed to appear for testing purposes. The intelligence researchers have been part of a public program run by the Commerce Department's National Institute of Standards and Technology, which is openly developing both manual and automated systems for video-surveillance event detection, search and retrieval capabilities.

Video search and retrieval has already become an active part of the Defense Department's intelligence operations overseas, thanks to unmanned and manned aircraft that operate almost round-the-clock in Iraq and Afghanistan. One current use is to retrieve tapes where attacks or road bombings have occurred in order to determine whether individuals who were at the scene were participants.

The new report also discusses a program called Reynard, which it describes as "a seedling effort." It began by studying the feasibility of monitoring activity in virtual worlds and online games to attempt modeling of what terrorist activity in those worlds would look like in the real world.

Now it is trying to contract with social science researchers to encourage study of virtual- world behavioral indicators that could serve to identify real-world attributes of individuals or groups, based on such characteristics as gender, age, economic status, ideology or worldview. The long-term concern is that terrorists may be using the virtual world or video games to send messages.

Several past IARPA programs have been discontinued, but they give an indication of how wide-ranging the database searches are. One, called Knowledge Discovery and Dissemination, sought to match patterns of known entity deception drawn simultaneously from various "lawfully collected foreign data bases."

Another, called Tangram, attempted to determine the feasibility of conducting continuous surveillance of suspects to detect any pattern changes that could warn of an elevated risk. A third, called Paint, for "proactive intelligence," examined activities of enemy targets, not just terrorists, to try to find any distinct patterns that would indicate "nefarious activity," according to the report. Elements of this program have been moved into a "proof of concept" phase.

National security and intelligence reporter Walter Pincus pores over the speeches, reports, transcripts and other documents that flood Washington and every week uncovers the fine print that rarely makes headlines -- but should. If you have any items that fit the bill, please send them to

© 2009 The Washington Post Company