Ashcroft was 17 speeches into a national speaking tour defending the Patriot Act, a law expanding government surveillance powers that passed nearly unanimously in the wake of the Sept. 11, 2001, terrorist attacks. And all along the way, the librarians showed up to protest.
In the case of government surveillance, they are not shushing. They've been among the loudest voices urging freedom of information and privacy protections.
Edward Snowden's campaign against the National Security Agency's data collection program has energized this group once again. And a new call to action from the ALA's president means their voices could be louder and more coordinated than ever.
Guarding patrons' library activities is considered a core value of the profession, written into the ALA's code of ethics: "We protect each library user's right to privacy and confidentiality with respect to information sought or received and resources consulted, borrowed, acquired or transmitted."
Over the years, the U.S. government has tested the limits of how far librarians will go to defend that code. Near the end of the Cold War, FBI agents asked New York City librarians to watch for patrons who might be diplomats from foreign hostile powers trying to recruit intelligence agents or gathering intelligence. Library officials were alarmed.
"These things are so far removed from the professional duties of a librarian that I find it almost inconceivable that this whole thing is happening,'' Nancy Lian, head of the New York Library Association, said in a New York Times article at the time. A judicial order, she noted, would be required for such action.
Library groups were wary of the Patriot Act from the start, according to Emily Sheketoff, head the ALA's Washington office.
"We were engaged because we feared that the government would overstep," Sheketoff said, a fear she believes has been validated.
Given the political climate after Sept. 11, even individual legislators who said they agreed in principle didn't pull their support of the bill. "It was a steamroll that we couldn't even slow down," she said. After the bill passed, the group trained library employees around the country on how to respond if the government requested information.
Section 215 of the act - later used to justify the bulk collection of domestic phone metadata as revealed by Snowden - was called the "library provision." The implication was that the government could use it to get library records. By 2003, some libraries placed signs in their lobbies, warning patrons that the government could obtain their records under the bill. Hundreds of meetings were organized to discuss the privacy implications of the law at libraries around the country.
Stewart Baker, the assistant secretary for policy at the Department of Homeland Security under President George W. Bush, said it was a "disservice to public debate" that Section 215 was ever known as the library provision, arguing that there was no evidence it was used in the context suggested by library groups.
When Ashcroft's promotion tour rolled around, many of his talks took place behind closed doors while librarians and civil liberties advocates protested nearby.
The backlash to his comments was fierce. "It turns out people like and trust their local librarians," Sheketoff said. As a result, Ashcroft telephoned Carla Hayden, then-president of the association.
"I must say, it was quite something," Hayden said. "When you have the opportunity to express something to a person in that position, you want to do it well."
Hayden said she reminded Ashcroft that libraries were open to working with law enforcement through judicial channels. But, she said, she believed the Patriot Act went beyond that relationship.
Hayden said Ashcroft "expressed that he was sorry that he might have said something that could have been offensive - and that he didn't intend that." In fact, Hayden said, he told her that he had "good library experiences in his life."
Ashcroft did not respond to a request for comment for this story. Baker believes the apology came as a result of the librarians' media campaign.
But in his view, Ashcroft was justified. As he put it: "The implication of their campaign was that library records across the country were being rifled through by people in the name of national security trying to figure out who was reading books they disapproved of - that was pretty clearly what the parade of horribles suggested and that Section 215 was somehow implicated in that. And none of that is true."
The Connecticut Four
Within two years of Ashcroft's apology, four library officials in Connecticut found themselves in a legal battle over handing over patron information - and unable to talk about it.
In July 2005, two FBI agents presented George Christian with a national security letter - an information request that doesn't require judicial approval - seeking data about patrons' library use. He was the executive director of a consortium called Library Connection. Such letters existed before the Patriot Act and did not derive from Section 215. But their use was expanded by the law.
The government wanted to know who had used a device at an IP address in February, Christian said. The address was assigned to a router that served many devices, and to hand over that data could have violated the privacy of many patrons beyond the subjects of the FBI's investigation, said Christian, an accidental librarian who is trained as a software developer.
Christian questioned the constitutionality of the search. The most astounding part, he said, was the gag order attached to the letter. "It was perpetual, with no expiration date," he said. Such conditions are a standard feature of national security letters, requiring recipients to keep secret even the fact that they received a letter.
"Gag orders were made controversial," Baker said. "They're a routine feature of a lot of criminal and other investigations whenever you don't want the defendant or the suspect to know you're investigating them."
Instead of keeping quiet, Christian called an emergency board meeting. His colleagues unanimously decided to challenge the request. The American Civil Liberties Union agreed to take up the case. The group was already representing an Internet service provider fighting a similar request and was looking for a library to join the challenge.
Because of the gag order, none of the four library officials - Christian, Barbara Bailey, Peter Chase and Jan Nocek - could add their names to the suit. The four were known as John Doe throughout the case. During the first hearing, none could be in the courtroom, lest their identities be revealed. "We watched it from a closed-circuit television in Hartford," Bailey said.
Their identities eventually slipped to the media through faulty redaction in court documents related to the case. Even then, the four were unable to speak out - or testify about it during a debate over the Patriot Act's renewal.
"It was incredibly stressful," Christian said. The group was bound to silence until May 2006, when the government withdrew the request - and with it the gag order. That made the "Connecticut Four," as they have come to be known in library circles, among the only recipients of such an order who can publicly discuss the experience.
Library advocacy in a post-Snowden world
Snowden's leaks have only deepened the concern about the degradation of privacy rights. "Now we know that it really didn't matter what they passed," Sheketoff said. "What they were sweeping up was everything, way beyond what anybody had ever envisioned."
Librarians have deployed new methods to protect patron privacy.
"As technology has changed and we've moved from the card catalogue and paper records to electronic records, we are always looking to destroy the record as soon we can," Sheketoff said. "When you return a book, the record is destroyed so that when the government comes we can say that we legitimately only know what you have out at the time."
In Massachusetts, the local branch of the American Civil Liberties Union has partnered with some librarians to deploy services like anonymous browsing tool Tor that can shield patrons' activity from electronic snooping.
The ALA backs reform of surveillance laws - specifically the bipartisan Senate version of the USA Freedom Act, which would limit data collection under Section 215.
Allies in the civil liberties community praise the group's track record. As Kevin Bankston, policy director at New America's Open Technology Institute, put it: "Librarians are on the front lines of the information society, charged with providing easy access to information for all, and are often some of the first to raise their voices over emerging Internet policy issues, whether around online censorship, digital copyright, or most often post-9/11, overreaching surveillance."
ALA President Courtney Young has called for the group's 57,000 members to pledge an hour a week to advocacy - including reaching out to members of Congress.
For his part, Christian has grown less optimistic as the push to reform the law has dragged on.
"We are obviously all in trouble," he said. "What happened to us seems like kindergarten compared to the revelations from Snowden."