Washington’s effort to recover American hostages held overseas is “dysfunctional” and mired in failures hidden by bureaucracy, an Army Special Forces officer once involved in the Pentagon’s part of the mission told the Senate on Thursday during a hearing for whistleblowers.

Lt. Col. Jason Amerine testified that he started working on hostage policy at the Pentagon in early 2013. At some point, he became frustrated with the inaction to free Americans and said he took his concerns to Rep. Duncan D. Hunter (R-Calif.), a member of the House Armed Services Committee, after exhausting all other options. The Army, when it learned about his discussions on Capitol Hill, responded by removing him from his job, suspending his security clearance and launching a criminal investigation into his actions, Amerine said.

“My team had a difficult mission and I used all legal means available to recover the hostages,” Amerine said in prepared testimony. “You, the Congress, were my last resort. But now I am labeled a whistleblower, a term both radioactive and derogatory. I am before you because I did my duty, and you need to ensure all in uniform can go on doing their duty without fear of reprisal.”

Amerine testified before the Senate Homeland Security and Governmental Affairs Committee during a hearing called “Blowing the Whistle on Retaliation: Accounts of Current and Former Federal Agency Whistleblowers.” He first acknowledged facing an Army investigation and communicating his concerns about U.S. hostage policy to Hunter in a Facebook post on May 15.

The case pits one of the first heroes of the Afghanistan War against the Army. Amerine led a Special Forces team there in 2001 that protected future Afghan President Hamid Karzai. Amerine was wounded by an errant American bomb on Dec. 5, 2001, that killed three other Special Forces soldiers. He later received the Bronze Star with “V” and the Purple Heart, and was labeled by the Army as a “Real Hero” in the 2006 version of its popular video game, “America’s Army.”

The Army Criminal Investigation Command (CID) has disputed that it is investigating Amerine as an act of reprisal, and declined to say whether the soldier is under investigation at all. A spokesman for CID, Chris Grey, declined to comment Thursday on Amerine’s testimony.

Amerine told the Senate that in 2013, his office at the Pentagon was asked to help recover Sgt. Bowe Bergdahl, the U.S. soldier held captive for five years overseas and charged earlier this year with desertion. He was later recovered May 31, 2014, in a controversial swap for five Taliban officials.

“We audited the recovery effort and determined that the reason the effort failed for four years was because our nation lacked an organization that can synchronize the efforts of all our government agencies to get our hostages home,” Amerine said. “We also realized that there were civilian hostages in Pakistan that nobody was trying to free, so we added them to our mission.”

Amerine’s team worked to develop a viable trade for Bergdahl, bring the Taliban to the negotiating table and fix the interagency recovery efforts, he said. He “used all legal means” available to recover the hostages, and then went to Congress when he ran out of other options, he said. That prompted the FBI to complain that he was sharing classified information, he said. The Defense Department inspector general later determined he did not, he added.

Amerine credited the Defense Department inspector general with handling a reprisal complaint he filed well. The FBI has previously declined to comment on Amerine’s allegations.

Amerine credited Hunter with influencing the Pentagon to appoint Michael D. Lumpkin, a retired Navy SEAL and current deputy undersecretary of defense, as the Defense Department’s hostage recovery coordinator. Doing so allowed the Pentagon to respond quickly when a deal was struck to recover Bergdahl in exchange for five Taliban officials, Amerine said.

But the other civilians held hostage — including Warren Weinstein, who was accidentally killed in a U.S. drone strike in January — were left behind, Amerine said. One of the options Amerine’s team developed would have swapped seven Westerners for one Taliban drug trafficker and warlord: Haji Bashir Noorzai. He was convicted of drug trafficking and sentenced to life in prison after being lured to the United States in 2005.

Amerine told the Senate that the trade developed would have freed Bergdahl, Weinstein, Canadian Colin Rutherford and a family of three: Canadian Joshua Boyle, his American wife Caitlan Coleman, and their child born in captivity. He declined to identify the seventh hostage.

It’s unclear how the Noorzai swap would have worked. Bergdahl was held by insurgents affiliated with the Taliban, while Weinstein was held captive by al-Qaeda.

“Is the system broken?” Amerine asked. “Layers upon layers of bureaucracy hid the extent of our failure from our leaders. I believe we all failed the commander in chief by not getting critical advice to him. I believe we all failed the secretary of defense, who likely never knew the extent of the interagency dysfunction that was our recovery effort.”

Hunter said on the House floor last month that Amerine was critical in providing information that helped craft a congressional amendment that would require President Obama to appoint a specific federal official to oversee all hostage recovery efforts.

“Lieutenant Colonel Jason Amerine has worked with my office now for about two years on this amendment, and he is someone that really cares,” Hunter said. “He’s been working hostage stuff with about every government agency that there is, and he played a big role in getting this to where it’s at now.”

Amerine said he “failed” Weinstein and the four other Western hostages still in captivity.

“We must not forget: Warren Weinstein is dead while Colin Rutherford, Josh Boyle, Caitlin Coleman and her child remain hostages,” Amerine said. “Who’s fighting for them?”