Harold Brown, the defense secretary in the Carter administration who was mandated to cut military spending but instead laid some of the groundwork for the U.S. arms buildup of the 1980s and who helped oversee a disastrous military raid to rescue U.S. hostages in Iran, died Jan. 4 at his home in Rancho Santa Fe, Calif. He was 91.

The cause was pancreatic cancer, said a daughter, Deborah Brown.

A onetime physics prodigy who earned a doctorate at 21, Dr. Brown became the first scientist to head the Pentagon. His predecessors had been business, political or military leaders accustomed to the ways of massive bureaucracies.

Dr. Brown built his initial reputation as a nuclear weapons designer at what is now the Lawrence Livermore National Laboratory in California. He went on to direct the laboratory, replacing his mentor Edward Teller, the Hungarian-born physicist widely recognized as the “father” of the hydrogen bomb. That position and others later held by Dr. Brown made him a central figure in the U.S. defense establishment during the Cold War.

In 1961, he became one of Defense Secretary Robert S. McNamara’s team of bright young “whiz kids.” At 33, he was director of defense research and engineering, the third-ranking civilian at the Pentagon. From 1965 to 1969, he was secretary of the Air Force.

Over the decades, he was regarded by colleagues as brilliant, quick to understand a broad spectrum of difficult political and military issues, and supremely confident in his analysis when making hard decisions that probably would cost him friends. In a memoir, former president Jimmy Carter praised his “technical competence” and called him one of his finest Cabinet officers.

Dr. Brown could also seem arrogant and distant. Gen. Colin Powell, who served as chairman of the Joint Chiefs of Staff and then as secretary of state, had worked for Dr. Brown as a younger officer.

“I always had the impression that Brown would be just as happy if we slipped his paperwork under the door and left him alone to pore over it,” Powell wrote in his autobiography “My American Journey.”

In the turbulent years of growing conflict in Vietnam, Dr. Brown defied easy categorization as a hawk or a dove. He earned the nickname “Dr. No” for his role in helping scrap the B-70 strategic bomber and the Skybolt air-to-surface missile. As the civilian leader of the Air Force, he appeared to recommend the intensification of aerial bombing of North Vietnam, one of the most controversial policies of the war.

He left the Pentagon in 1969 to lead the California Institute of Technology after Richard M. Nixon’s election as president. Dr. Brown returned to the Defense Department when Carter, a Democrat, entered the White House in 1977.

Dr. Brown spent most of his four-year tenure mediating between Carter, who in the wake of the Vietnam War fiasco promised to eliminate Pentagon waste and reduce defense spending by 5 percent, and a military establishment that demanded more firepower to counter threats coming from the Soviet Union and Middle East.

“I think of myself as a pragmatist with a world view,” Dr. Brown told Time shortly before he was sworn in as defense secretary in 1977. “I believe in a strong defense; I don’t believe that defense is all there is to national security. Economic strength, political cohesion, good relations with allies are equally part of national security.”

Soviet threat, Iranian crisis

In his 2006 book “SECDEF: The Nearly Impossible Job of Secretary of Defense,” foreign policy scholar Charles A. Stevenson wrote that Dr. Brown’s biggest political triumph was keeping the Joint Chiefs of Staff onboard amid Carter’s initial streamlining.

The downsizing included ceding control of the U.S.-built Panama Canal to the Panamanian government, the partial withdrawal of U.S. troops from South Korea and the cancellation of the B-1 bomber. The B-1 was revived under Carter’s successor, Ronald Reagan.

As a Pentagon official in the early 1960s, Dr. Brown had worked on the B-1, which was designed to replace the aging fleet of B-52s and become the military’s main strategic bomber. But he came to view its $100 million price tag as too high, feared it would be vulnerable to air defenses, and agreed with Carter that upgraded B-52s armed with cruise missiles could keep America safe.

But by the late 1970s, it was widely believed that the Soviets had achieved nuclear parity with the United States, and further cutbacks were seen as politically untenable.

In 1979, Soviet troops invaded Afghanistan, prompting U.S. sanctions against Moscow. The same year, the shah of Iran, a key U.S. ally, fled as Islamic radicals took control of the oil-rich nation and seized 52 U.S. diplomats and military personnel, who were held for 444 days.

Those events, coupled with concerns about the readiness of post-Vietnam U.S. military forces, prompted Dr. Brown to push Carter toward a more hawkish stance and greater defense spending.

On Dr. Brown’s advice, Carter approved development of the land-based MX missile system, a new ballistic missile submarine program and radar-evading stealth technology for aircraft. In advocating a stronger North Atlantic Treaty Organization, Dr. Brown also helped persuade several Western European nations to deploy U.S.-made Pershing II and land-based cruise missiles to counter new Soviet weapons.

Many of the weapons developed under Carter came to fruition under Reagan, a Republican who claimed national security had been dangerously compromised by Carter and defeated him in the 1980 election. The Reagan-era military buildup is widely credited with precipitating the collapse of the Soviet Union, which lacked the resources to continue the arms race.

“Within a few short years under Reagan, the military was in much better shape,” said Michael O’Hanlon, a senior fellow at the Brookings Institution . “But a lot of what the military brought to the table was researched and developed by Brown when he was defense secretary.”

Dr. Brown’s other main areas of influence were on China policy and nuclear strategy. In 1979, around the time Carter normalized diplomatic relations with China, Dr. Brown traveled to Beijing. The visit led to the establishment of electronic monitoring stations in western China that allowed the Pentagon to collect intelligence on Soviet space launches and ballistic-missile tests.

Since the early 1970s, Dr. Brown had been involved in Strategic Arms Limitation Talks between the United States and the Soviet Union. He took part in negotiations that led to the SALT II agreement in 1979. But after the Soviet invasion of Afghanistan that year, Carter withdrew the treaty from Senate consideration.

Despite his support of arms treaties, Dr. Brown said the United States must maintain its nuclear triad of land- and submarine-based ballistic missiles and strategic bombers to deter the Soviets. And amid concerns that Moscow was preparing for a prolonged nuclear confrontation, Dr. Brown urged Carter to modify U.S. nuclear policy.

The new doctrine held that in the event of a Soviet first strike, the United States would respond so forcefully as to persuade Soviet planners to quickly terminate their attack or risk almost guaranteed annihilation. This so-called “countervailing” strategy was adopted and refined by later administrations.

By all accounts, Dr. Brown’s biggest failure as defense secretary was the botched effort in April 1980 to rescue the U.S. hostages in Tehran.

Called Operation Eagle Claw, it was to be an audacious, two-night raid involving two staging areas in the Iranian desert and personnel from four service branches. With little hope of a diplomatic breakthrough to free the Americans, Dr. Brown called it “the best of a lousy set of options” and recommended going forward.

Amid mechanical problems, a dust storm and the collision of a helicopter with a transport plane that killed eight U.S. servicemen, the raid was aborted. The incident highlighted deficiencies in the U.S. military command structure that Dr. Brown had tried to address, and it led to reforms in the 1980s that improved joint operations. But in the immediate aftermath, it damaged U.S. prestige and helped doom Carter’s reelection bid.

As Dr. Brown wrote decades later in his book “Star Spangled Security,” he “considered the failed rescue attempt my greatest regret and most painful lesson learned.”

Harold Brown was born in New York City on Sept. 19, 1927. His father was a lawyer, his mother a homemaker.

He showed a precocious interest in technology. At 4, he removed the back panel of a refrigerator to see how the motor worked. To help her introverted son break out of his shell, his mother encouraged him to take up swimming and tennis.

“I was devoted to books and to working on lessons,” Dr. Brown later told the Los Angeles Times. “I was rather socially maladjusted.”

He graduated from the Bronx High School of Science at 15 with top marks and subsequently earned bachelor’s, master’s and doctoral degrees in physics from Columbia University.

After leaving the Defense Department, Dr. Brown spent many years in Washington as chairman of the Foreign Policy Institute at Johns Hopkins University’s School of Advanced International Studies and taught at the Center for Strategic and International Studies.

In 1953, he married Colene McDowell. She died in October. Survivors include two daughters, Deborah Brown of New York and Ellen Brown of La Jolla in San Diego; a sister; and two grandchildren.

In 1981, Carter awarded Dr. Brown the Presidential Medal of Freedom, the nation’s highest civilian honor. Dr. Brown received the U.S. Energy Department’s Enrico Fermi Award in 1992 for his lifetime contributions to national security. He had a long association with the Rand Corp. in California and served on its board of directors from 1982 until his death.

The burdens of managing the Pentagon had worn down many of Dr. Brown’s predecessors. As a reminder, he adorned his office with a portrait of James Forrestal, the first defense secretary, who in 1949 collapsed under the strain and jumped to his death from a 16th-floor hospital window.

Dr. Brown never appeared ruffled by the pressures of his job. But two months after leaving the Pentagon, he vented some of his frustrations in a speech with a telling title: “ ‘Managing’ the Defense Department — Why It Can’t Be Done.”