Former defense secretary Harold Brown said yesterday that there is "no realistic prospect" that a "Star Wars" defense program could protect the U.S. population against all-out nuclear attack, even at a cost of $1 trillion. He called on President Reagan to give up on the ambitious and far-reaching space project.

Brown, who has dealt with space weaponry for more than 30 years as a physicist and government official, also estimated the costs, risks and feasibility of a full U.S. defense and two less ambitious strategic defense plans being considered in the executive branch as alternatives.

He complained of lack of clarity in the governmental and public debate about the Strategic Defense Initiative, dubbed "Star Wars," unveiled by Reagan in March 1983.

The major alternative plans, described by Brown in a paper for the Johns Hopkins School of Advanced International Studies, where he is chairman of the Foreign Policy Institute, are:

* A full-scale U.S. defense against nuclear attack, to replace the postwar reliance on "deterrence" -- the threat of nuclear retaliation -- as the basis for national security.

"Technology does not offer even a reasonable prospect of a successful population defense," he said. As one of several examples, he said the intensity of laser beams needed to attack enemy missiles early in flight would have to be "perhaps a million or more times greater than anything that has been achieved," in view of potential improvements to the missile systems.

A complete antiballistic missile defense, together with conventional air defense and civil defense necessary to make it airtight, "could approach a trillion dollars" in cost if practical, Brown said. That amount is about 3 1/2 times as large as the current U.S. defense budget.

At a news conference announcing his findings, Brown referred to the full-scale "Star Wars" plan, which has provided most of the political and public support for Reagan's initiative, as "an impossible dream."

* A limited program to defend land-based intercontinental ballistic missiles though a terminal defensive system.

The purpose of this would be to "ensure the U.S. strategic retaliatory capacity," he said. This, Brown noted, would mean "not defending people but defending missiles which threaten to destroy people" and thus would be the opposite of what Reagan has been discussing in advocating SDI.

Such a program, according to Brown, is "feasible with known technology" at a cost of $10 billion or more.

This program, like the others he laid out, would require U.S. abrogation of the 1972 ABM treaty with the Soviet Union. Brown said that no existing Soviet threat requires such a major defense of the U.S. retaliatory capability and that such a threat "is not likely to arise during this century if the U.S. pursues a strategic modernization program, generally along the lines of current plans."

Brown recommended continuation of "an extensive research, development and study program on antiballistic missiles" centered on protection of U.S. missile strike forces, to keep open the option to proceed later.

* A limited defense of cities against such small nuclear attacks as an accidental or unauthorized launch from the Soviet Union or an attack by a country or terrorist group possessing only a few nuclear weapons.

He compared this to the thin nationwide terminal defense system proposed in 1967 by President Lyndon B. Johnson. An alternative could be "a limited space-based system" giving a small degree of protection to the entire country.

Brown described this as a "damage limitation" plan to "hedge against the failure of deterrence." It would not stop a major or concentrated ballistic missile attack and would provide no defense against terrorists or small-country antagonists who smuggle a bomb into the United States or load one into low-flying aircraft.

Such a plan would cost "upwards of $100 billion," according to Brown. He said it "shouldn't proceed" in view of the narrow spectrum of threats it could counter.

A U.S. decision to pursue any of the plans would prompt Soviet countermeasures, Brown said, "certainly an increase in offensive missiles" and probably in bombers, cruise missiles and Soviet defensive programs. The result would be "a substantial competition which would leave both sides as least as insecure as before," he said.

Because large increases in offensive weapons are an answer to defensive improvements, Brown said, "I don't see any chance for an arms-control agreement with deep reductions [in offensive weapons] so long as the possibility of a strategic defense deployment is a very active one."

The former defense secretary in the Carter administration expressed understanding of the discomfort felt by Reagan and his predecessors with the strategy of deterring nuclear attacks on the United States through threat of massive retaliation. But he said practicalities leave "no choice."

In recognition of this, he said, Reagan should "reaffirm the current U.S. nuclear strategy." Reagan and other political and military leaders, Brown said, "should publicly acknowledge that there is no realistic prospect for a successful population defense, certainly for many decades and probably never."