President Reagan's Strategic Defense Initiative, which he said could render nuclear weapons "impotent and obsolete," may actually force the United States to develop a more powerful nuclear arsenal to overcome Soviet defenses, according to scientists at the federal nuclear weapons design laboratory here.

The scientists said that knowledge about SDI, a research program to determine whether missile defense is possible, remains too sketchy to allow firm predictions. But they suggested that if both superpowers build missile defenses, the United States might have to compensate by building heavier and more powerful nuclear warheads.

"If you face a wholly new threat, you must beef up your warheads to defeat that threat," Robert F. Perret, a physicist who heads Lawrence Livermore National Laboratory's systems analysis section, said in an interview. "You also have to assume degradation in accuracy. So what do you do to retain the effectiveness? You increase the yield."

The scenario outlined by scientists at the laboratory here, one of two that design nuclear weapons for the U.S. military, was among several views that challenge administration orthodoxy on the controversial SDI program, sometimes called "Star Wars." While most scientists here enthusiastically endorsed SDI research, much of which is being conducted locally, they also raised questions about the management of the $3 billion-a-year program:

*Perret said that a number of highly publicized and expensive experiments conducted by the Pentagon's SDI office appear aimed at winning public and congressional support rather than obtaining valuable scientific knowledge.

"There are good experiments and bad demonstrations," he said. "What I can answer is whether all of the demonstrations have supported good experiments. Some of them haven't."

*Perret and George H. Miller, who heads the lab's weapons program, said that the Pentagon may be setting an unrealistic timetable for SDI, thereby jeopardizing important research. Lt. Gen. James A. Abrahamson, head of the SDI Organization, has said that the government should be able to decide by the early 1990s whether to deploy missile defenses.

Miller said that Abrahamson's determination to prove practicality in the next few years may force him to "down-select" -- decrease funding for -- promising technologies.

"My personal opinion is, the broad-based research program that has been initiated must continue," Miller said. "My style of running the program would be to push back the milestones rather than making premature down-selects."

*Perret said that no missile defense should be deployed unless it is "cost-effective at the margin," meaning that it would cost the Soviets more to overwhelm the defense than it would cost the United States to maintain it. Paul H. Nitze, Reagan's senior arms control adviser, devised that formula, but the Pentagon has rejected it in recent months.

"The buy-in cost has to be affordable, and the maintenance cost has to be affordable," Perret said. "If he can overwhelm you through force attrition, you lose . . . . Nitze's right, and if they've changed their minds, I can't help that."

*Several experts at Lawrence Livermore said that no attempt to deploy missile defenses is likely to succeed unless accompanied by an arms control agreement with the Soviet Union.

"Right now SDI is a research program, and I think it should stay a research program until we get something that makes sense, and then we should sit down and negotiate," Paul S. Brown, the lab's arms control chief, said. "I don't think you can do it without arms control."

SDI is envisioned as a space- and ground-based system of weapons that would shoot down Soviet ballistic missiles. Scientists at Livermore are designing and testing lasers and advanced nuclear weapons that might form part of the shield. They also are investigating technologies to defeat a shield.

Opponents have said that deployment of defensive weapons could intensify the arms race as the Soviets build more warheads and different types of weapons to penetrate the U.S. defense. Miller said that must be carefully examined.

"While I'm very, very enthusiastic and optimistic about SDI, there are many, many questions we don't have the answers to," he said. "The whole SDI program right now is a study and analysis of move and countermove . . . . What could the Soviets do to counter? How easy is it for us to counter what they do? What are the cost trade-offs? In other words, we are still beginning to examine the issues."

Miller agreed with Perret that higher-yield nuclear weapons might be one response to a defensive system. If defensive weapons render missiles less accurate, the weapons might have to be more powerful to have the same destructive effect on "hard" targets such as missile silos or command bunkers and "soft" targets such as cities, he said.

"If you're trying to attack very hard targets, the loss of accuracy is probably not recoverable by a reasonable increase in yield," he said. "On the other hand, if you're attacking or defending cities, it might well be."

The lab is also investigating the possibility of maneuvering warheads that would evade defensive weapons and then home in on their targets, scientists said.