When President Reagan's advisers met three weeks ago to give final approval to naval exercises in the Gulf of Sidra, Secretary of State George P. Shultz said the potential for conflict with Libya could help the United States accomplish a larger goal.

According to a participant, Shultz said he wanted the soldiers manning the radar at a Libyan missile site to begin asking themselves "if they really want to follow" their leader, Muammar Qaddafi, into a military clash with the United States.

The secretary's remark underscores a critical but little-mentioned goal of the Reagan administration's recent actions toward Libya -- to build enough pressure on Qaddafi from within and without to destabilize his regime eventually.

The Reagan administration has insisted publicly that it only wants Qaddafi to change his ways and halt his support for international terrorism and his aggression in the region. And the recent naval crossing of Qaddafi's "line of death" in the gulf was justified primarily as a demonstration of freedom of navigation.

But privately, senior officials say that the exercises and other actions toward Libya have a broader goal of eroding Qaddafi's standing at home.

The officials said reports reaching the administration suggest that Qaddafi faces growing internal unrest from the military and his people. While the information is sketchy, they said, it has encouraged the administration.

According to one high-ranking policy-maker, pressure on Qaddafi has intensifed in recent months, with increasing isolation of Libya by other nations, the conflict in Chad, the effects of the U.S. economic boycott and the casualties inflicted at the missile site at Surt and in the gulf last week by the U.S. Navy.

"They are all cumulatively beginning to take their toll," said the official, who is familiar with intelligence reports on the matter and spoke on condition he not be identified. "The professional military establishment in Libya is increasingly wondering whether it makes a great deal of sense to follow Qaddafi's leadership," he added.

The official said Qaddafi "seems to be shedding his professional support groups and recruiting ideologues" and loyalists for the government. "The few professionals are on the way out. That is a sign of nervousness and anxiety.

"Even his own tribal base is no longer satisfied with his position," the official remarked.

"There is something about the structure of these regimes. . . they operate from a very narrow base," he said. "This puts them at a disadvantage from the beginning."

The Libyan people "don't like thugs running around the country telling them what to do," he said, referring to Qaddafi's "revolutionary councils." He also mentioned food shortages and said Libyans also "don't like being ordered on suicidal military missions."

"They don't like being ordered into Chad to satisfy his [Qaddafi's] pretensions of grandeur," he added. Qaddafi has been lending logistical support to rebels seeking to overthrow the government of President Hissene Habre; recently French warplanes bombed a Libyan-built airfield in northern Chad in retaliation for Qaddafi's support of the rebels.

There have been signs of unrest in Libya in recent years, including a series of coup attempts. In 1984, at least two dozen people who were apparently part of a plot aimed at Qaddafi were killed in a shootout with Libyan security forces, according to reports reaching Washington at the time.

Later that spring there were more reports of clashes between dissidents and Libyan forces. Also, the public hanging of two Tripoli students was reported to have fueled open dissent among thousands of students forced to watch.

In 1985, a segment of the Libyan military reportedly launched two assassination attempts against Qaddafi, who responded by executing dozens of officers.