At the height of the controversy about Soviet combat troops in Cuba, a senior U.S. official privately warned administration policymakers that the chief threat to the United States in the Caribbean is not Cuba but the region's "economic and social instability."

In an extensive report to Secretary of State Cyrus R. Vance, Philip C. Habib, formerly Vance's under secretary for political affairs, said this instability is creating "targets of opportunity" for Cuba to extend its influence through the region stretching from Central America in the west to the ministate islands of the eastern Caribbean.

Habib's report, made at Vance's request, stressed that the best way for the United States to combat the danger lies not in a stepped-up Caribbean military presence but in better use of U.S. economic aid and political cooperation to help the countries of the area achieve greater internal stability.

In his television report on Monday on the Cuba situation, President Carter emphasized the regional and worldwide military measures being readied by the United States to counter hostile forces in Cuba.

But, in listing these steps, Carter also said: "We will increase our eco- nomic assistance to alleviate the unmet economic and human needs in the Caribbean region and further to ensure the ability of troubled peoples to resist social turmoil and possible communist domination."

Although Carter did not elaborate, reliable sources familiar with the still-secret Habib report said it is being used as the springboard for a new administration approach to the Caribbean, the first result of which will be the unveiling within the next few days of a request to Congress for a sizable supplemental increase in U.S. financial aid to the region.

According to the sources, this aid will be directed principally toward the two poles of the region regarded as most unstable and most in need of immediate help: Nicaragua, which has been torn apart by a bloody civil war, and the eastern Caribbean, where high unemployment and unrest have been threatening to touch off a chain reaction of swings toward pro-Cuban radicalism.

The sources said the amounts involved in these new aid requests are still the subject of lively debate within the administration. But, they added, the proposed package for Nicaragua now appears likely to be about $75 million; and, they continued, despite some pressures for a holddown, the amount earmarked for the island ministates could go as high as $25 million.

In seeking this increased aid, the administration will be taking another step toward a priority attention for the Caribbean that began even before the furor broke out last summer over whether the Soviet Union has a combat brigade in Cuba.

Its principal impetus was the capture of power in Nicaragua by the Sandinista guerrila movement, which contains strongly Marxist, pro-Cuban elements. Despite fears in the Pentagon and the U.S. intelligence community about Nicaragua becoming a Cuban-dominated base for subversion in Central America, the administration has decided to gamble that U.S. aid and friendship can help guide Nicaragua toward a more moderate, politically pluralistic course.

Adding to official U.S. concern was a coup last March in the eastern Caribbean island of Grenada, which brought to power a government that has taken an increasingly pro-Cuban line. That triggered fears that what happened in Grenada could be repeated in the neighboring islands of Dominica, St. Lucia, St. Vincent, Antigua and St. Kitts-Nevis.

These ministates have a combined population of only about 540,000. But they sit aside the strategically important eastern door to the Caribbean, and there is concern within the administration that if they tumble into the Cuban orbit, the ripple effects could extend to such larger islands of the English-speaking Caribbean as Jamaica, Trinidad and Barbados.

It was this concern that prompted Vance to ask Habib, who had been forced to give up the under secretary's job after suffering a heart attack, to undertake a special study of the eastern Caribbean. The sources said his findings, gathered on an August trip through the region, have been incorporated into the administration's larger policy of dealing with the entire Caribbean basin.

Even before Habib undertook his mission, the administration, in the fiscal 1980 budget still awaiting approval by Congress, had asked for an increase in economic aid to the Caribbean from $124.7 million in fiscal 1979 to $155.2 million.

In addition, after the Grenada coup, the administration got approval for a special $7.5 million fund to be pumped into the eastern Caribbean for immediate starts on projects, such as roads and hospitals, that will both meet pressing human needs and provide high levels of employment.

However, the sources said that Habib, in his report, cautioned against attempts to push aid levels in the Caribbean to a point too high for recipient countries to absorb them.

Instead, the sources continued, he recommended that all of the federal agencies involved in the Caribbean undertake a review of the U.S. aid program to find what kinds of programs would be most effective and how the funds can be channeled in ways that would cut down on corruption, delays and inefficiency.

In addition, the sources said, Habib cited the need for Washington to focus more closely on problems of internal security and "ideological direction" in the English-speaking Caribbean.

By that, the sources explained, he meant that these former colonies, which used to depend on Britain for their security and which inherited the British parliamentary system of government, have been left vulnerable to outside subversive pressures by weak, ineffective police forces and by growing doubts about whether the so-called "Whitehall system" serves the political, social and economic needs of their populations.

The sources said these are problems more difficult to deal with than relatively straightforward questions of economic aid and development. But, they added, the State Department already is beginning to study ways in which the United States can establish better training and cooperation ties with Caribbean police forces and encourage exploration of political and economic models other than that offered by Cuba.