Tensions in Congress over U.S. policy in Nicaragua, already high because of recent and pending critical votes, are being pushed higher by news of heavy fighting along Nicaragua's borders and by not-so-subtle administration reminders that the U.S. military is ready for anything.

President Reagan has been telephoning House and Senate members to urge approval of nonlethal aid to antigovernment rebels, who are retreating into Costa Rica and Honduras from an apparent all-out effort by the leftist Sandinista government to eradicate them.

The Sandinistas' drive is logical from their point of view if renewed U.S. aid is likely, because the counterrevolutionaries, or contras, would be stronger later than they are now.

But the contras' retreat has provided Reagan with a backdrop for wielding his worst-case scenario: an eventual need for U.S. troops to halt leftist expansion in Central America.

According to several congressional offices, Reagan is arguing that failure to back the rebels now could cripple them permanently and that U.S. troops would then be the only option in the event of Sandinista aggression against Costa Rica or Honduras. The Sandinistas' "hot pursuit" of rebel forces into those nations has alarmed their governments, and Costa Rica asked the Organization of American States on Friday to send an investigating commission.

U.S. critics are arguing that aiding the rebels will increase, not decrease, the danger of ultimate U.S. troop involvement in defense of a hopeless cause and that the only solution lies in a settlement.

Reagan's argument is not new, but this is the first time it has been used in the context of a major military clash.

U.S. policy in Central America has been forged since 1981 as a test of whether it is possible to halt Soviet advances in an underdeveloped area without sending in the Marines.

Officials have long painted the rebels as the only way short of U.S. troops to put real pressure on Nicaragua.

But Reagan has nonetheless been careful never to rule out direct intervention and has prepared for it as a warning to Nicaragua not to get cocky about U.S. reluctance to send troops.

Secretary of State George P. Shultz was restating this position last month when he said, "Direct application of U.S. military force must be recognized as an eventual option, given our stakes in the region." Some administration officials were quoted in a New York Times series last week as outlining U.S. readiness for action and as being confident that a U.S. strike could be quickly successful. Military leaders have previously rejected that position.

The administration has been open about wanting Nicaragua to be nervous. It has seized even petty opportunities to harass the Sandinistas, from delaying or denying travel permits to rattling windows in Managua with sonic booms from low-flying jets.

Reagan last week ridiculed President Daniel Ortega, calling him "a little dictator who went to Moscow in his green fatigues to receive a bear hug."

Nicaragua has responded on cue. Ortega has predicted a U.S. invasion at least a dozen times. He helped verify Reagan's scenario in April by seeking increased Soviet help just after Congress refused to rearm the contras against him, and he has escalated his rhetoric of defiance. Nicaragua has few defenders left in Congress.

Still, the Senate voted to authorize $38 million in nonlethal aid to the contras last week by a much smaller margin than its promoters or critics had expected. The 55-to-42 tally represented only two more votes in favor of aid than were cast in April's 53-to-46 count, the closest vote the Republican-controlled body had cast on that issue.

The narrow split reflected recent polls showing that the public is still overwhelmingly opposed to U.S. aid to the rebels. It has renewed the energies of both sides for House votes next week and for a further clash in the Senate on appropriating the funds that now have been authorized.

Rep. Lee H. Hamilton (D-Ind.), chairman of the House intelligence committee, said last week that U.S. policy is at "a critical juncture." Hamilton heads the list of Democrats sponsoring a measure to provide closely controlled aid to Nicaraguan refugees -- not to the contras -- on grounds that the contras cannot succeed militarily and that more U.S. backing only guarantees that U.S. troops will ultimately be required to prevent the contras' annihilation.

"As they grow in strength with U.S. aid , so will our commitment to them and to their goal of overthrowing the Sandinistas," Hamilton said. "Funding the contras is not an alternative to intervention but an avenue to intervention."

By rejecting renewed talks with Nicaragua and giving lukewarm backing to the regional Contadora peace process, Reagan may be closing off options other than force, Hamilton and other critics have argued.

The administration counters that Nicaragua, by refusing to talk with the contras and giving no ground in regional discussions, may be doing the same thing.

The House on Tuesday is to consider several alternatives to aiding the rebels, including postponing the debate another six months. That is designed to give the Contadora group more time to work. Meanwhile, the fighting in Central America continues.