The fight over further U.S. aid to Nicaraguan rebels intensified yesterday on Capitol Hill as Democratic legislators, in an extraordinary confrontation with Secretary of State George P. Shultz, accused the Reagan administration of "red-baiting" and "demagoguery."

"You have a lot of nerve to come before this committee and criticize the members of this committee for being demagogues," Rep. Peter H. Kostmayer (D-Pa.) told Shultz after several rounds of stormy testimony before a House Foreign Affairs subcommittee. "It is really, after all, you and the president that began this escalation of rhetoric . . . . There has been a lot of red-baiting going on in the administration. I think it began at the White House and I hope it will end very soon."

Committee member Ted Weiss (D-N.Y.) accused Shultz of "twisting facts, distorting facts and misstating facts" about Nicaragua and compared the tactics to those used by the late senator Joseph R. McCarthy (R-Wis.).

A red-faced and angry Shultz called the charge "the ultimate perversion" and refused to answer further questions from Weiss until he apologized, which he later did.

Some panel members complained that administration officials were equating a vote against covert aid for the Nicaraguan rebels, also known as "contras," with a vote for communism in Central America.

The confrontation came on a day when a prominent government official acknowledged that the administration lacks the votes in the House to win approval of $14 million it wants for the rebels, whom President Reagan calls "freedom fighters" and "our brothers."

Langhorne A. (Tony) Motley, the State Department's chief Latin American expert, said at a White House press briefing that it was "a fair assessment" to say that the administration lacks the support in the House to overturn a congressional prohibition on further covert aid to the rebels.

He added that if Congress cannot be persuaded to reverse itself, the administration intends to abide "strictly" by the prohibition on direct or indirect aid to the contras.

Officials who discussed the administration's strategy on condition they not be identified said Reagan and Shultz were "trying to turn the heat up" on Congress but would not proceed with a formal request for aid to the rebels until after key votes next month on the MX missile, another administration priority.

These officials said the administration probably would not press for a vote on aid for the rebels before "the April-May time frame" because of the current lack of support for the issue in the House. Congressional opposition is currently so great that the administration has refused to discuss a lesser alternative, which one official said would mean "certain defeat" for any plan to revive the aid program.

While the administration was lobbying for public support yesterday, Nicaragua was conducting its own campaign for congressional backing. Hill sources said that Nicaragua's deputy foreign minister, Hugo Tinoco, visited at least 10 legislators to discuss President Daniel Ortega's proposal to have a congressional group visit Nicaragua and set the stage for renewed U.S.-Nicaraguan talks.