The House met in an extraordinary secret session yesterday to begin debate on the Reagan administration's escalating "secret war" against the leftist government of Nicaragua, now perhaps the most controversial foreign policy issue of this pre-election year.

The debate came as the Navy announced it was dispatching an eight-ship carrier battle group to the west coast of Central America as a "demonstration of U.S. interest in the region," and as officials confirmed plans to hold a large-scale U. S.-Honduran military exercise near the Honduras-Nicaragua border next month, also as a show of U.S. resolve.

As the House doors were about to be closed yesterday afternoon, Rep. Don Edwards (D-Calif.), an administration critic just returned from a four-day fact-finding mission to Central America, warned his colleagues on the floor that "there's a big invasion going on right now," and Rep. George Miller (D-Calif.), who accompanied him, said the "so-called secret war in Nicaragua is much more extensive than the American public has been led to believe."

But administration defenders said that whatever war exists was started by the other side.

"What's at stake here," Rep. Don Ritter (R-Pa.) said after the debate, "is a massive commitment" of Soviet and Cuban arms to leftist regimes and rebels in the region, "perhaps the most massive violation of the Monroe Doctrine that we have seen in the history of this hemisphere."

President Reagan used similar language yesterday in a ceremony marking Captive Nations Week, accusing the Soviets and Cuba of "building a war machine in Nicaragua."

The four hours of debate behind closed and guarded doors, only the fifth such session in House history, produced no clear result, leaving uncertain the prospect for Democratic-sponsored legislation which would ban further undercover U.S. aid to anti-government guerrillas in Nicaragua.

The bill, which has been approved by the Democratic majorities on both the House Foreign Affairs and Intelligence committees, instead would set up an overt $80 million program of aid to friendly governments in the region; it would be up to them to fight the leftists.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) said yesterday's session, called to discuss "confidential communications," was the best attended of the year and said he doubted that more than 20 members were absent.

Other members, however, said there were few surprises. Ritter said the audience fell off sharply after the first hour.

"We felt we were going to learn top secret information," he said. "Everybody waited with bated breath . . . . After an hour it was clear we weren't going to find out anything we didn't know already."

House Intelligence Committee Chairman Edward P. Boland (D-Mass.), one of the key sponsors of the bill to shift from covert to overt aid, led off with a history of the controversy.

Several members said Rep. Lee H. Hamilton (D-Ind.), who is a member of both committees with jurisdiction, made a spirited presentation of the case for terminating the undercover program.

Rep. C.W. (Bill) Young (R-Fla.), a member of the Intelligence Committee, presented the case against the cutoff, citing secret committee transcripts in an effort to prove that the administration is complying with restrictions approved by Congress last year.

The House is supposed now to move to a further debate in open session next week, then vote. First, however, the Democratic leadership, which is backing the cutoff, plans to take a head count. Several sources said that if they lack the votes the leaders may delay the showdown.

O'Neill said before yesterday's session began, "I think it's very close right now."

The Reagan administration, however, has been trying to work out a compromise ending CIA-backed covert operations only if the Sandinista regime in Nicaragua will certify that its own aid to leftist rebels in El Salvador and elsewhere in the region has ended, and Young told reporters the felt the debate had enhanced this plan's prospects.

"Many people in that chamber were looking for an honest compromise," Young said.

To pass the flat ban on undercover operations in Nicaragua, Young charged, would only provide "another forum for Sandinista propaganda" without having any practical effect. He said the Senate would not pass such a flat ban and that, even if it did, Reagan would veto it.

Current law requires that covert aid be used only to interdict arms shipments to leftists in Central America. Critics claim the administration is ignoring this and that its aid program is aimed at overturning the Nicaraguan government, the main established leftist regime in the region. They also say no arms shipments have been stopped.

"There's been absolutely no evidence, public or private," Rep. Thomas Harkin (D-Iowa.), told reporters after the debate, "of any interdiction of arms flow from Nicaragua to El Salvador" as a result of current U. S.-backed activities.

Edwards, reporting on his trip, said "there are at least three anti-Sandinista guerrilla operations on the north of Nicaragua , two on the south and one on the Atlantic coast. They've got a James Bond type of war going on--with frogmen, flashlights with TNT, and Claymore mines that say, 'This Side--Toward the Enemy' . . . . It has nothing to do with interdicting arms."

Administration officials said next month's planned joint military exercise will be an expanded version of war games the United States and Honduras conducted near the Honduran-Nicaraguan border in February. It will give the United States a chance to continue building up the military infrastructure in friendly Honduras, officials said.

February's military exercise, dubbed Ahuas Tara (Big Pine in Miskito Indian language), involved about 2,000 U.S. personnel. Although the Joint Chiefs of Staff are still putting the final touches on the second Ahuas Tara plan, officials expect closer to 5,000 U.S. military people to be involved this time.

In addition, the aircraft carrier Ranger is steaming south in the Pacific with its battle group to participate in maneuvers off the coast of Central America. Those operations could be part of the war games or might precede them, Pentagon officials said.

The war games will allow the U.S. Air Force, Army and Navy to practice operations with the major radar facility the U.S. built in Honduras during the February exercise.

The United States also will upgrade three small air fields in Honduras so they can handle any size Air Force jet in an emergency.

Still, Pentagon officials called the exercise "modest" compared with some the United States will conduct in other parts of the world.

They said that it represents an intensification, but not a new dimension, of pressure on Nicaragua, and that there are no plans to establish a blockade of Cuba or Nicaragua and no interest in doing so now on the part of military officials.

"That's a lot of unnecessary rhetoric," one military official said. "Obviously, if we wanted to we could, but it would take an awful lot of assets and tie down a lot of the Navy in one part of the world, which we really shouldn't do."

Staff writer Don Oberdorfer contributed to this report.