House Republicans, in the climax to 10 days of increasingly hostile relations with the Democratic majority, walked out of the House chamber en masse yesterday to protest the seating of Democrat Frank McCloskey in the disputed election in Indiana's 8th Congressional District.

The walkout was the first since 1890, when Republicans controlled the House and Democrats boycotted the swearing-in of a Republican in a similarly disputed South Carolina election, according to the House parliamentarian's office.

It was not clear how long the walkout would last. Republican leaders warned the Democrats that the disputed election -- now a symbol to the GOP of broader frustrations over Democratic domination of the House -- would not soon be forgotten. They threatened to continue indefinitely the disruptive tactics they began using last week to tie up the House.

But Republicans also said they would do nothing to jeopardize President Reagan's legislative agenda for the rest of the 99th Congress.

The walkout came moments after the House voted 236-to-190, along party lines, to seat McCloskey, as his subdued Republican rival, Richard D. McIntyre, watched from the back of the chamber. Ten Democrats joined the Republicans in opposing his seating.

House Minority Leader Robert H. Michel (R-Ill.) then tried to adjourn the House before McCloskey was sworn in. With the motion clearly headed toward defeat, the GOP lawmakers began filing out the center aisle of the chamber. As they did, Democrats applauded, mockingly waved goodbye, shook their hands and in other ways celebrated victory.

The Republicans filed out of the Capitol and down the steps outside the House chamber for a well-choreographed photo opportunity and mass news conference featuring McIntyre.

"This has united the Republican Party as nothing else," McIntyre told a bank of television cameras lined up on a grassy triangle outside the Capitol. "This will live on after Rick McIntyre goes back home to his family this evening. The American people are not going to forget."

As the Republicans were holding their news conference, McCloskey was escorted into the half-empty chamber to a standing ovation from the Democrats. The oath of office was administered by House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) -- to another round of applause.

Later, McCloskey told the House, "I'd just like to thank everyone in the House for their patience and tolerance. I'm really sorry for anyone on either side who has been saddened or inconvenienced . . . . I know we can settle down and work together over the next 15 months."

A moderate who was mayor of Bloomington for 11 years, McCloskey was elected to the House in 1982 by a slim margin. His southwest Indiana district is considered one of the most volatile in the nation.

He was returned to his seat yesterday after the closest House election in this century. McCloskey was declared the winner over McIntyre by four votes after a recount supervised by a House task force composed of two Democrats and a Republican. The task force split along party lines in its final recommendation.

On election night, McCloskey appeared to have won the seat, but errors in counting shifted the victory to McIntyre, who was then certified the winner by Indiana's secretary of state, a Republican.

On Jan. 3, the House split along party lines and voted not to seat McIntyre after questions were raised about the vote in Indiana. A state-supervised recount then gave McIntyre the victory by more than 400 votes, but Democrats charged that more than 4,800 ballots, many from predominantly black precincts, had been unfairly disallowed.

That led to creation of the House task force, which completed its work on April 18 in a fit of partisan anger over whether to count 32 unnotarized or unwitnessed absentee ballots.

When the Democrats on the task force decided not to count those votes, the result was 10 days of Republican rage described by House Majority Leader James C. Wright Jr. (D-Tex.) as "synthetic fury."

In two days of debate, Republicans charged that the Democrats had rigged the task force and its rules to ensure a Democratic victory and that the two Democrats on the task force had stopped counting votes once McCloskey was ahead. Republicans said the only solution was a special election.

"The task force simply found enough votes to elect its man McCloskey and then stopped counting," said Rep. Bill Frenzel (R-Minn.). "I believe McIntyre was beaten . . . by subjective judgments of McCloskey's Democratic cronies."

Democrats argued that the recount, conducted by the nonpartisan General Accounting Office, was fair. If McIntyre had emerged with a four-vote victory the Republicans would have demanded his seating, they said. In addition they said it would be wrong to deny McCloskey his victory in order to cool Republican tempers.

"The decisions were justified, they were supported and Mr. McCloskey ought to be seated," said the task force's chairman, Rep. Leon E. Panetta (D-Calif.), as the sole Republican member of the task force, Rep. William M. Thomas (R-Calif.), began shouting angrily at him across the chamber.

McCloskey said, "I won. I can look anybody in the eye and show I won."

Tempers flared several times during the debate, and the language used all week was usually harsh and personal. Democrats were called "slime," "thugs," and "corrupt." Republicans warned of "anarchy" and "poisoned" relations and called the House "a rotting carcass" of democracy.

Democrats were particularly offended at charges made on the floor that O'Neill "stole" the election and that Panetta, who has a reputation for fairness and independence, had rigged the outcome.

The aftermath of yesterday's vote is expected to be felt both on the House floor and in the 1986 elections. Republicans hope to come up with a strategy for continued disruption of the House, but no one in the leadership yesterday would predict what form it would take.

Michel said Reagan's legislative agenda will be "uppermost" in the minds of the leadership, but some younger, more aggressive members have been arguing for more confrontation.

"The nature of guerrilla warfare . . . is you do it when it's to your advantage and don't do it when it's not to your advantage." said Rep. Newt Gingrich (R-Ga.).

Democrats have predicted that most House Republicans will forget the episode quickly and move on to such pressing issues as the budget.

But Republicans said today Democrats were foolish to assume that. "I don't think the Democrats should rest on the idea that it ends today," said Rep. Olympia Snowe (R-Maine). "If anything, it's just beginning."

Republicans said yesterday they hope to make the Indiana contest a major issue in selected Democratic districts in the 1986 elections. The Republicans have prepared ads to run against Democrats they consider vulnerable.

But many Republicans, along with most Democrats, acknowledge that the issue has little appeal outside the House chamber and southwest Indiana