House Democrats and Republicans met separately behind closed doors yesterday and reelected their leaders to office as conservative Democrats abandoned their effort to unseat House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.).

O'Neill, 71, in an acceptance speech that several lawmakers described as a swan song before his announced retirement in two years, promised to listen more attentively to conservatives and restive younger Democrats.

He voiced disappointment at ending his political career fighting a Republican administration and said the presidential results clearly demonstrated that it is time for Democrats to find "some new themes and some new directions."

The Democratic National Committee has not performed well in the last few years and needs to be improved, he said.

Rep. Charles W. Stenholm (D-Tex.), a leader of the conservative "Boll Weevils" who had talked of challenging O'Neill, said that while he felt there was a need for "a new coach," he decided not to run because he didn't have the votes and because O'Neill had assured him that conservatives would have a "voice at the table."

O'Neill will be officially elected speaker in what is generally a party-line vote by the full House when the 99th Congress reconvenes Jan. 3. Democrats will control the new House 253 to 182.

On the Republican side, Minority Leader Robert H. Michel (R-Ill.), 61, never had a challenge, and his acceptance speech yesterday was a rallying cry for GOP unity. With President Reagan a lame duck, Republican pragmatists and the ideological right wing already have begun to bicker over the future of the party.

"Everybody has the right to have his say -- but no one can expect to always get his way," Michel said. "Better the discord come from the other side of the aisle and that we all work together harmoniously."

Michel also promised that House Republicans will not be a quiescent minority in the new Congress but will aggressively take on the Democrats, a pledge that will be quickly tested. The Democrats today are likely to approve a new rule for the House that would limit the after-session television time available for floor speeches.

A group of House Republicans used that time every night in the last Congress to attack Democrats for hours, and have charged that the proposed one-hour limit on each party each night amounts to a gag rule.

On a day that felt much like the return to school after summer vacation, with most lawmakers catching up on their colleagues' election totals, most controversial issues were put off. Lawmakers said the mood in both caucuses was optimistic, with the expectation that this year will be full of partisan warfare but not as bitter and hostile as it was last year in the heat of presidential and congressional elections.

Democrats, who lost fewer House seats than expected -- a net loss of 14 -- and gained two in the Senate, were especially upbeat, despite the presidential loss.

"We're feeling our Cheerios. . . . Every person in the House feels they have a real role to play in shaping the future of the party," said Rep. Mary Rose Oakar (D-Ohio), who was elected secretary of the caucus, replacing Rep. Geraldine A. Ferraro (D-N.Y.), who left the House to run for vice president.

In addition to electing O'Neill by acclamation, the Democratic Caucus reelected Rep. James C. Wright Jr. (D-Tex.) as majority leader. Wright, lawmakers said, got the biggest standing ovation of the day when he said, "We are not prepared to be the patsies for the artful pretenders at the other end of Pennsylvania Avenue," meaning in the White House. He added that if there is a tax bill in 1985, it will be "Reagan's tax bill."

Rep. Richard A. Gephardt (D-Mo.), 43, a leader of the growing group of younger House Democrats, replaced Rep. Gillis W. Long (D-La.), 61, as caucus chairman. The caucus is expected to play an increasingly active role this year in setting the House's agenda and policies. No other changes in the House Democratic leadership occurred.

In the House Republican lineup, minority whip Trent Lott (R-Miss.) was reelected. The only change for Republicans occurred with the election of Rep. Lynn Martin (R-Ill.) who succeeded Rep. Jack Edwards (R-Ala.) as vice chairman of the GOP conference, the Republican name for its caucus.

Edwards did not run for reelection to the House this year. Martin will be the highest ranking woman ever in the House Republican leadership.

In addition to the television time limit, Democrats are expected to consider a drastic revamping of the entire fiscal processes of the House today, as well as whether to extend the terms of budget committee members. The latter rules change would allow budget chairman James R. Jones (D-Okla.) and his chief rival for the chairmanship, Rep. Leon Panetta (D-Calif.) to remain on the committee. Both have exhausted the time limit set out in current House rules