Tip O'Neill may be getting his House in a little better order.

At least he has bounced back from that last week in September which has to rank as the low point of this year for the House leadership.

That was the week the leadership went to the floor with four major measures, including a bill to effect the Panama Canal treaties and the second and binding congressional budget resolution for this fiscal year, and lost all four.

About the same time, rumors began circulating here and in Boston that this might be Speaker Thomas P. O'Neill Jr.'s last term -- that the genial, portly Massachusetts Democrat might retire, discouraged by his inability to control the House and disgruntled with younger Democrats whose conservative bent and lack of party loyalty he doesn't understand.

O'Neill vigorously denies the rumors. "Ou can be assured I'll be a candidate for reelection," he said in an interview last week.

O'Neill noted he has been campaigning widely for House Democrats in such states as Missouri, California and New York, and intimated he would not be making such trips if he were stepping down as leader.

He also defends the leadership's record -- particulary the recent one.

"We haven't had a loss since that week" in September, O'Neill said.

Since then the House has passed a bill setting up a new department of education, two important pieces of Carter's energy package, the budget and a controversial welfare revision bill. (It did defeat President Carter's hospital cost-containment bill the other day, but that was brought up atCarter's insistence and over O'Neill's objection.The Speaker knew he did not have the votes.)

O'Neill says statistics show Carter is having more success with Congress "than any president in 20 years."

He also admittd he heeded the grumbling after those September defeats and took some steps to improve the situation.

The first step was jawboning with Democrats who had been compiling a record of voting against the party and the administration. "I started with personal conversations and meetings and showing them their voting records," O'Neill said. "Many of them couldn't believe they had moved from moderate to conservative as far as they had.

"I told them, 'You're not voting the philosophy of your area. I know your district and it hasn't gone as far as you have gone.'"

With the most conservative, he was tougher. "I don't know what you're doing in the Democratic Party," he told two member. "We'd be better off if you got out."

O'Neill was urged to follow up the tough talk with some disciplinary measures. Senior Democrats suggested using comittee assignents to reward and punish.

The next step involved changes to help the leadership keep a tighter rein. Younger Democrats wanted the Rules Committee to restrict the number of amendments that could be offered to a bill, thereby saving them from the temptation to vote for politically appealing amendments offered by Republicans.

They also wanted to find a way to prevent appropriations bills from being used as vehicles to force votes on controversial issues such as abortion and pay raises.

O'Neill had breakfast and dinner with the Rules Committee, during which it was decided that cutting off a member's right to amend a bill had to be used judiciously. But the Rules Committee has brought out more bills with restrictions on amendments, including the controversial welfare reform proposal and a proposal to limit the investigatory powers of the Federal Trade Commission.

O'Neill's troubles with the House spring from a variety of sources both inside and outside the institution. President Carter's weaknes on the Hill hasn't helped.

Generally, a new type of Democrat was elected after the Vietnam War and Watergate. He was anti-Big Government, less respectful of the presidency and determined to be independent. He learned to get elected by taking care of his constituents.

Inside the institution, committee chairmen were neutralized by reforms that curbed their power. Power was diffused to subcommittees, often headed by younger memers. Every man a king and no man a subject became the rule.

In the absence of other influences, special-interest influence has often been the main factor in defeat or passage of a bill.

At the same time, more aggressive Republican freshmen have forced leaders in that party to become more partisan, more conservative and less willing to accommodate the Democrats.

One of those freshmen, Rep. Newt Gingrich (R-Ga.) is trying to woo conservative Democrats into forming a new Republican-conservative Democrat coalition.

Gingrich said O'Neill is too liberal for the majority of the House. "The gatekeepers have the wrong viewpoint. They are trying to prevent the rule of the majority."

Gingrich believes that while a majority of the Democrats are moderates to liberals, conservatives are in the majority in the House as a whole. "As the country gets into economic trouble they're [the leadership] going to parapyze themselves until they force changes in leadership," he said.

Reportedly only a half-dozen or so conservative Democrats have taken Gingrich up on his invitation to join in a coalition. But Gingrich believes that if more Republicans are elected next year, such a coalition might take shape, if only informally.

One senior Democrat commented, "What you could be looking at is a real battle for the soul of this place after next year's election. I haven't heard any of the speculation about O'Neill quitting, but I know him well enough to know he doesn't walk away from a fight."