House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.), mocking President Reagan's pledge to "go the extra mile" to reach a budget compromise, said yesterday that Reagan will have to "go first" if he wants Democratic support for an accord.

In a day of escalating partisan recriminations and diminishing hopes for compromise, House Republicans accused O'Neill of thwarting a compromise for political advantage, and O'Neill accused Reagan of posturing for the television cameras.

"President Reagan proved yesterday that he was willing to walk a mile--for a camera," O'Neill said. "He has yet to prove he is willing to walk a mile for a compromise. If he is sincere in his desire to get the nation out of the terrible economic mess it is in, then he must come forward with some specifics."

House Republicans saw it differently. At a closed-door caucus, the prevailing view of a dozen or more speakers was that a compromise with O'Neill and the Democrats "may be impossible," reported Michael Johnson, spokesman for Minority Leader Robert H. Michel (R-Ill.).

"Many said they haven't seen an example of good faith efforts by the speaker, and they ask, 'Are we being taken for a ride here?' " Johnson said.

It was in this kind of atmosphere of mounting suspicions on both sides that congressional leaders from both parties met again last night with White House aides to try to hammer out deficit-reducing budget revisions that could satisfy both Reagan and O'Neill. They failed to reach agreement, but plan to meet again, perhaps today.

In one of the few positive signs of a compromise, Sen. Ernest F. Hollings (D-S.C.) said he was persuaded that Reagan was serious about reaching an agreement after meeting with the president during Tuesday's negotiating session.

Reagan reportedly had indicated willingness to meet with the bargainers, and Hollings was the only one to take him up on the offer. Reagan reportedly joined Hollings in a private talk outside the room.

Another sign was recurrent talk of a Democratic counterproposal at some stage in the talks, perhaps to avert a total breakdown. Hollings said he understood there would be one, and House Majority Leader James C. Wright Jr. (D-Tex.) told reporters, "We're not going to leave that table without making constructive and concrete suggestions to reduce the budget deficits."

O'Neill also said the Democrats have proposals in mind for Social Security financing that do not involve cutting cost-of-living increases.

The proposals, which reportedly include alternative sources of financing such as general revenues and money from off-shore oil leases, do not anticipate benefit cuts or limitations and would not lead to deficit reductions, according to other sources. But the general mood was one of pessimism. Although several participants insisted that there was still a chance of success, both parties in the House appeared to be digging in their heels, making compromise more difficult.

After a meeting of the 29-member House Democratic Policy and Steering Committee, Rep. Leon E. Panetta (D-Calif.) said the consensus seemed to be that there could be no further cuts in programs for the poor, that Reagan's tax cut program had to be modified and that the Federal Reserve would have to give some assurance of lower interest rates if the deficit were reduced.

But many Republicans at the House GOP caucus reportedly indicated strong opposition to tax increases, and none spoke in favor of scaling back the 10 percent tax cut scheduled for next year. Most Democratic leaders are advocating scaling back the cut in the face of adamant opposition from Reagan.

"You've got positions hardening on all sides," said one of the negotiators, stressing the need for quick action to avoid a political polarization that could kill chances for a bipartisan compromise.

The negotiators said the talks probably will continue today if last night's session warrants a continuation. Without an agreement by week's end, Majority Leader Howard H. Baker Jr. (R-Tenn.) has said he would crank up the Senate to begin writing its own budget.

After the Democratic policy group's meeting, O'Neill said the committee had urged Democratic negotiators to continue, at least through today, but with a mandate to try to find out "what he wants, what he is willing to accept."

"That won't do. He won't do that," said Senate Finance Committee Chairman Robert J. Dole (R-Kan.), one of the congressional negotiators.

In addition to pressing the president to "go first," O'Neill said the House Democrats want the Republican-controlled Senate to act first on the budget. "We intend to have them act first," O'Neill said. The White House reportedly has been pushing for action first in the Democratic-run House.

At the Republican caucus, some members talked in frustration of trying to put together a budget alternative of their own or supporting a Senate move to draft a budget, but there was said to be no consensus on possible future strategy.

"The consensus," said Michel aide Johnson, "was that it may be impossible to get a compromise" out of the talks with Democratic leaders.

Rep. Jack Kemp (R-N.Y.), chairman of the House Republican Conference, said he thought it may be time for Republicans to start talking with Democrats "who share our views," rather than with the Democratic leadership.

Such a coalition between Republicans and conservative Democratic "Boll Weevils" was responsible for passage of Reagan's budget in the House last year.