House and Senate Democratic leaders expressed concern yesterday about the progress of their party's presidential campaign as they mounted a concerted, acerbic attack on President Reagan to open the final pre-election session of the 98th Congress.

In an unusual move aimed at lighting a fire under Democratic presidential nominee Walter F. Mondale, House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) called on Mondale to stop "allowing himself to be punched around" by Republicans and urged him "come out slugging" against Reagan.

In the same vein, Senate Minority Leader Robert C. Byrd (D-W.Va.) said the Mondale-Ferraro ticket "got off to a rocky start . . . a hard start, a wobbly start" although he added he was confident that the Democratic campaign would improve over the next two months before the elections.

Despite the partisan overtones, O'Neill and Senate Majority Leader Howard H. Baker Jr. (R-Tenn.) opened exploratory talks that could lead to a compromise on defense spending, the major obstacle standing in the way of adjournment by early next month.

"I'm pretty optimistic we can work something out," said Baker after a private meeting he held with O'Neill in a low-key, conciliatory atmosphere contrasting sharply with the other rhetoric of the day.

Another Baker-O'Neill session is planned for early next week, when aides indicated that a compromise may be discussed.

Asked if the meetings were a prelude to a legislative "summit conference" on defense, which Senate Democrats have demanded and Baker indicated last month he would help set up, Baker said of the meeting with O'Neill, "That may have been a summit."

The Republican Senate and Democratic House have been deadlocked over defense spending largely because the White House has held firm in its demand for $299 billion in defense spending next year, an after-inflation increase of nearly 8 percent. By contrast, the House has approved an increase of 3.5 percent and indicated it would accept 5 percent.

Baker is said to have discussed his plans with the White House, although he would not say whether it is now willing to compromise.

Most of yesterday's partisan pyrotechnics came from the Democrats, although Rep. Barber B. Conable Jr. (R-N.Y.), ranking Republican on the Ways and Means Committee, held a news conference to challenge Mondale to "come clean" on how he would reduce deficits without raising taxes for low-to-moderate income Americans.

Otherwise, most Republicans appeared content to keep their heads low and let the Democrats flail away, at least for the time being.

In announcing the Senate Republicans' agenda for the next month, Baker led off with appropriations bills and other items, omitting virtually all of Reagan's wish-list for what is left of the 98th Congress.

Along with legislation to assure student religious groups access to public high schools, which has been enacted, Reagan called earlier this summer for passage of a balanced-budget constitutional amendment, tuition tax credits, and other incentives for neighborhood "enterprise zones" and tax-free retirement accounts for non-working spouses.

Baker mentioned none of these, although a Baker aide said Senate Finance Committee Chairman Robert J. Dole (R-Kan.) plans to push for enactment of the enterprise zone legislation.

O'Neill's advice to Mondale -- offered to reporters in his office and again before television cameras -- came on the eve of a meeting today between Mondale and Democrats in both houses, some of whom are nervous that a poor showing by their presidential ticket could affect their own reelection chances.

In tough language, O'Neill said Mondale was "being too much of a gentleman . . . trying to talk issues" while the Republicans have been "beating the hell" out of him.

Asked if he believes that nice guys finish last, O'Neill said, "Yeah, that's an old axiom, isn't it? . . . . Yeah, I'd have to agree with you." Mondale, said O'Neill, "came out fighting against Sen. Gary Hart when things were going bad and it's time he came out fighting against this fellow Reagan instead of letting him push him around."

O'Neill said Mondale should hit back at Reagan on the "war and peace" issue, budget deficits, the "fairness issue," Social Security cuts that O'Neill believes Reagan would advocate in a second term, and what he called "the incompetence of this man, unless he has his 3-by-5 cards."

In an attempt to raise the specter of Social Security cuts, O'Neill said: "As of today, every plan this administration has put forward for long-term deficit reduction has relied on massive cuts in Social Security."