Congressional Democrats gave the first firm signal yesterday that tax overhaul faces serious problems in the House, where President Reagan is counting on a victory this year that would put pressure on the more-skeptical Senate to approve the proposal.

House Democrats emerged from their caucus on tax revision deeply divided and skeptical about prospects for passage. Lawmakers said that while many members fear being blamed by the president for killing tax overhaul, there was sharp disagreement over how such a restructuring should proceed and whether to use a tax-revision measure to raise revenue and reduce the federal deficit.

At the same time, a small but growing number of lawmakers said President Reagan has been unable to generate public enthusiasm for tax reform and that the Democrats, instead, should push a minimum tax to force corporations and the wealthy to pay their fair share.

"I think the plan's in deep trouble, and I regret it," said Rep. William B. Richardson (D-N.M.), a sentiment echoed by many attending the two-hour closed caucus. "I think we Democrats are underestimating the popular strength of tax reform."

Rep. Richard A. Gephardt (D-Mo.), an author of one of the major tax-revision measures, acknowledged yesterday that passage of a tax plan faces major stumbling blocks and said it is largely up to Reagan to overcome them.

"Tax reform depends in large part on the president's ability to create a constituency for it," Gephardt said. "I don't think it exists now."

Gephardt said that if Reagan is able to generate public enthusiasm for tax reform, a measure will be passed by the House this year. But, he added, "if he doesn't, it won't."

House Ways and Mean Committee Chairman Dan Rostenkowski (D-Ill.), a leading Democratic spokesman for the tax plan, said he was not discouraged by the caucus. "The impression was there was movement to set tax reform aside . . . that wasn't the consensus of the caucus," he said.

The caucus met as congressional Republican leaders conferred with Reagan at the White House. Reagan made a strong appeal to the GOP leaders to get his tax plan passed this year.

After the meeting, Senate Majority Leader Robert J. Dole (R-Kan.), who in the past has said the Senate would not have time to take up a tax bill this year, indicated that, if a tax bill arrives by Nov. 1, he would be willing to keep the Republican-led Senate in session after Thanskgiving to work on it.

"We're prepared to come back after Thanksgiving to work on a tax bill if it's available," Dole said. "We plan on being here all year if there's anything to do. It's really up to the House."

Dole said the Senate needed time to digest, and presumably alter, any bill that emerges from the House and that he hoped the House will complete action on a bill by Nov. 1.

House Minority Leader Robert H. Michel (R-Ill.) said yesterday that it was possible that a bill approved by the House would not be acceptable to Reagan and the Senate would have to take its time "correcting" the bill.

Republicans in both chambers have shown little enthusiasm for the president's tax proposals. There have been suggestions that some Republicans would try to draft an alternative. "The silence on their side is deafening," said one Democratic congressional aide.

House Democrats, meanwhile, have grown increasingly skittish as the Ways and Means Committee has moved into high gear this week on tax revision.

Committee members yesterday overturned operating procedures they had set last week, agreeing to tax-writing rules that will make it more difficult to keep their legislation from increasing the federal deficit.

After a three-hour meeting, Ways and Means members agreed that each amendment to the legislation they draft will not have to raise taxes in one area if it cuts them in another. However, the package the committee votes out must still be "revenue neutral."

Members also accepted a motion by Rep. Sam M. Gibbons (D-Fla.) requiring that provisions of the tax code remain intact if a majority of the committee does not vote to change them. Thus, the alternative proposals developed by the committee staff and offered by Rostenkowski cannot automatically become part of a tax-overhaul bill. It also means that large portions of Reagan's tax-revision plan will be considered by the committee only if they are offered as amendments.

Yesterday's caucus, demanded by the liberal Democratic Study Group, was sparked largely by concerns among some Democrats that the tax measure was moving too fast and without adequate consultations.

Democrats said after the caucus that they appear to have split into three groups on the tax issue.

A large faction, led by Ways and Means' Rostenkowski, thinks that the Democrats cannot be seen as hindering tax reform and should take credit for it. This "revenue-neutral" group strongly opposes using tax reform to reduce the deficit because Reagan has said he opposes that approach. These Democrats fear that he would pummel them politically for raising taxes.

House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) came down strongly on the side of this group yesterday, saying, "Any tax while I am here will have to emanate from the president." O'Neill has made it clear that he is determined to get the tax bill out of the House so that if the measure does not pass this year the Republicans in the Senate will be blamed.

However, another large group of Democrats, including members of the Democratic Study Group, thinks that most Americans are less interested in tax reform than in reducing the $200 billion annual federal deficit. Members of this faction said that any tax-reform bill should use revenue generated from closing loopholes to reduce the deficit, not tax rates for individual taxpayers.

And a smaller group, but one that Democrats said appears to be gaining support, has begun to question the wisdom of passing a major overhaul measure.

"They want to pass a minimum tax that would force General Dynamics and other companies to pay their fair share, use some of the revenue for the deficit and call it tax reform," said one Democratic aide.

Members of this group said the tax issue was forcing Democrats to "bleed" back home, where much attention is focusing on deductions that might be lost.