President Reagan was said yesterday to have enough Democratic votes to offset expected Republican defections and pass his $98.3 billion tax increase in the Senate, but he still faced serious trouble in the House, where his Monday night speech left a mixed reaction within his own divided party.

Senate Finance Committee Chairman Robert J. Dole (R-Kan.) said at a news conference that three Democrats had assured him they would vote for the bill. Other sources said as many as 10 Democrats might do so, and Sen. Edward M. Kennedy (D-Mass.) went so far as to announce his support. All Democrats voted against the bill when it first passed the Senate by only three votes, 50 to 47.

In the House, however, both sides remained uncertain as to the likely outcome Thursday, when the big bill is scheduled to come up for a vote.

The White House was relying on the president's speech to help sway House members and chief of staff James A. Baker III said yesterday the nationally televised performance had "had a large impact . . . . It was well received by many members of Congress."

The White House also said the rise of the stock market yesterday was a sign that the president had been successful. "We can't help but interpret that the president's speech last night had some impact," said deputy press secretary Larry Speakes.

But some House members reported that they had received as many telephone calls against the bill as for the president, and some said they had received more against than for.

And the deep rift of the last 10 days remained within House Republican ranks between the senior, moderate leadership, and the younger, more intensely conservative members.

"This is the most traumatic fight we've had since Michel took over the leadership 18 months ago," an aide to Rep. Robert H. Michel (R-Ill), the House minority leader, said. Leadership sources claimed to be slightly optimistic over their chances to win approval, although they acknowledged they need strong Democratic support.

At a breakfast meeting with reporters, David A. Stockman, director of the Office of Management and Budget, claimed that Democrats have a higher responsiblity to vote for the tax increase than Republicans because, he said, Democrats are to blame for deficits.

"They have more responsiblity to vote for the revenue bill," he said.

Stockman only endorsed the bill in lukewarm terms. It is, he said, "not the greatest thing since sliced bread . . . . There are lots of things -- including withholding of taxes on dividends and interest, which the bill would require -- you wouldn't want in an ideal world . . . . A tax increase is nobody's idea of good policy."

Rep. Barber B. Conable Jr. (R-N.Y.), ranking Republican on the Ways and Means Committee, said the tax controversy has caused such problems within GOP ranks that the leadership, which backs the bill, has been unable to rely on the whip structure to determine vote counts.

Conable said that aides to some of the whips are undermining the process. The aides, who are against the bill, are giving the names of members indicating a willingness to vote for the measure to conservative organizations, which are then lobbying to turn the members around.

Conable was notably unenthusiastic, largely because he lost a series of attempts to amend the bill so as to lessen tax increases on business.

Conable said he told Rep. Dan Rostenkowski (D-Ill.), chairman of the Ways and Means Committee, "I needed some things to get Republicans to vote for the bill. He didn't give them to me, so I won't feel very sorry if it goes down the drain . . . . "

The president, continuing his efforts to pick up conservative support, released a letter to Rep. Trent Lott (R-Miss.), the House minority whip, promising, "The administration will not relax one bit in our continued pursuit of spending cuts."

"I will, of course, resist any budget-busting spending measures that come across my desk," he said. Conservatives want to reduce the deficit by cutting spending rather than raising taxes.

The administration has also offended conservatives this week by announcing it would limit further arms sales to Taiwan in deference to China, but administration aides said they doubted this would hurt them on the tax bill, in part, as one put it, because no member, no matter how often offended, can vote no more than once.

On the right, meanwhile, the rebellion against the legislation continued. Richard Viguerie, publisher of the Conservative Digest, announced formation of an organization called "Conservatives Opposed to the Tax Increase," the members of which include Howard E. Phillips, chairman of the Conservative Caucus, John T. (Terry) Dolan, chairman of the National Conservative Political Action Committee, and Peter Gemma, head of the National Pro-Life PAC.

Almost all these groups were strong Reagan supporters during the 1980 election and during his first year in office.

And the leading congressional opponent of the tax bill, Rep. Jack Kemp (R-N.Y.), disputed Reagan's claim that the tax bill will not hurt the lower and middle class:

"As long as you . . . don't use the telephone, don't have big medical expenses, don't pay medical insurance premiums, don't suffer losses due to theft or casualty, don't smoke, don't ride in airplanes, don't have a savings account . . . you won't be affected," he said.