When senior financial advisers had an emergency meeting with President Reagan in the White House living quarters Tuesday after the 508-point plunge Monday in the Dow Jones Industrial Average, there was a surprising omission.

Office of Management and Budget Director James C. Miller III wasn't invited.

The following day another White House official twice forgot to name Miller when he was announcing four members of the administration's budget negotiating team.

More importantly, when the president and his aides were trying to trumpet a single conciliatory message in the wake of the stock market "meltdown," Miller was sending what House Budget Committee Chairman William H. Gray III (D-Pa.) called a "double signal" that the president may not be willing to compromise.

Sources said yesterday that the OMB director is in serious trouble at the White House and might be eased out.

Miller said yesterday that such rumors regularly circulate about OMB directors, and "nobody has talked to me about leaving, least of all the president of the United States."

Increasingly, sources said, Miller has become a high-profile speechmaker for Reagan's conservative fiscal policies while the real decisions are made at the White House and the actual negotiations are conducted by Treasury Secretary James A. Baker III, OMB deputy director Joseph R. Wright Jr., OMB executive associate director Timothy J. Muris and others.

Top administration officials were willing to deal with Miller as OMB caretaker when Reagan was focusing in the last few months of his administration on arms control and foreign affairs, sources said. But now that the stock market has made economic stability the No. 1 issue, White House officials may seek someone regarded as a heavyweight, according to sources, although "there is a reluctance to make changes" in the midst of a crisis.

"Director Miller has not had the kind of rapport with the president to allow him to be a significant player," said Rep. Vic Fazio (D-Calif.), a member of the House Budget Committee. "When problems develop, Jim Baker steps back in. He has not had the role that {then-OMB Director David A.} Stockman did. He doesn't have as much credibility here {on the Hill}."

Under Stockman, Fazio said, Congress came to expect the budget director to be close at hand at this time of year, when appropriations and reconciliation bills are moving. "Stockman was always here as an administration sounding board that was always accurate. We don't have that now," Fazio said.

Miller "is not in the policy loop in the West Wing" of the White House, said Lawrence Kudlow, chief economist for Bear Stearns & Co., and a former OMB official.

"He's ideologically correct, but politically inept," said one conservative appointee in the executive branch. "He comes in late, doesn't seem up to speed, and beyond his talking points doesn't seem to know how to maneuver."

Another Reagan appointee just below Cabinet level said, "He calls me and asks for approval, rather than the other way around."

Miller, however, has loyal fans among conservatives for his consistent opposition to tax increases or higher spending. "The rumor around town that Jim {Miller} is not plugged into the White House is a lot more wishful thinking than reality," said Richard Fink, president of Citizens for a Sound Economy, a Washington-based interest group.

"You can't beat the guy down," Fink said. "He's like one of those balloons, you punch him down and he comes right back up."

Indeed, the president Thursday night named Miller to his three-member team to represent the administration in negotiations with Congress over deficit reduction.

Rudolph Penner, an old friend of Miller's who describes him as a "smart fellow," said that the president almost has to name the budget director to any negotiating team.

T. Kenneth Cribb, assistant to the president for domestic affairs, said yesterday that he had been authorized by White House chief of staff Howard H. Baker Jr. to say that "both he and Jim Baker are going to be depending on Jim Miller not only for the technical details of the negotiations but for policy advice."

The public exclusion of Miller from Tuesday's emergency meeting to try to reassure Wall Street could have been humiliating for an official with thinner skin.

But after the meeting, Miller said in an interview, "I mentioned it to Howard Baker . . . . 'Yes, you should have been there,' " he quoted Baker as saying.