A short time before the House of Representatives uncharacteristically mustered enough conviction to say no to President Reagan yesterday, Rep. Jamie Whitten rose to vent his outrage.

The Mississippi Democrat had been up until 3:30 a.m. while the House-Senate conference committee he headed worked out a resolution to keep the government from shutting down for lack of its normal approriations bills, and now, he said, he was being told that Reagan would veto the resolution because it would cost too much.

Let him go ahead and veto it, said Whitten, a legislative pragmatist not known for avoiding compromise. The country is in trouble, he went on, "when the legislature gets to the point that it has to call the president to see what the legislature has to do."

It was the first sign yesterday that Reagan, the master of Congress much of this year, may have finally gone a bit too far in threatening to veto the continuing resolution which conferees--including those from the Republican-controlled Senate--had passed after two tiring days and nights of arguments.

The scene in the House reminded an old hand in the press gallery of the memorable remark of a Maryland politician who, in an exasperating moment, complained to the General Assembly: "This matter has deteriorated to the level of principle."

The "principle" that House Democrats finally were able to invoke is that, even in this year of dreadful Democratic defeats, the Congress has some role to play in writing budgets. In the end, there was enough in that appeal to institutional and tribal loyalties for the House, on a vote of 205 to 194, to approve the conference report and wait for the worst to happen.

A Democratic aide to the House leadership explained the turnaround this way: "It was institutional pride as much as anything else. There was a feeling of betrayal in the conference. They'd been working all week with people who seemed to be speaking for the president and then comes this threat to veto." Members felt the administration was almost dictating to Congress, he said.

Exasperation had been welling up all weekend as the House-Senate conferees bickered over what Whitten kept calling "this third bite out of the apple"--the third time this year Reagan demanded budget cuts.

The irritation may have been all the greater because the remaining differences seemed relatively small, only about $2 billion out of more than $400 billion to be spent. It was not the country's economic future that seemed at issue. Political face was also at stake, on both sides.

By the numbers prepared in the House, the conferees already had cut enough to meet the president's latest budget-cutting test. The White House didn't agree, but the Republican-controlled Senate's conferees had signed the same conference report the House members had. "I got home at 3:30 this morning without any feeling that the president would veto this bill," Whitten said on the House floor.

A mystery that puzzled many was why Senate Republicans, including Majority Leader Howard H. Baker Jr. (R-Tenn.), had signed off on the conference report early yesterday morning, signaling that the White House would accept it. Baker was placed in the embarrassing position of urging members to buy something it turned out the president would not buy. Baker explained last night that the president had reached his contrary decision on the basis of morning-after Office of Management and Budget cost estimates that differed sharply from those that had been used in the conference.

The House Democrats have been bending and buckling all year under Reagan's pressure, with enough of their own people crossing over to vote with Republicans to assure a series of glittering White House victories. But yesterday something special happened, and it brought enough conservative Democratic votes back home to make the difference.

Rep. James C. Wright Jr. (D-Tex.), the House majority leader, provided a measure of the change in mood. In an interview last week, Wright surveyed his party's legislative disasters this year and acknowledged that his colleagues had little appetite for a government-stopping confrontation on the continuing resolution.

Reagan, he said, had all the cards, because his veto, if it came, could be sustained. "He can get just about anything his whims call for," Wright said.

But yesterday, Wright was part of the turning tide and angry enough to let out something of a secret. Reagan had called him the night before, asking for help in adding more foreign aid funds to the resolution, Wright said. With great relish, he described a president urging more money for foreign aid on one hand and threatening a veto on the other if domestic programs were not cut further.

"We can't just surrender to the White House," he said, appealing to those institutional loyalties. "We can't just let them tell us what we may do and what we may not do.

"I don't understand, unless the president wants to bring the government to a halt."