Like a warning shot across the bow, House rejection of production funds for the MX missile put President Reagan on notice that he can no longer count on Congress to give him the benefit of the doubt -- even on a sensitive question of national security. But neither Democrats nor Republicans interpreted the impressive 245-to-176 MX vote on Tuesday as a prelude to a full-fledged congressional rebellion against Reagan and his ongoing defense buildup nor as an indication of some impending reversal of the priorities he has set. Rather, they said, it was merely the latest and strongest of many indications that Congress is finally drawing a line as to how far it can be pushed to support administration ventures it regards as dubious at best. Or, as some put it, Reagan is no longer larger than life, thanks to deepening economic troubles, Republican losses in the recent elections and other factors. "The shine is off right now," said House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) yesterday in predicting that Reagan will have a tougher time with Congress next year than he did in his first two years, especially in light of the 26 new Democrats who will solidify Democratic control of the House for the first time since Reagan took office. "When the administration makes sensible requests, Congress is still going to listen," said Rep. Paul Simon (D-Ill.). But "rubber-stamp approval of any administration request, regardless of its merits, is a thing of the past," he added. "Everything is going to be scrutinized a lot more closely . . . including defense," said House Minority Whip Trent Lott (D-Miss.). What was significant about Tuesday's vote was that, while Congress has been increasingly willing to break ranks with Reagan on domestic issues, it has been reluctant to do so on defense, except to squeeze a little from the huge spending increases he has proposed for the Pentagon. Last year, Congress unceremoniously rejected his proposals for Social Security cutbacks, rejected his budget and wrote one of its own, then overrode his veto of an omnibus supplemental appropriations bill and refused to go along with his proposed constitutional amendment to require balanced budgets. But, while Tuesday's vote signaled that defense is no longer sacrosanct, there remains "a strong consensus to continue the military buildup," with differences only over the extent of the buildup, as House Democratic Whip Thomas S. Foley (D-Wash.) observed yesterday. Even with all the cuts that Congress may make before approving defense appropriations for this year, it will still be giving Reagan most of what he wants for the military. Even some liberals are talking about a 5 percent increase for defense after accounting for inflation, less than Reagan wants but large by pre-Reagan standards, at a time when most domestic programs are being heavily cut back. According to many, the MX vote was in product the result of quite specific skepticism about that missile, and especially the administration's "Dense Pack" plan for deploying it. This could mean trouble for other new weapons systems that run into similar doubts, but not necessarily trouble for the bulk of the defense buildup. "It was a bad product and it was badly sold," said O'Neill. Nevertheless, some defense advocates are worried that economic troubles, including huge budget deficits, are going to force more military cutbacks. "It would be a mistake for the administration to underestimate the significance of the [MX] vote," said Rep. Richard B. Cheney (R-Wyo.). "You can't take for granted the big base of support [for defense spending] that existed six or 12 months ago," he said, although he added that lame-duck sessions are "notoriously flaky" and poor guides to future action. Many House members agreed yesterday that the November elections had a major impact on the MX vote. A total of 50 Republicans broke ranks with Reagan on the MX missile, including most of the "Gypsy Moth" moderates who swallowed misgivings and voted with Reagan much of the time over the last two years, only to get roughed up by voters last fall because of it. Reagan also lost the support of many conservative Democratic "Boll Weevils," who had also gotten an election message from their generally hawkish constitutents that even defense spending can be carried too far. Members generally agreed that Reagan is likely to suffer more losses in the 98th Congress than he did in the 97th but probably only on selected issues and only if he doesn't respond to the signals that Congress was sending on the MX vote by trimming his ideological sails. "It's still going to be a closely balanced Congress," said Rep. David R. Obey (D-Wis.), "but the economy especially has given us a lot more running room than we had before." Said Rep. Richard A. Gephardt (D-Mo.): "There was a tendency before to give the president the benefit of the doubt. Now we've returned to a more normal situation and all the doubts won't be resolved in the president's favor."