The aborted hostage rescue mission in Iran has strengthened the hand of military spending advocates in their upcoming guns-versus-butter battle over budget priorities for 1981, congressional sources indicated yesterday.

In what House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) described as a "wave of patriotism sweeping the country," support was reportedly mounting for a proposal to increase defense spending by $5.1 billion over what has been recommended by the House Budget Committee.

The proposed amendment to the 1981 budget resolution, sponsored by Reps. Majorie S. Holt (R-Md.) and Phil Gramm (D-Tex.), calls for a corresponding $5.1 billion cut in nondefense spending, including food stamps, job training, nonmilitary foreign aid and other programs.

Sources who said last week they thought the Holt-Gramm effort would be defeated were conceding yesterday that the outcome was no longer clear. Asked by reporters if he thought the failed rescue mission would influence the outlook for the Holt amendment, which the House is scheduled to act on later this week, O'Neill said, "It could very well" shape the outcome.

At one point, House Budget Committee Chairman Robert N. Giaimo (D-Conn.) considered trying to head off the Holt-Gramm amendment by proposing a less drastic defense spending increase, aides said, but abandoned the idea because of timing problems and the difficulty of finding acceptable offsetting cuts in domestic programs.

With some of his domestic spending proposals jeopardized by the defense push, President Carter invited about 100 House Democrats last night to discuss the budget and related issues, including the presidentially imposed oil import fee that some lawmakers are trying to block.

House Majority Leader Jim Wright (D-Tex.) said Carter reaffirmed his support for a proposal to increase, rather than decrease, domestic spending by more than $1 billion over what the Budget Committee recommended for next year. Wright said he also urged the lawmakers not to try to block the import fee.

In the Senate, where an already defense-heavy budget resolution may come up for debate later this week, the hostage rescue effort may have doomed an already feeble attempt by liberals to add to domestic spending. "The gas has gone out to their balloon," said one well-placed source.