A rebellious House last night rejected a compromise balanced-budget plan for fiscal 1981 and then, in a surprise move, voted overwhelmingly to stick by the plan's defense-heavy spending priorities.

"They're going in both directions at the same time . . . I got two mandates, and I got headaches," protested House Budget Committee Chairman Robert N. Giaimo (D-Conn.) as he prepared to go back to conference with the Senate next week in hopes of salvaging a balanced budget -- which would be the country's first in 12 years.

The budget action followed another surprise move by the House to demand, over strenuous objections from its leaders, a vote as soon as possible on killing President Carter's proposed 10-cent-a-gallon gasoline fee to curb oil imports.

In the process, the House held hostage a resolution to extend the federal debt ceiling by 30 days, forcing the House Rules Committee to meet today in special session to consider a way of accommodating the demand for a vote on the controversial fee.

The debt ceiling expires Sunday. Although Congress has gone through similar exercises before without the government grinding to a halt, House Speaker Thomas P. (Tip) O'Neill. Jr. (D-Mass.) warned that banks could stop cashing Social Security checks next week if Congress does not extend the ceiling to continue the government's borrowing authority in the meantime.

The 242-to-141 vote to kill the $613.3 billion budget plan was more lopsided than expected, with a majority of Democrats as well as Republicans joining in scuttling it.

The vote at first appeared to be a victory for President Carter, O'Neill and the moderate-to-liberal Democrats who complained that the compromise hammered out by a House-Senate conference last week provided too much money for defense at the expense of social programs.

But that apparently was not the message -- or at least not the whole message.

Within a couple of hours of defeating the budget resolution, the House by voice vote agreed to a proposal by Rep. Delbert Latta (R-Ohio), ranking minority member of the House Budget Committee, to instruct the conference to insist on their earlier defense figure of $153.7 billion.

This figure was much closer to the Senate's defense proposal than it was to the $147.9 billion in military outlays originally recommended by the House. It represented an increase of $18 billion, or about 13 percent, over anticipated defense spending for the current fiscal year, compared with an average increase of about 5 percent for nondefense programs. The increase in long-term budget authority for defense was even larger.

The vote on Latta's motion indicated that many who opposed the budget resolution, Democrats as well as Republicans, may not have done so because of problems with the big defense increase. Also, many of those who opposed the budget resolution had gone home before the vote on Latta's proposal, although Giaimo did not contend that attendance was the chief problem.

"The overwhelming support is for more defense . . . and for higher social programs, too," Giaimo told reporters after the vote, suggesting that "it may well be that congress doesn't want budget discipline and wants to give up the [budget control] process."

He said the House had voted "ridiculously" but thought there may be a mood for accommodation in the conference in hopes of salvaging the balanced budget, upon which both Carter and Congress pinned so many political and economic hopes just a few weeks ago.

As for Carter's last-minute intervention in the process by announcing his opposition to the budget on Tuesday, Giaimo said, "The president didn't help the situation." Actually, Carter had set somewhat of a precedent for the House by calling for an increase in military compensation one day and then opposing the congressional budget plan the next day -- for which he was called a hypocrite by Senate Budget Committee Chairman Ernest F. Hollings (D-S.C.).

Last night Hollings issued a statement saying that "the principal blame for the defeat of the [resolution] must be laid at the feet of the president."

Another possible factor in the big vote against the resolution was O'Neill's switch from benign neglect toward the compromise to outright opposition on Wednesday. "It's not in tune with the basic philosophy of the Democratic Party and the programs I worked for all these years," O'Neill told reporters shortly before the vote. But even O'Neill said he thought "it would probably have been best if [carter] had stayed out."

Republicans delivered 44 votes for the resolution, double their number when the House approved its budget resolution earlier this month. But this was far from enough to offset the 146 Democratic votes against the package. Only 97 Democrats supported it.

Among Washington area legislators, only Reps. Joseph Fisher (D-Va.) and Marjorie Holt (R-Md.) voted for the budget plan.

The House also rejected a Budget Committee proposal to let Congress move ahead immediately with approval of a new budget ceiling for 1980, which would have lifted the freeze on spending bills that has been in effect since the old budget ceiling was breached earlier this year.

Now the 1980 ceiling will probably not be changed until a final package is approved. Giaimo expressed concern last night that the budget process faces "another big threat" when Appropriations committees move to seek waivers from the budget rules in order to get votes on high-priority spending measures for the rest of this year.