President Carter yesterday accused Congress of finching in the face of political pressures and breaking commitments by its leaders to support his new oil-import fee and balance the budget.

Carter's remarks, his strongest criticism of the lawmakers in months, set the stage for a possibly serious confrontation between the White House and Congress over the controversial import levy.

Both houses have made it clear in recent days they have substantial marjorities ready to roll back the import fee. The Senate Finance Committee and a House Ways and Means subcommittee both voted overwhelmingly Wednesday to repeal it.

However, Carter reminded the lawmakers yesterday he imposed the levy "unilaterally" after congressional leaders "asked me to" during negotiations last March over the administration's new anti-inflation plan.

Now, the president protested in a speech before a White House small business conference, "the pressures of a political election year are . . . working counter to the realization of" the fee and the balanced budget goals.

He noted that "now there are pressures in the Congress to take legal action to stop the imposition of this . . . fee. In recent days, we have seen disturbing signs in the Congress that this commitment to a balanced budget might be in danger."

Congressional sources who were in on the negotiations this March confirmed there was "a clear consensus" among House and Senate leaders that Carter should impose the fee, which would raise gasoline prices 10 cents a gallon.

At one point during the closed-door talks, sources said, Senate Majority Leader Robert C. Byrd (D-W.Va.) even chided Treasury Secretary G. William Miller for trying to reopen the fee issue. Byrd said the issue had been closed.

Although Byrd and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) have steadfastly supported the import fee since then, O'Neill and other leaders have conceded there are enough votes both to repeal the levy overwhelmingly and override any presidential veto.

It the meantime, a U.S. District Court has ruled the import fee illegal on grounds that Carter chose to require refiners to pass the levy on to consumers only in the form of higher gasoline prices, leaving heating fuel prices untouched.

The U.S. Court of Appeals for the District of Columbia agreed yesterday to grant the administration a speedy hearing on appeal, scheduling arguments for early June. However, the fee has been shelved pending a decision.

The levy, which would have raised prices at the pump yesterday morning had it not been blocked by the courts, would bring in an estimated $10.3 billion in new revenues each year.

Although Carter did not propose using the money to balance the budget, both the House and the Senate counted the extra funds in their fiscal 1981 budget plans, and the House earmarked the surplus for a tax cut.

Yesterday, Rep. Al Ullman (D-Ore.), chairman of the Ways and Means Commitee, said repeal of the fee would further cloud prospects for a tax cut this year. Ulman opposes a cut now anyway as too inflationary.

In his speech yesterday, Carter also contended that repeal of the import fee would result in higher oil prices because consumers would not reduce gasoline use enough to discourage oil-exporting nations from raising prices sharply.

Meanwhile, the administration won a one-week respite before the Ways and Means Committee votes on whether to repeal the fee, but the delay involved only a parliamentary snag and was not expected to save the levy from defeat.

The action came after Rep. Sam Gibbons (D-Fla.), at the request of the administration, protested that the panel had not waited 48 hours after the measure was voted on by a subcommittee, as required by committee rules.

Carter has estimated that imposition of the 10-cents-a-gallon fee would reduce U.S. oil imports by about 100,000 barrels a day by the end of one year. The United States currently imports about 6.2 million barrels of oil a day.

Meanwhile, House and Senate budget conferees passed their nominal deadline for action on the 1981 budget plan as the two sides shadow-boxed over the relative priorities of defense and domestic spending.

No action is expected before Monday when House Budget Committee Chairman Robert N. Giaima (D-Conn.), who is recuperating from a stomach ailment, is expected to return and initiate real bargaining.

House conferees refused to take up defense spending before Giaimo returns; Senate conferees complained that they were dragging their heels in hopes of forestalling action that might lock them into heavy domestic spending cuts. The House conferees apparently hope to avoid this by presenting a package of cuts in both domestic and defense spending on Monday.

After nearly two days of talk about the 1981 spending plan, the conferees succeeded in making just enough cuts to rescue the Senate-approved budget plan from its precarious balance on revenues from Carter's oil-import fee.

Senate Budget Committee chairman Ernest F. Hollings (D-S.C.), who had been annoyed by newspaper stories suggesting that the Senate had to dip into the import fee revenues to balance its budget, proudly annouced the achievement and asked reporters to note that the Senate had "undipped."

The balance was restored largely by reducing some of the veterans' benefits money and postal subsidies that the Senate added to its budget earlier this week. The postal subsidy cut will make it harder -- but not impossible -- for the postal Service to continue Saturday mail deliveries, according to Budget Committee aides.