Congressional conferees agreed to use an unprecedented means for enforcing spending self-discipline yesterday as they wrapped up a barely balanced $613.3 billion budget for next year and sent it to an uncertain fate in the House.

Even before the conferees finished their work, House Speaker Thomas P. O'Neil Jr. (D-Mass.) warned that the defense-heavy spending blueprint for fiscal 1981 faces "a tough fight, a hard fight" in the House next week because of moderate-to-liberal opposition to social program cuts.

What the conferees did was to cut a swath through domestic programs to give the military an $18 billion increase in spending next year. Bowing to guns-over-butter demands from the Senate, the House conferees agreed late Wednesday to $153.7 billion for defense, $2 billion less than the Senate wanted and $5.8 billion more than the House had proposed.

As the morning-after damage assessments began, some programs appeared to be in better shape than expected, while others suffered more than anticipated. For many, because cuts were spread over broad categories of services, the damage was difficult to assess.

Some examples:

Transportation was harder hit than most programs, with mass transit and highways expected to bear the major brunt of an $850 million cut from the House version. "We can't cut the operating programs, so we'll have to go after the big grant programs, and the biggest new grant program is transit," said a congressional appropriations staff worker.

President Carter's proposal for 300,000 new subsidized housing units would be reduced by at least 20,000, according to congressional budget sources.

Putlic service jobs, not counting training jobs aimed at the hardcore unemployed, would be reduced from about 200,000 positions to somewhere between 100,000 and 150,000. The Senate had sought even fewer, along with a phaseout of the program by the end of 1981.

Low-income fuel assistance would be continued at the current funding level of $1.6 billion, which is $800 million less than Carter proposed and $600 million less than the House wanted.

Funds to assure a continuation of Saturday mail deliveries were dropped, although the House had to abandon an effort to ban Saturday deliveries outright.

Under the spending discipline procedures approved the by conferees, congressional committees would be directed to achieve $6.4 billion in reductions from existing programs in 1981 and $1 billion for the more than half-completed 1980 fiscal year. The two revenue-raising committees also would be instructed to come up with $4.2 billion in new revenue for 1981.

This so-called "reconciliation" procedure has existed in law for six years but has never been employed so extensively or in an initial target-setting budget resolution.

The deadline for committee action is June 20 for 1981 savings. The reductions than would be packaged for a vote by both houses in what is generally viewed as another key test of congressional resolve to abide by its spending self-discipline.

An earlier test will come next Thursday when the House is scheduled to vote on the overall spending blueprint.

Speaker O'Neill said about 40 Democrats who voted for the less defense-weighted House version of the budget several weeks ago had said they doubted they can vote for the conferees' compromise.

But Rep. Delbert Latta (R-Ohio), rankig minority member of the House Budget Committee, said he hopes the compromise will get 40 to 50 Republican votes, compared with the 22 that the original proposal got. The earlier House vote was 225 to 193.