House Democratic leaders reacted angrily yesterday to what they said was heavy, behind-the-scenes administration lobbying that thwarted the House's attempt to adopt a preliminary budget target for next year.

Early yesterday morning the House overwhelmingly rejected a fiscal 1978 budget that contained too much defense spending and too little social spending for liberals, and a federal deficit too big to satisfy conservatives.

Later in the day House Speaker Thomas P. (Tip) O'Neill accused Secretary of Defense Harold Brown of making an "end-around" play. Brown, said O'Neill, bypassed the leadership and lobbied directly with Armed Services Committee members and others to gain support for an amendment by Rep. Omar Burleson (D-Tex.) to add $2.3 billion to defense spending for the federal spending year that begins Oct. 1.

That amendment restored defense spending to the level proposed by President Carter last February and angered House liberals such as Rep. Parren Mitchell (D-Md.) who said such an "enormous increase . . . was just not acceptable" when domestic programs are underfunded.

O'Neill - and Majority Leader Jim Wright (D-Tex.) told reporters yesterday that President Carter told them in separate phone conversations that he would defer to their judgment that such a big increase in defense spending would scuttle the budget.

Wright had proposed a compromise amendment which boosted defense outlays by $200 million and the Pentagon's authority to commit funds in the future by $1 billion. When that was defeated, however, he voted for the full restoration of the President's request.

O'Neill agreed with a report's observation that Brown, by continuing to lobby for a full restoration of defense funds, "made an end-around play and kind of fouled things up."

House Budget Committee Chairman Robert N. Giaimo (D-Conn.) was more critical of the President. By throwing his weight behind the Burleson amendment and by abandoning the $50-a-person tax rebate, "many members felt that the President's policy was veering much too sharply toward defense and the business establishment than could be justified."

"This is the United States Congress. It is not the Georgia legislature." an angry Giaimo told reporters yesterday. Carter must learn that he cannot dictate to Congress "what it must do," Giaimo said.

Under the law, the House and Senate must agree on a preliminary budget target for the next fiscal year by May 15. No spending bills can be considered for next year until a congressional budget resolution is passed by the House and Senate.

Giaimo said the Budget Committee will meet this morning to put the pieces back together and hopes to be on the House floor by next Friday with a new proposal.

The Budget Committee had recommended that the Government spend $462.4 billion, on revenues of $398.1 billion and run a deficit of $64.3 billion in fiscal 1978, which starts next October.

A series of amendments over two days of floor debate increased the deficit to $68.6 billion. When the House finally voted on the budget resolution, it was rejected 320 to 84 and sent back at committee.

The Senate will take up its budget resolution next week. The Senate Budget Committee has recommended a deficit of $63.2 billion, on spending of $453.8 billion and revenues of $395.6 billion. President Carter, in his proposals, called for a deficit of $57.9 billion next year.

Senate officials expect little trouble for the committee's resolution on the floor next week. The fledgling congressional budget process -- only in its second year -- has received strong bipartisan support in the Senate.

The process was set up to give Congress the same ability to deal with overall federal spending and compare one program with another that the President has had for years.

The House leadership is convinced that it will be able to reach a compromise budget in time to meet the May 15 deadline.

Giaimo, in his press conference yesterday, chided not only the administration for its back-door lobbying, but Republicans who voted to increase defense spending, but then voted against the overall budget anyway because of the big deficit.

The House committee had recommended spending $109.5 billion on defense, about $2.3 billion less than President Carter wants. Giaimo said that a compromise like the unsuccessful one proposed by Wright to boast spending $200 million for weapons programs is probably the right approach to producing a successful resolution.