A House-Senate conference committee broke up in disarray yesterday after failing to break an impasse over defense and domestic affairs issues in the fiscal 1980 congressional budget resolution.

After a token meeting in early afternoon, the two sides went back to their respective houses to ask that a new conference committee be named with specific instruction on how far to go toward a compromise.

The unprecedented action, which capped 2 1/2 weeks of stalemate over key issues, was a dramatic indication of how strained the budget fight is this year between the two houses -- particularly involving domestic program vs. defense.

Although both houses want to hold down the deficit, the Senate is insisting on higher defense spending, in part to placate critics of the stratefic arms limitation treaty, while the House wants higher domestic outlays.

The two sides also disagree sharply over whether to trim already enacted appropriations bills to stay within the tentative budget targets Congress voted last spring. The Senate wants to make these cuts, but the House does not.

Under the new congressional budget process, the two houses were supposed to have finished work on the final fiscal 1980 budget resolution by Sept. 15. The measure is intended to set binding spending ceilings that Congress must obey.