House and Senate negotiators reached tentative agreement yesterday on legislation to force a balanced budget within five years and predicted it will be passed by Congress and signed into law by President Reagan.

The landmark budget-control bill sets fixed ceilings to reduce deficits now hovering around $200 billion to zero by fiscal 1991 and requires automatic, across-the-board cuts in most programs to achieve the targets if Congress falls short of the mark.

Under the plan, Social Security and most antipoverty programs would be exempt from cutbacks. Cuts would be divided evenly between other domestic programs and defense, starting with a forced reduction of about $12 billion in outlays for the current fiscal year. The provisions could be waived only during recessions or wartime.

The administration was allowed flexibility to allocate cuts within specific program categories of the military budget for fiscal 1986 only. This was done in large part to satisfy the White House and Pentagon, which had complained that defense might otherwise be harmed by the measure.

The agreement makes no provision for tax increases to help meet the deficit targets, but Senate Finance Committee Chairman Bob Packwood (R-Ore.) and Senate Budget Committee Chairman Pete V. Domenici (R-N.M.) indicated it will be difficult, if not impossible, to meet the targets by spending cuts alone.

With a deficit target of $144 billion for the budget the administration is drafting for fiscal 1987, 30 to 50 domestic programs would have to be terminated if the administration continues to insist on 3 percent annual growth in defense spending without a tax increase, Packwood said.

The agreement, announced at a news conference held by the chief negotiators and principal Senate sponsors Phil Gramm (R-Tex.), Warren B. Rudman (R-N.M.) and Ernest F. Hollings (D-S.C.), followed two months of struggle over the legislation, which had been tacked onto legislation raising the debt ceiling to $2 trillion. The debt measure, delayed several times because of the budget dispute, must be passed before Thursday to prevent a default by the government.

The pact will go to the full House-Senate conference on the bill Monday. If the conference accepts it, as expected, it will go to the House and Senate on Wednesday.

Despite objections that have been raised by the White House, Domenici said he expected Reagan will sign the bill. "I believe the White House will be satisfied and the president will sign it. Then he will have a hard time producing a budget," Domenici said.

White House assistant press secretary Albert R. (Rusty) Brashear said of the agreement, "We think this is certainly a step in the right direction. There are some technical questions that remain, some of which cannot be answered until the conference resumes on Monday, but all in all, this is a positive step."

The agreement sets a deficit target of $172 billion for the current fiscal year, about $30 billion less than is now expected by the administration. The initial mandatory cutback under the program would be limited to about $12 billion in outlays. These cuts in outlays, or actual expenditures, would have to be made by March 1.

Congress would have to reduce spending authority for certain programs, including defense, by much more, however, to achieve the savings. For defense, negotiators estimated that $15 billion to $18 billion would have to be cut from spending authority, currently being set by Congress.

The agreement seeks to limit presidential authority in administering the cuts by putting budget estimates jointly under control of the Congressional Budget Office and the administration's Office of Management and Budget, with the General Accounting Office -- an agency with ties to both branches -- as arbiter.

Exemption of antipoverty programs and limited protections for health programs were a major concession by the Republican-controlled Senate to win agreement from the Democratic-run House.

In addition to Social Security, which both houses exempted for political reasons, the agreement exempts Medicaid, Aid to Families with Dependent Children (AFDC), Supplemental Security Income (SSI), food stamps and other nutrition programs and veterans' compensation and pensions.

Medicare and other health programs would be subject to cuts of no more than 1 percent this year and 2 percent thereafter.

Meanwhile, the Senate opened debate on a $498 billion omnibus spending bill for the government that has been threatened with a presidential veto.

Although the Senate version of the continuing resolution comes closer to Reagan's budget benchmarks than a version passed earlier by the House, the White House said yesterday that the Senate measure also was unacceptable.

"I'm convinced OMB Office of Management and Budget is determined somehow to find an excuse to veto any continuing resolution we send down there," said Senate Appropriations Committee Chairman Mark O. Hatfield (R-Ore.). Hatfield took issue with some of the administration's complaints about Senate proposals for domestic spending, accusing the White House of "changing the goal posts" to suit its purposes in order to "have some kind of confrontation."

The bill must be passed by Thursday night, when stopgap funding for the government expires. It is needed to fund agencies for which Congress has not passed regular appropriations bills for fiscal 1986. Congress has passed seven bills. Four have been signed. One has been vetoed. And two were sent to Reagan yesterday.

The bills approved yesterday provided $105.8 billion for the departments of Education, Labor and Health and Human Services and $11.9 billion for the departments of State, Justice and Commerce.