A Social Security rescue plan backed by both President Reagan and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) was approved last night by a 12-to-3 vote of the president's Social Security advisory commission.

The plan would provide $169 billion in tax increases and benefit reductions over the next seven years to keep the system from default.

It calls for mandatory Social Security coverage for new federal workers and all employes of nonprofit organizations; acceleration of Social Security payroll taxes already scheduled to go into effect in 1985 and 1990; a six-month delay in this year's cost-of-living benefit increases, and taxation of half the Social Security benefits received by high-income retirees.

"Each of us recognizes that this is a compromise solution. As such, it includes elements which each of us could not support if they were not part of a bipartisan compromise," Reagan said. "However, in the interests of solving the Social Security problem promptly, equitably and on a bipartisan basis, we have agreed to support and work for this bipartisan solution."

O'Neill called the plan "a bipartisan agreement acceptable to the president and to me, one which I can support and which I will work for."

The only members of the 15-person commission to oppose the compromise were Sen. William L. Armstrong (R-Colo.) Rep. Bill Archer (R-Tex.) and former Rep. Joe Waggonner Jr. (D-La.). AFL-CIO president Lane Kirkland said he couldn't back the provisions on federal employes but endorsed the rest of the agreement.

Rep. Claude Pepper (D-Fla.), a member of the commission, declared after the vote, "The nation's 36 million Social Security beneficiaries and 116 million contributors can breathe a collective sigh of relief." But Armstrong, in a dissent, said the proposal leans too heavily on tax increases.

Senate Majority Leader Howard H. Baker Jr. (R-Tenn.), in a statement from Nashville pledged to work for the plan and called it "sound." Sen. Robert J. Dole (R-Kan.) said he believed the plan also had the assent of House Ways and Means Chairman Dan Rostenkowski (D-Ill.) and the House Republican leader Robert H. Michel (R-Ill.) issued a statement supporting it.

The agreement, which will be sent to Congress, was announced after 12 hours of bargaining yesterday between White House aides and leaders of the commission, capping a year of study and two weeks of intensive bargaining between commission leaders and a White House team headed by presidential chief of staff James A. Baker III.

Only a few weeks ago chances of a compromise appeared hopeless. But the White House was eager to include any savings the plan might produce in its fiscal 1984 budget, and anxious, as were commission members, to head off a tremendous partisan fight in Congress. So the White House agreed to a last round of bargaining suggested by Sens. Daniel Patrick Moynihan (D-N.Y.) and Dole, commission Chairman Alan Greenspan, former Social Security commissioner Robert Ball (R-Kansas), Rep. Barber B. Conable Jr. (R-N.Y.) and others.

Three times yesterday Baker,and presidential aides David A. Stockman, Richard Darman and Ken Duberstein walked across Pennsylvania Avenue to Blair House to talk to the commission leadership group with whom they were trying to negotiate a compromise. Each time Baker would only say: "We're still working."

As night fell, Baker and the other aides met with the president, apparently to detail the best offer they could get. Then it was back across Pennsylvania Avenue, through a pack of TV cameramen and reporters, for a short, 20-minute final meeting with the commission leadership group.

The commission has estimated that $150 billion to $200 billion will be needed over the next seven years to keep the system solvent. The $169 billion in the commission's package would fill that gap.

The commission also has estimated that over the next 75 years, the system will face a deficit of 1.8 percent of taxable payroll. The agreement reached last night would wipe out more than two-thirds of this long-range deficit. The group proposed several alternative solutions to eliminate the remaining deficit, 0.58 percent of payroll.

The commission plan included these provisions:

Mandatory Social Security coverage would be extended to new federal employes and to all nonprofit organization employes as of Jan. 1, 1984. Federal employes are not covered by Social Security at present on a mandatory basis, although some qualify because they once worked in the private sector, and nonprofit organizations may join if they choose. State and local government agencies now covered would be forbidden to withdraw. Savings over calendar years 1983 through 1989: $23 billion.

A three-tenths of 1 percent increase in the Social Security tax already scheduled for 1985 would be advanced to 1984; and three-quarters of a five-tenths of 1 percent increase scheduled for 1990 would be advanced to 1988. Workers would be able to offset the 1984 increase against income taxes. New revenues through 1989: $40 billion.

Self-employed persons would be required to pay the same total tax on their incomes as an employer and employe together now pay on a worker's salary of the same amount, instead of paying only three-quarters of the combined employer-employe tax. The self-employed would be allowed to offset half their payment. New revenues: $18 billion.

Social Security beneficiaries would be required to pay income tax on half their benefits, but only if adjusted gross income from all sources exceeded $20,000 a year in the case of a single person or $25,000 in the case of a married couple filing jointly. Social Security benefits are not taxed now. The money received from taxing the benefits would be fed back into the Social Security trust fund to help keep it solvent. New revenues: $30 billion.

This year's automtatic cost-of-living increase in benefits would be postponed for six months, from July to January, and thereafter would be paid in January for a $40 billion gain.

The system would be reimbursed for certain credits granted to members of the armed forces, and also would receive credit for Social Security checks that go uncashed. New revenues: $18 billion gain.

Certain benefits for widows, divorced wives and their male counterparts would be improved, at a negligible cost.

Beginning in 1988, if trust fund reserves fall below a fifth of a year's benefits, the annual cost of living increase would be based on the lower of wages or prices.

The package also provided that starting in the year 1990, people retiring after 65 would receive an 8 percent higher benefit for each year they delayed retirement.

The president extended the commission's deadline for making its report until Thursday.