In the first votes on Social Security in the new Congress a House subcommittee yesterday approved major elements of a $165 billion rescue plan that would defer this year's cost-of-living increase in benefits, raise next year's payroll taxes and make all new federal employes part of the system.

The cost-of-living allowance would be deferred from July to next Jan. 1, and paid every January thereafter. In addition, the bill would tax half the Social Security benefits received by higher-income people.

Under the bill the president, vice president, members of Congress and federal judges also would become subject to Social Security.

Except for federal judges, this provision would be effective on enactment, meaning that current members of Congress plus President Reagan and Vice President Bush would be covered.

In addition, all employes of nonprofit organizations--present and future--would be required to join. They now have a choice, and 1 million have chosen to stay out of Social Security.

All of the major provisions approved by the Ways and Means subcommittee were put forward earlier this year by the president's bipartisan Social Security advisory committee and endorsed by Reagan and House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.).

For lack of agreement, however, two crucial and controversial parts of the rescue plan were put off until today.

One involves long-term financing. To keep Social Security whole in the long run, many Democrats favor raising taxes beyond what already is scheduled beginning in the year 2010.

Republicans lean instead toward gradually raising the retirement age from 65 to 66 early in the next century, which O'Neill and House Rules Committee Chairman Claude Pepper (D-Fla.) strongly oppose.

The second unresolved issue is finding a "fail-safe" mechanism that would kick in automatically to shore up the system if the economy falls short of expectations and revenues are lower, or outlays higher, than now forecast.

Democrats have proposed letting Social Security borrow from the Treasury, with repayment required in three years. Republicans oppose this as a kind of blank check.

Yesterday's session started with some partisan skirmishing, but quickly smoothed out as most proposals were adopted in rapid order by bipartisan voice vote.

The new federal employe provision would apply to those hired after Jan. 1, 1984, and would cover not only executive branch employes but those working for Congress and the judiciary.

Although the subcommittee voted to postpone future cost-of-living allowances, or COLAs, one new cost-of-living benefit was added by Rep. Barber B. Conable Jr. (R-N.Y.). It would allow a COLA to be paid next January even if the consumer price index rises less than 3 percent this year.

Under current law, increases of less than 3 percent need not be paid.

The bill advanced the effective date of Social Security tax increases on employers and employes, now scheduled for 1985 and 1990, so that the rate next January would be 7 percent instead of 6.7 percent on the first $37,800 of pay. The rate would rise more thereafter.

The bill also would require self-employed persons to pay 100 percent of the combined employer-employe tax rate instead of 75 percent.

Also included in the bill are certain benefits for wives and widows, a provision reducing the COLA to the lower of wages or prices, starting in 1988 if trust fund reserves fall, and a provision keeping Social Security in the unified federal budget.