The Senate Finance Committee voted yesterday to increase the Social Security taxes paid by employers a total of $40 billion over the next five years.

The committee would do that by requiring employers to pay the tax on the first $100,000 of each employee's earnings, starting in 1979, instead of the first $18,900 - the scheduled cut-off under existing law.

The senators, seeking new funds for the ailing Social Security system, also voted to increase this so-called wage base for the part of the tax paid by employees. But their increase would be gradual and much smaller, so that by the year 1985 they would be paying on about the first $30,300.

In addition, the committee decided to increase the Social Security tax rate, which is now 5.85 per cent for both employers and employees and is scheduled to rise to 6.05 per cent next year. It would go up in stages there after to reach 7.035 per cent by 1986 and 9.10 per cent by 2011.

In other action on a busy day, the senators wrote a new money-saving benefit formula, so that in the future the average retired worker's first benefit check would be about 43 per cent of his last paycheck, slightly below the present average of 45 per cent.

The senators also agreed to raise the amount a recipient can earn and still collect full benefits. That limit is now $3,000 a year, and would rise to $3,240 next year under present law, which makes yearly allowances for inflation. The committee voted to lift it to $4,500 next year and $6,000 thereafter. A House Ways and Means subcommittee voted this week to do the same.

The wage base changes approved by the committee would be a Social Security milestone. For the first time, employer would pay more than employee, instead of both paying the same.

The wage base changes were part of a comprehensive financial rescue plan put together by Sen. Gaylord Nelson (D-Wis.) and pushed through the committee yesterday with the backing of Chairman Russell B. Long (D-La.).

Without these changes, the system would run out of money in a few years.

Social Security Commissioner James B. Cardwell said the final product is "very close to the administration proposal" even though it drops President Carter's request to let Social Security take some income tax revenues from the Treasury to improve its financial condition.

Almost everything the committee did yesterdsay was intended to raise funds or reduce the growth of benefits far off in the future, and thus save monye. However, on an 11 to 0 vote it did throw in one big benefit improvement to sweeten the pain . . . the amendment by Lloyd M. Bentsen (D-Tex.) and Nelson permitting retirees to earn more each year without losing benefits.

Taxes to pay for this new benefit were included in the proposed tax schedule, but they would not be levied until 1979. Many elderly persons argue they simply can't live on their pensions plus earnings of only $3,000 a year.

The raising of the employer wage base to $100,000 was the most radical change voted yesterday, and it was strongly resisted by Carl Curtis (R-Neb.) and other Republicans who argued it would throw too heavy a weight on business.

For years social security benefits have been financed by a payroll tax of exactly the same rate for employer and employee, levied on exactly the same wage base. It currently is 5.85 percent each of the first $16,500 of earnings and will rise to 6.05 percent each on the first $17,000 each under existing law on January 1.

Curtis argued that the fair way to raise more money for the system would simply be to raise the payroll tax by one-half per cent each on employers and employees, levied on the same wage base for both.

John C. Danforth (R-Mo.) said the Nelson plan would have "an extremely hard effect" on some emploers, but Long said they would simply pass along the extra cost to consumers.

"Every proposal is unpopular because every proposal has to raise revenues," said Danforth. But Nelson laughingly retorted that his package, including the wage base rise, "is less worse than any other one I've looked at."

The administration had sought outright abolition of the wage ceiling for employers, in three steps. It wasn't unhappy with the final provision, which was initially approved on a 10 to 4 party-line vote, and then put into its final form by voice vote. Curtis' plan was beaten, 10 to 6.

In addition to dropping the administration plan for using income tax revenues to help finance the social security system, the committee at the insistence of Herman Talmadge (D-Ga.) aslo dropped a proposal to divert $13 billion to the disability and old-age funds from the health insurance fund.

Talmadge called the idea absurd, saying the health insurance fund is also running out of money, though more slowly than the other two. Instead, the proposed tax rate for te two cash funds was raised more than Nelson had initially proposed.

Nelson said his final plan, as adopted by the committee, would put the Social Security old-age and disability funds into the black on an actuarial basis for the next 75 years, assuring its soundness and assuring all benefits will be paid.

At present a worker with maximum taxable earnings pays 5.85 per cent of $16,500 or $967.25 a year and so does his or her employer. By 1981, under formulas in the bill, the worker will pay about 6.6 per cent of the first $23,100 of his or her earnings - about $1,521.

By 1987, to take another example, the rate would be 7.035 per cent, the probable wage base $33,900 and the worker's tax $2,384.86.

The committee voted to raise the employee and self-employed taxable wage ceiling by $600 a year, in 1979, 1981, and 1983, and 1985, beyond the annual raises based on cost-of-living increases under current law. This would produce an employee wage base of $33,900 by 1987, and yield about $3.4 billion in new revenues over next five years.

The employer-employee tax rate would rise to 6.135 per cent in 1979, 6.585 per cent in 1981, 6,885 per cent in 1985, and 7.035 per cent in 1986, with further steps startin in 1990 which will boost it to 9.10 percent by 2011. Self-employed would pay 1 1/2 times the employee rate. Resulting new revenues: about $4.5 billion over the next five years.

The senators also decided to reduce benefits to dependents and survivors by whatever amount they received from other public retirement or disability pension systems - saving $3.2 billion over five years.