A last-ditch effort to put a federal limit on rising hospital costs was begun yesterday by Sens. Gaylord Nelson (D-Wis.) and Edward M. Kennedy (D-Mass.) and Health, Education and Welfare Secretary Joseph A. Califano Jr.

Kennedy and Califano publicly backed a Nelson proposal to put controts into effect if hospitals fail to limit their cost increases to 12 percent yearly for the next five years. Hospitals meanwhile could continue their present voluntary cost control efforts without federal interference as long as they succeed.

But Kennedy and Nelson will have to fight on the Senate floor for the plan. They will try to make it an amendment to a measure sponsored and approved by the Senate Finance Committee.

Califano said the Nelson plan could save the public $33 billion by 1983, compared with less than $400 million that might be saved by the Talmadge plan.

They blamed the medical lobby for blocking an administration-backed bill earlier this summer.

President Carter in April 1977 first tried to get Congress to put a 9-percent-a-year lid on hospital price increases. Only the Senate Human Resources Committee - whose health subcommittee Kennedy heads - has approved such legislation.

The House Commerce Committee voted to establish a federal commission to watch hospitals' voluntary efforts.

The Senate Finance Committee watered down the Talmadge plan to establish a degree of control over Medicare and Medicaid bills. That bill would cover only room and board charges, not charges for X-rays, laboratory tests and other "ancillaries" that add up to 50 to 60 percent of the average hospital bill.

Kennedy said it may take "parliamentary" moves to get the Nelson plan to the floor. Senate observers gave the effort less than an even chance.