An influential House subcommittee chairman dealt an unexpected and perhaps fatal blow yesterday to the Carter administration's plan for slowing the upward spiral of hospital costs.

Rep. Dan Rostenkowski (D-Ill.), chairman of the House Ways and Means health subcommittee, announced he will ask his panel to set aside the administration bill. He said it would consider instead a proposal that would rely on voluntary action to reduce hospital spending and trigger federal action only if that failed.

The announcement caught many members of Congress and the administration by surprise. It was made at a meeting of the House of Delegates of the American Hospital Association, one of the most vehement opponents of the hospital cost containment legislation.

Health, Education and Welfare Secretary Joseph A. Califano Jr., whose relations with Rostenkowski are considered chilly at best, was not informed of the move by Rostenkowski prior to yesterday's speech.

President Carter referred to the cost containment bill in his State of the Union Message as "one of my main legislative goals for this year" and "our principal weapon in the effort to decrease health care costs." Health care spending has been increasing at an annual rate of 15 percent for the last several years and now is 8.6 percent of the gross national product.

The administration proposal has two main parts: a limit of 9 to 12 percent on increases in individual hospital expenditures and a $2.5 billion annual national limit on spending for hospital construction.

Progress on the bill since its introduction last April has been slow. Only one of the four committees that must act on the bill - Senate Human Resources - has done so. Rostenowski said that members of his subcommittee had "raised many fundamental questions which were left unresolved at the conclusion of the first session."

On difficult issues Rostenkowski was unable to muster more than a 7 to 6 majority for the administration's position in his subcommittee. "It was a question of whether or not I had the enthusiasm to carry through the full committee," Rostenkowski said after his speech.

The voluntary effort Rostenkowski proposes to rely on was put together by the American Hospital Association, the Federation of American Hospitals and the American Medical Association.

Under Rostenkowski's proposal - important details of which have yet to be worked out - hospital costs would be reduced 2 percent this year and 2 percent next year from an "appropriate" but still undecided benchmark. If those voluntary goals were not met, then Rostenkowski proposed that federal legislation would activate controls.

Additionally, Rostenkowski would transfer the limits on expenditures for construction to the health planning act, which is up for renewal in Congress this year.

Sen. Edward M. Kennedy (D-Mass.), whose Senate Human Resources subcommittee left the administration's proposal substantially intact, said the voluntary effort was "doomed to failure" and that he remains committed to the bill his committee reported. Rep. Paul G. Rogers (D-Fla.), chairman of the House Interstate Commerce health subcommittee, said he could not accept Rostenkowski's proposal.

In an interview yesterday, Califano said he remained "optimistic" about getting legislation passed. But he rejected the voluntary approach outlined by Rostenkowski. "We don't think there's any way of achieving this on a voluntary basis," Califano said. The voluntary effort would wind up costing Americans billions, he said.

Rostenkowski said he hoped to have a bill for his subcommittee to consider by Feb. 16. "We may be able to bring that closer to us than he's indicated," Califano said.