President Carter's plan to curb hospital costs was junked by the House Commerce Committee yesterday in a major defeat for the adminstration's anti-inflation fight.

The committee voted 15 to 12 for substitute legislation endorsing a voluntary effort by hospitals to reduce their costs and to establish a federal commission to recommend action if the hospitals effort fails.

Carter had wanted mandatory federal cantrols if over two years the voluntary effort did not slow the increase in hospital costs.

Health Education and Welfare Secretary Joseph A. Califano Jr. called the substitute, offered by Rep. James T. Broyhill (R-N.C.) a "sham" and "an affront to the American people." He added that the most effective possible short-term weapon against inflation has been thrown away.

Rep. Paul G. Rogers (D-Fla.), chairman of the committee's health subcommittee and a leading supporter of controlling hospital costs, said the Broyhill substitute "simply guts the adminstration bill".

Administration officials and Capitol Hill backers began meeting hurriedly yesterday afternoon to try to find a new strategy. But most Hill observers said the Carter plan, which would have put a 9 percent annual lid on hospital cost increases if the voluntary effort failed, is probably dead in this session.

The president announced that goal three months after taking office in 1977 and asked Congress to rush the controls into place by last October. A slightly modified version of his plan, exempting wage increases for hospital workers, was passed by the Senate Human Resources Committee in August 1977.

But it has never reached the Senate floor.

Last February the House Ways and Means health subcommittee voted to give hospitals two years to control cost themselves - but said federal controls should automatically go into effect if hospitals failed to reduce the rate of inflation by 2 percent each year.

Eight weeks ago the administration unofficially settled for that compromise, and Rogers begin trying to win Commerce Committee approval.

Hospital and American Medical Association lobbyists opposed anything but the voluntary cost-control effort, which they say has already cut the hospital inflation rate from 15.8 to 12.7 percent this year.

Labor lobbyists opposed any bill that didnot leave their workers free to win wage boosts to counter general inflation. Republican committee members - and Democrats James D. Santini of Nevada and David E. Satterfield of Virginia - sponsored a series of amendments to weaken the Rogers adminstration [WORD ILLEGIBLE]

Members heard from hospital trustees, hospital administrators and doctors from their districts. These constituents said the Carter curbs would hurt medical care by keeping hospitals from buying essential equipment.

Except for older persons' and retire eacher's groups, no major constituencies emerged in favor of curbing hospital costs, despite the fact that, in polls, many Americans always identify health costs as a major problem.

Yesterday Broyhill offered an amendment to do two things: kill the threat of mandatory controls and in effect kill an overall $4 billion a year limit on hospital spending for major new equipment and any construction that adds new beds.

Broyhill moved to apply the $4 billion limit only to adding new beds. Hospitals are adding $1.2 billion worth of beds this year.

As a vote approached, it became clear that Rogers had lost the support of Rep. Martin Russo (D-Ill.), who, until last week, was with him on almost every point. Russo said later that, "With all the amendments we passed, I became convinced the bill wouldn't accomplish what it was supposed to," and "Increasing regulations on the business community just don't seem to be working."

With many administration supporters in effect giving up, the committee then voted 15 to 12 to pass the watered-down bill wth the Broyhill appendage.Rogers voted for the Broyhill version "to get this matter out of committee," he said.

But he said the committee "has not acted responsibly," adding that the public should know that a chance to save the public $26 billion in hospital bills, by administration estimate, over the coming five years had been lost.

Rogers said yesterday there was a chance that the House Ways and Means health subcommittee might revive the Carter bill, but Rep. Dan Rostenkowski (D-Ill.), the subcommittee chairman, said it would be "difficult."

Rep. Al Ullman (D-Ore.), chairman of the committee, has told administration officials he is pessimistic about the bill's chances.

One other avenue available to the administration is the Senate Finance Committee, whose health subcommittee is considering a different approach to controls.