The chairman of the House Energy Committee said yesterday that passage of President Carter's energy program could be jeopardized by signs that Carter is "soft" on returning the proceeds of his proposed energy taxes to the people.

Rep. Thomas L. Ashely (D-Ohio), head of the ad hoc panel shepherding the energy legislation through the House, said Carter did not seem "firm" on refunding the money to be raised by oil and gas taxes. He said any wavering on that point would complicate prospects for congressional approval.

Bryd told reporters over the weekend that he had urged Carter to use some of the energy taxes to provide jobs for the hard-core unemployed in such energy-related fields as repairing railroad roadbeds and renovating subway cars.

He quoted the President as saying that he "expects Congress to provide alternative proposals."

Others at the White House leadership meeting said Byrd had told the President he was very doubtful about rebating the energy taxes to the public through individual income tax credits, as Carter has proposed. Carter, they said, indicated taht if Congress preferred to use the money diffrently, he had no objections.

It weas this attitude which prompted Ashley to say that "in this area, as in others he (Carter) has been soft."

In an April 22 press conference, the President first said the gasoline tax he has proposed would be "refunded directly to every person" as an income tax credit. Later, he said he needed "some flexibility" and could not "certify today that every nickel" would be returned.

Ashley said that if Carter did not insist on returning substantially all the energy taxes to the people, as he first proposed, "the liberals" in Congress would try to use the money to finance welfare reform or other social legislation.

That, he said, could jeopardize passage of the Carter tax package.Whiel the administration has not furnished firm figures to Congress, unofficial estimates are that the crude oil equalization (wellhead) tax, industrial fuel taxes and retail gasoline taxes could amount to $80 billion a year by 1988.

"That can represent a hell of an impact in the blue-collar worker" if the money is not returned through tax credits, Ashley said.

Despite his expressed concern on this point, the energy committee chairman said, "The odds are good there'll be a major piece of (energy) legislation this year and it will be recognizable" as the Carter proposal.

On the other hand, Ashley said, failure in the energy area by Carter "would reduce his credibility to the point he'd have difficulty getting a resolution through in support of the flag,"

Aslhey said he thought Carter might have made a mistake in insisting on the April 20 deadline for submitting his energy program. "When he said in January he'd have it on April 20, I don't think he knew what he was talking about," Ashley said.

On the other hand, he added, Carter's insistance on that deadline did communicate "sense ofurgency" to the country and make it possible for Congress to pass the energy legislation within the same calendar year.

But he said the administration estimates on the economic impact of the program and the investment costs for meeting the coal conversasion and nuclear power targets were shaky enough to require "close scrutiny" by Congress.