President Carter sounded a conciliatory note toward Congress yesterday, but then turned around to warn that only the pressure of an aroused American public will prevent the victory of special-interest groups now seeking to dismantle his energy program on Capitol Hill.

At a nationally televised press conference, the President said he has never "criticized the Congress as a whole" and he expressed confidence in "the sound judgment of the Congress." But in a statement similar to one made by White House press secretary Jody Powell Friday, aftercongressional committees had dealt the administration a series of setbacks on the energy plan, he added:

"I am deeply concerned about the inordinate influence of the lobbyists and representatives of the oil companies and the automobile manufacturers . . .

"It is important that the American people be aroused to the fact that unless they are deeply involved in helping the Congress and me to come up with a substantive, comprehensive, fair and adequate energy policy, that the special-interest groups will prevail."

Carter said that if he and Congress fail to produce an adequate energy program they will deserve "legitimate criticism by the American people for timidity and for an absence of concerns about what I still consider to be the gravest domestic issue which I shall face during my own tern as President."

On Friday, Powell, speaking for the President, virtually accused congressional committees of caving in to pressure from auto and oil lobbyists. This was after the committees killed the administration's standby gasoline tax and its rebate plan for high mileage cars, weakened a provision to tax gas-guzzler cars and voted, against Carter's wishes, to decontrol the price of new natural gas.

The President said yesterday that no single setback was of crucial importance, but that the "cumulative effects" of a series of setback leading to a failure to reach his energy goals "would be catastrophic."

Carter sounded two other themes that are becoming familiar in the increasingly public dialouge with congressional leaders over the fate of the energy program. The first was to reject a suggestion that the administration's own lobbying for the program has been inadequate. He said his views were being presented to Congress "in an adequate fashion."

The second was to suggest that Congress develop its own proposals to reach the administration's energy goals.

"I might say we don't consider ourselves to be infallible," the President said. "Over the three or four months that we considered this plan before it was presented to Congress, there were a lot of differences of opinion. Some of the judgments we made were quite closely called ones.

"The Congress is now finding an equal difficulty in dealing with the controversial issue. So I don't say that everrything we propose has got to be passed just as though weput it forward."

On other topics during the press conference, Carter:

Said he will decide "in a lonely way" later this month whether to approve production of the controversial B-1 bomber, sidestepping a question on why he is struggling with the issue when during his campaign he dismissed the B-1 as "wasteful" and said it should not be produced.

Announced that he will soon appoint a commission to study the entire question of retirement benefits, including the practice of "double dipping" by government employees who draw benefits from other government units. Early in his administration, the President said he would name a similar panel to study military retirement benefits but he has not thus far.The commission he mentioned yesterday apparently would study both public and private retirement systems.

Said he will soon receive a recomendation on restructuring the U.S. intelligence community. He said he has not "predisposition" on the issue, but emphasized the importance of "diversity of opinion" in assessing intelligence information.

Expressed hope that "justice will be done" in the case of the so-called "Wilmington 10," but said it would be improper to comment on the case beyond that.

Attorney General Griffin B. Bell has promised to investigate the case, which involves the convictions of a black United Church of Christ minister and nine others on charges stemming from school desegregation-related violence in Wilmington, N.C.; in February, 1971.