President Carter heard out the congressional opponents of the B-1 bomber yesterday but provided little reason to stem the growing impression that he will order at least limited production of the controversial aircraft.

In the aftermath of the 40-minute meeting a dispute arose between the White House and some participants over a Carter remark concerning Defense Secretary Harold Brown's attitude toward the plane.

Reps. Robert Drinan (D-Mass.) and Thomas Downey (D-N.Y.) said the President told them that Brown favors production of the B-1.

"He made a reference to the fact that Brown was in favor of it and Brown smiled," Downey said.

But the White House, in a statement released yesterday afternoon, said the President's comments referred to Brown's general attitude toward the B-1 before he took over at the Pentagon. "Secretary Brown has not made a recommendation to the President on the matter and the President has not made a decision on it," the statement said.

Brown is one of the most influential members of the Cabinet. Whatever he recommends is bound to weigh heavily as Carter makes by far his most important defense policy decision since taking office.

The Air Force is seeking to build a fleet of 244 supersonic-speed B-1s at an estimated cost of $24.8 billion - $101.7 million each.

In recent days there have been suggestions of a compromise involving limited production of the B-1. After a similar meeting with Carter Tuesday, Sen. Barry Goldwater (R-Ariz.), a member of a congressional delegation supporting the plane, suggested that a fleet of 155 would satisfy U.S. strategic needs.

Two of these who met with the President yesterday, Drinan and Rep Ronald Dellums (D-Calif.), said they came away with the impression Carter will approve production of the B-1.

"I think the President is leaning very definitely toward building it," Drinan said.

But most others disputed the impression of Drinan and Dellums. "I don't think there is any way to tell which way he is leaning and he gave us no indication," said Sen. John Culver (D-Iowa).

The decision is important not only because of the enormous cost of the plane and its possible impact on strategic arms limitation talks with the Soviet Union, but also because approval of the aircraft would further anger liberal Democrats who believe Carter has backed down on other campaign promises.

Last July, Carter told the Democratic National Committee platform committee, "The B-1 bomber is an example of a proposed system which should not be funded and would be wasteful of taxpayers' dollars."

Sen. George McGovern (D-S.D.) said yesterday he reminded the President of his campaign statements and "urged him to stay with that position to protect his credibility."

As the meeting began, with reporters and photograhpers present, Carter underscored the importance of the decision - unintentionally evoking memories of former President Nixon's decison-making style.

He called it "a lonely decision," which he will make by the end of the month. "I will probably pretty well go into seclusion and study what the information is and make a decision," the President said.

The Air Force argues that the nation's fleet of 25-year-old B-52s is growing obsolete. The B-1, according to studies the Pentagon commissioned for Carter-s consideration, would be the most effective way to maintain the "triad" U.S. defense posture - land-based missiles, sea-based missiles and manned bombers.

Opponents argue that the age of all manned bombers is rapidly ending and that even if a need for bombers continues there are cheaper ways to meet it.

"The B-1 bomber is a weapon whose time has passed," Rep. M. Robert Carr (D-Mich.) said after the meeting.