President Carter met yesterday with opponents his decision to return the Crown of St. Stephen to Hungary and told them that, while he understands their concern, he intends to go ahead with the plan.

Administration sources said Carter joined a White House meeting between Vice President Mondale and a 15-member delegations that had come to discuss the plan to return the 1,000-year-old relic, the symbol of Hungarian sovereignty. The crown came into American hands for safekeeping at the end of World War II.

Carter told the visitors, not all of whom were against his proposal, that, despite critisms of the regime of Hungarian leader Janos Kadar the Hungarian people need to have a symbol of their Heritage.

He said he feared the crown would become "a distant memory" if it remained in this country.

The President stressed that the return would not reflect an national endorsement of communism. "I deplore communism," Carter reportedly said. He added that he wants the blessing of Hungarian church leaders for the return of the crown.

Carter said he had not decided how or when the crown should be given back, and asked the group for its advice, sources said.

Earlier at a House hearing opponents of the return accused Carter of reneging on his promise to promote human rights throughout the world.

About 200 Hungarian-American and their supporters demonstrated against the return in front of the Capitol and then crowded into the standingroom-only hearing called by the House International Relations Subcommittee on Europe and the Middle East to discuss Carter's plan, which became known last week.

Most of the spectators cheered as Rep. Mary Rose Oaker (D-Ohio) said the present Hungarian regime "is not a government worthy of receiving this sacred crown."

Oaker, who represents a large Hungarian-American constituency in Cleveland, recalled that Carter said during a campaign debate with then-President Ford last year that people in several Eastern European countries, including Hungary, were under Soviet domination.

By proposing to return the crown, "the President appears to be repudiating his own human rights policy insofar as the Eastern Europeans are concerned," she said.

At the end of her testimony Oaker asked two colleagues to display a pre-1975 red, white and green Hungarian flag with the nation's coat of arms topped by the famed crown in the center. "When the Communists took over, they replaced the crown and coat of arms with the hammer and sickle," she remarked. The crowd booed the Soviet symbol.

The subcommittee adjourned without taking any action on a bill by Oaker to prevent the return without congressional approval. Congress, which is in semi-recess, is not expected to try to stop the return.

The subcommittee heard 13 witnesses opposed to giving back the crown and five in favor.

Rep. Bill Frenzel (R-Minn.) was booed as he called Carter's decision "an act of great courage and fundamental morality." He added, "We have recognized the government of Hungary. The Cold War is over."

"No, it isn't," shouted several people in the audience.

"How long will we have to wait to find a government that we like in Hungary?" Frenzel asked. "The fundamental question is that we are holding, without authority, something that doesn't belong to us. It belongs to the people of Hungary. We are not giving the crown back to Mr. Kadar. We are giving it to the people of Hungary."

Rep. Frank Horton (R-N.Y.) said returning the crown, which Pope Sylvester II gave to Stephen, Hungary's first king, was nothing more than "a political gift from our President to Mr. Kadar. . . . How would the world have reacted if, in 1941, the chief rabbi of Warsaw had presented a sacred Torah to Adolph Hitler in recognition of his 'fine' treatment of the Jewish people?"

Andas H. Pogany, spokesman for the Coordinating Committee of Hungarian Organizations of North America, argued that the crown "cannot be returned to the Hungarian people because there is no legal and constitutional representation of the Hungarian people in Hungary."