Walter Mondale went home today after a diplomaic tour de force in Europe that set an assertive, liberal tone for American foreign policy, which stands in sharp contrast to the Carter administration's more cautious approach to domestic problems.

The Vice President, who left London for Washington this afternoon, used the 10-day voyage to put his own imprint on African policy and to try to infuse it with the moral fervor and conviction of the civil-rights crusade in the American South a decade ago.

Echoes of Selma and Birmingham wafted into Vienna's Hufburg Palace, where Mondale faced South African Prime Minister John Vorster to argue race relations last week. At times, Mondale's Air Force airliner seemed to be serving as an intercontinental versin of a freedom riders' bus as it swept across Europe.

Despite Vorster's firm rejection of Mississippi and Alabama as signposts for South Africa, Mondale betrayed no doubt that the civil-rights crusade can be updated for export. He pressed the analogy home in his meeting with Vorster and in nearly all of his public statements.

"I think most Americans feel very good about having come clean on this issue in the last 10 years," Mondale told reporters on the plane from Belgrade to London yesterday.Mondale and some fo the younger aides accompanying him were deeply involved in the civil-rights movement in the 1960s.

"We expect very broad, very deep public support," for the policy toward South Africa, extending beyond the American black community, Mondale predicted. The Carter-Mondale ticket, he said, carried every Southern state. There has been a transformation of American society that we can be proud of."

Emphasizing the high domestic content of the administration's foreign policy enabled Mondale to remind Americans of a central theme of the Carter administration - that the American people can put Vietnam behind them and feel good again about the country's place in the world.

On another level, the meeting with Vorster and the stopovers to talk human rights with European leaders offered Mondale a chance to channel his liberal instincts into foreign policy. He seized it, returning eagerly to a battle-ground that most liberals see as a clear-cut contest of good and evil.

Since the election, such chances have been rare in domestic politics, dominated thus far by the gray decision-making of budget priorities and maintaining business confidence. The administration has come under criticism for pullbacks on stimulating the economy and welfare reform from some of the liberal Democrats who where Mondale's traditional political allies.

Mondale defended the administration's domestic policies during a session with reporters on the trip, but he did indicate in response to questions that "very, very severe" budget restraints were impeding some programs he favored.

"There are always hopes for more that are understandable. But remember we have only been in office a few months. I think we're on course and it's a good course," he said.

Mondale emerged on this trip as an effective and at times eloquent spokesman for the moral type of policy that Carter seems determined to press abroad. He frequently lightened his mission, however, by flashing a sharply honed wit. It seemed to fail only with Vorster.

The dour, stolid Vorster spent much of the eight hours of talks reciting to Mondale population statistics of tribal nations in South Africa, wage levels of black workers and other data. At one point, Mondale pulled out of his briefcase a gargantuan almanac published by the South Africa Information Service and began reading back to Vorster statistics to back up the American argument. The twinkle in Mondale's eye brought no change to Vorster's features.

Mondale carried the brunt of the argument himself, although he was flanked by six State Department and National Security Council experts, who handled specific questions.

"If I don't know what I'm talking about, pass me a note," Mondale reportedly told the staff before the talks began. "If I go on and still don't know what I'm talking about, then just feel sorry for your country. I have to ro this one."

State Department specialists who briefed Mondale for the talks were impressed by his relentless boning up for the confrontation and his determination not to get bogged down in a South African negotiaing style that emphasizes hair-splitting, semantic traps and going on the offensive.

Vorster hit him early in the meeting with America's record on Indians. Acknowledging that deplorable chapter in history, Mondale quickly went on to say that American efforts to shut Indians and blacks out of the mainstream had not worked and similar policies would not work for the South Africans, either.

The Vice President also appeared to succeed in a task he set for himself early in the meeting - not wanting to appear to be insulting or berating Vorster, despite the tough words he had to deliver.

Vorster bolstered this impression by telling reporters in private chats after his press conference that he still hoped to be able to persuade the Carter administration in time not to be as hostile to apartheid as Mondale had been in Vienna.

"I hoped sound judgment will prevail," Vorster said.

Administration sources also indicated that the first abrasive contact may be followed by a lowering of the heat, especially if Vorster does demonstrate a willingness to move on Namibia and Rhodesia.