President Carter decided yesterday to stand by his embattled Rhodesia policy, which is under direct and growing challenge in Congress and which has stirred debate within the administration, government sources reported.

Meeting with Vice President Mondale and other senior advisers on foreign policy in the third White House review of Rhodesia policy within the last month, Carter reportedly decided to continue an activist diplomatic approach to southern Africa's racial conflicts and to support liberals in Congress trying to head off any immediate test on lifting sanctions against Rhodesia.

Sen. Jesse A. Helms (R-N.C.) introduced legislation yesterday that would force Carter to accept formally the results of last week's voting in Rhodesia and restore financial and trade relations with Salisbury. Capitol Hill supporters and opponents of sanctions said in interviews yesterday that the election had significantly improved Helm's chances for success.

Faced with the prospect of an embarrassing foreign policy defeat on Rhodesia while the administration is preparing to seek Senate ratification of a strategic arms limitation treaty (SALT), some of the president's foreign policy advisers have been urging him not to commit his prestige to countering the drive in Congress to lift sanctions.

These aides also have suggested that Carter reassess the administration's deep diplomatic involvement in the conflict, administration and Congressional sources said.

Rep. Stephen J. Solarz (D-N.Y.) told reporters yesterday that "a struggle for the soul of the administration" on Rhodesia policy was under way at the White House. Solarz, a key figure in the liberal efforts to keep sanctions in force, publicly appealed to the president and his aides "to fight for their policy" on Capitol Hill or face the near certain prospect of a major diplomatic setback in black Africa.

Solarz's appeal evidently brought quick action from the administration to combat an anticipated move by conservatives in the House to seek a vote yesterday on lifting sanctions.

Richard Moose, assistant secretary of state for African affairs, spent much of the day on Capitol Hill, and Secretary of State Cyrus R. Vance reportedly made several telephone calls concerning Rhodesia to key members of Congress.

Under compromise legislation adopted last year, Carter must lift the sanctions if he determines that free and fair elections have been held and the new government has made a good faith effort to negotiate an end to the guerrilla war.

The central argument voiced yesterday in the administrator's intensified lobbying effort in the House was that at a minimum Congress should give Carter time to make the determination required by last years' legislation. A new government is expected to be installed in Rhodesia by June 1.

The administration is reiterating privately and publicly that Carter will not prejudge the sanctions issue before receiving full reports. But the new involvmen t by senior officials in the lobbying was reportedly seen by some of those involved in the discussions as presaging an active administration battle to keep sanctions in force.

Helm's bill would immediately overturn the executive orders under which sanctions are enforced against the government of Prime Minister Ian Smith.

"The most free and open election in the history of the continent of Africa has just been concluded," Helms said of the reported 63 percent turnout in the voting organized by the Smith government. "All the conditions have been met" for lifting sanctions and recognizing the government produced by the elections, Helms asserted.

Acknowledging that the bill was likely to be bottlede up in the Senate Foreign Relations Committee, Helms aides said the senator probably would wait two weeks to give Carter time to consider lifting sanctions and then would attach his proposal as an amendment to legislation supported by the administration.

The Helms proposal followed a similar move Monday by Sen. Richard S. Schweiker (R-Pa.), who said yesterday that the antisanction forces would work out a common strategy and win in the Senate even if Carter personally entered the battle. "The momentum is on our side." he said.

Solarz, who was in Rhodesia during the election of the multiracial parliament and who conferred with the leaders of neighboring African states who support the Patriotic Front guerrilla forces, challenged Helm's assertion that the elections had been conducted freely and fairly.

More than 90 percent of the countryside is under martial law, the Patriotic Front political parties were outlawer from taking part in the campaign and the constitution that established the elections was approved by whites only, he said.

A U.S. to life sanctions would cause any hope for a negotiated settlement "to go down the drain," Solarz said, adding that the leaders of Tanzania, Botswana, Zambia and Mozambique would welcome a new Anglo-American diplomatic initiative.