Secretary of State Cyrus R. Vance tried yesterday to persuade Rhodesian guerilla leaders to get together with rival black factions on a compromise plan for bringing black majority rule to their African country.

However, after a two-hour meeting at the State Department, the two heads of the Patriotic Front -- Joshua Nkomo and Robert Mugabe -- reminded firm in their rejection of the "internal settlement" signed March 3 by three black nationalist leaders and the white minority government of Prime Minister Ian Smith.

Nkomo and Mugabe, who head guerilla forces fighting the Smith regime from bases outside Rhodesia, have denounced the internal settlement as a sham designed to perpetuate white rule.

President Carter said Thursday he doubted there could be an enduring settlement unless all Rhodesian black leaders, including Nkomo and Mugabe, participated. He called for the leaders of the fewding black factions to confer and try to work out a compromise betwen the internal settlement and other plans to shift power from the 263,000 whites to Rhodesia's 6.7 million blacks.

Yesterday, though, Mugabe and Nkomo reiterated that they consider "the only basis for negotiation" a U.S.-British plan that differs substantially from the internal settlement.

Although they did not rule out the possibility of talks with the nationalist leaders who joined with Smith, they insisted it would have to be on the basis of the Anglo-American plan rather than the internal settlement.

Richard M. Moose Jr., assistant secretary of state for African affairs, said afterward, "It was a discussion that had ot take place. We needed to explain to them what we want to do and hear their reaction.

"They are concerned that we're trying to press them to join the internal settlement," Moose added. "We explained that we're simply trying to get all the parties together to find a formula involving all of them."

However, Moose refused to speculate on whether the meeting had eased the concerns of the Patriotic Front leaders. He said: "We've each explained our positions. Now the two sides are going to have to think it over."