The Carter administration yesterday invited Prime Minister Ian Smith and his three black coleaders in Rhodesia's transitional government to new talks here Friday on the possibility of a peace conference with their guerrilla opponents.

In announcing the invitation, the State Department said it was prompted by Smith's statements last week that he and his three colleagues are willing to attend a conference of all the parties in the Rhodesian dispute "with no preconditions."

A conference bringing together both the leaders of the Smith government and the Patriotic Front guerrillas fighting it from bases outside Rhodesia is an essential element of the joint U.S. British plan for ending the strife in the breakaway British colony and moving it to majority black rule.

So far, Smith and the front's leaders Joshua Nkomo and Robert Mugabe, have been refusing to deal with each other. In the meantime, Smith as been moving ahead with his so-called "internal settlement" plan that is supposed to produce elections for majority rule in Rhodesia by the end of this year.

Smith, who is touring the United States in an effort to gain U.S. support for his plan, balked at the idea of an all-parties conference as recently as last Monday when he met with Secretary of State Cyrus R. Vance.

His stance heightened the tension between his government and the Carter administration and prompted President Carter to publicly reject Smith's bid for a meeting at the White House.

As the week went on, though, Smith shifted position. On Thursday, he told members of the Senate Foreign Relations Committee he is prepared to attend a conference "without preconditons." Then, in a Saturday speech in San Diego, he said this position was shared by his three colleagues: Bishop Abel Muzorewa, the Rev. Ndabaningi Sithole and Chief Jeremiah Chirau.

As a result, the State Department said yesterday, Smith has accepted "in principle" an invitation to meet with "senior department officials and British representatives" for further discussion of how far his government is willing to go toward negotiating with the guerrilla leaders.

Department sources said the meeting will take place Friday morning at the State Department. They added that the U.S. participants will include Richard Moose, assistant secretary for African affairs, and Anthony Lake, chief of the policy planning staff. Both have played leading roles in framing U.S. policy toward Rhodesia.

The sources said it was unclear whether any higher-ranking department officials will take part. Both Vance and Deputy Secretary of State Warren M. Christopher will be out of the country Friday, but the sources said there was a possibility of United Nations Ambassador Andrews Young and David Newsom, under secretary for political affairs, attending.

The department's statement said "there will be discussions with the other parties as well" - a diplomatic reference to the question of whether the front leaders can be induced to attend an all-parties conference. Until now, Nkomo and Mugabe have been vowing to bring down the Smith government through armed conflict.

Last week, Mugabe's faction of the front denounced Washington for allowing Smith's U.S. visit and rejected any further mediating role for the United States in the conflict. However, whether its position is unyielding or simply a tractical, rhetorical exercise is unclear.

Mugabe, whose forces are based in Mozambique, calls himself a Marxist. Nkomo, leader of the front forces in Zambia, has accepted Soviet, and Cuban help but is considered non-Marxist and the more politically moderate of the two.

For that reason, some diplomats who have tried to deal with the Rhodesia situation believe that continued pressure for an all-parties conference could provide a means of splitting the partnership between Nkomo and Mugabe.