Prime Minister Ian Smith and Secretary of State Cyrus R. Vance met for two hours yesterday in what both sides later agreed was an unsuccessful attempt to narrow their differences over how to move Rhodesia to emonstratic, majority black rule.

Smith, whose white minority government has been considered illegal by the United States since Rhodesia's 1965 break from Britain, left no doubt that the meeting had ended in deadlock.

When reporters asked, "Did you change the mind of the U.S. administration?" he shook his head. Then, in response to questions about what the gesture meant, he said, "Negative."

A few minutes later, the same point was made by the State Department spokesman, Hodding Carter, who said: "It was a meeting with all th bark off. Both sides expressed their points of view very clearly and forcefully. There was no progress made."

They key issue in the meeting, arranged after the Carter administration reluctantly gave Smith a visa for a U.S. visit, was the relationship between Smith's transitional government and the black Patriotic Front guerrilla forces waging war against it from bases outside Rhodesia.

The United States, in partnership with Britain, contends that any peaceful solution to what the State Department yesterday called "the rapidly deteriorating situation in Rhodesia" requires the cooperation of the Front's leaders, Joshua Nkomo and Robert Mugabe.

To that end, the United States has been calling for a peace conference of all parties to work out a power-sharing arrangement that will pave the way for elections leading to majority rule in the southern Africa country, which has a 96 percent black population.

So far though, Smith and the Patriotic Front leaders have refused to deal with each other. In the meantime, Smith, in conjuction with the three black leaders associated with him in the transitional government, has been moving ahead with his own plan that he claims is intended to produce a black majority government by the end of this year.

Smith, who contends the U.S. and British governments are biased in favor of the Patriotic Front, has said he hopes, in the course of his American visit, to swing American public opinion and the administration behind his so-called "internal settlement" plan for Rhodesia.

Following yesterday's meeting at the State Department, he and one of his black partners in the Rhodesian government, the Rev. Ndabaningi Sithole, conceded they had failed to budge Vance. Similarily, they said Vance had been unable to convince them that they should accept an all-parties conference.

Asked if he was disappointed, Smith replied: "Whenever I don't get my own way, I'm disappointed. We've got to think again. We've all got to think again."

He said there had been discussion of the idea of another meeting with Vance later in the week, when the other two black leaders of Smith's government arrive here. However, he added, since Vance's impending trip to South Africa appeared to make the scheduling impossible, he had raised the idea of a meeting with President Carter.

State Department officials confirmed Vance's willingness to have a second meeting if the scheduling can be worked out, but they said the question of a meeting with Carter would have to answered by the White House.

The department also issued a statement following the meeting stressing the joint U.S.-British view that "the only way to avoid a growing, bloody and dangerous civil war in Rhodesia is for all the parties to meet and agree on a fair and peaceful solution."

The statement also reasserted U.S. and British claims of neutrality in the dispute. It said: "The recent claims on each side that we favor the other are simply false. If we favored either one, we would hinder our ability to work for a fair process that would end the bloodshed."

The department also sought to play down a report in Sunday's editions of The Washington Post. It said the United States and Britain have dropped their insistence on elections as a precondition of western recognition of an interim biracial government in Rhodesia, provided an all-parties conference agrees to such a government.

Hodding Carter said "a number of options or approaches have been discussed" in past months. But he added, "Our objective remains the same - elections leading to transition to a democratic, majority-ruled independent country."

If the contesting parties agree on a different approach than those suggested in the past, the United States and Britain will go along, he said. But he stressed: "Approaches are not important. The goal is, and the goal remains the same."

Meanwhile, 800 to 1,000 demonstrators picketed peacefully in front of the White House yesterday to protest the Smith visit. Leaders of the demonstration promised to continue their protests throughout Smith's stay in the United States.