In an unaccustomed burst of talkativeness flowing from an experiment in press policy, the State Department yesterday criticized Rhodesian Prime Minister Ian Smith's position on racial negotiations and condemned violations of human rights in Czechoslovakia.

Without being asked, State Department spokesman Frederick Z. Brown also sought to clarify the U.S. position on talks with Vietnam aimed at establishment of diplomatic relations.

The usual practice until yesterday was for State Department policymaking officials and press officers to prepare official comments on diplomatic developments - but to make these positions public only if asked by correspondents.

This often made the daily midday news briefing a quessing game in which reporters tried to name topics for which answers had been prepared and left the State Department tongue-tied on many subjects if nobody thought to ask about them.

The experiment in volunteering views flowed in part from a dinner discussion Tuesday between Secretary of State Cyrus R. Vance, incoming public affairs chief Hodding Carter and a group of U.S. reporters who regularly cover the department.

Vance, who had previously pledged to hold press conferences at least once a month, was urged to experiment with other ways to accomplish his declared goal of "openness" in foreign policy.

Brown's volunteered statements of U.S. policy at yesterday's midday press session were brief and carefully limited, and he was unable to answer many of the followup questions flowing from them. But reporters received more information than on most days.

The statement on Rhodesia expressed continuing U.S. support for the British plan for a peaceful racial transition, which was rejected early this week by Rohodesia's Smith. He said he will continue seeking a peaceful settlement by means of an "internal solution" which bypasses black liberation groups involved in guerrilla activities.

"In our considered view the socalled 'internal solution' will not produce a peaceful settlement and therefore does not have the support of the United States," the spokesman said Brown would not say whether the United States is prepared to resume a direct role in the negotiations - in the opinion of many the only way they can succeed.

President Carter Conferred last night on the Rhodesian negotiations and other matters with British Ambassador Sir Peter Ramsbotham, but the White House gave no details.

The State Department criticism of Czechoslovakia, handed out in written form, concerned the detention and harassment of Czechs who signed a recent manifesto demanding more political freedoms.

The document, sent to several newspapers in the West, said Czech citizens are deprived of rights supposedly guaranteed by that nation's laws and by the 1975 Helsinki agreement.

"All signatories of the Helsinki Final Act are pledged to promote, respect and observe human rights and fundamental freedoms for all. We must strongly deplore the violation of such rights and freedoms wherever they occur," said the State Department.

Brown could not say whether the unusual statement is likely to be followed by greater outspokenness about human rights in other areas. Carter called for the United States to speak up clearly on human rights, as part of his presidential campaign.

In another volunteered statement. Brown declared that "The U.S. does have the intention to move forthrightly toward normalization of relations with Vietnam," but said that a number of questions have to be cleared up, including further information on the U.S. missing in action in the Vietnam war.

He would not specifically say whether the Carter administration intends to move forthrightly toward admission of Vietnam to the United Nations, as stated in testimony Tuesday by Rep. Andrew Young (D-Ga). Young was confirmed by the Senate, 89 to 3, yesterday as U.S. ambassador to the United Nations, the first black to hold this position.

Young said in his testimony - and Brown repeated yesterday - that the United States hopes the U.N. issue will not arise until Washington has had time to improve its bilateral relations with Hanoi.