Secretary of State Cyrus Vance warned Rhodesia yesterday that the United States will never help it to circumvent the formula for black majority rule that Prime Minister Ian Smith has rejected.

By spurning the American-backed British plan for continuing the Geneva conference on Rhodesia, Vance told his first news conference, Smith has produced "a new and more dangerous situation regarding the prospects for peace in Rhodesia."

"The Rhodesian authorities should understand clearly that under no circumstances can they count on any form of American assistance in their effort to prevent majority rule in Rhodesia or to enter into negotiations which exclude leaders of nationalist movements," Vance said.

Vance was specifically rejecting any American support for what he described as "the so-called internal solution" for Rhodesia, advocated by Smith. This is a plan to try to negotiate a Rhodesian settlement with more moderate blacks inside the nation to try to circumvent the stiffer terms of Africa's black leaders.

Human rights issues dominated vance's initial meeting with reporters. Questions were raised on whether the Carter administration is embarked on "a double standard" of human rights activism, with stronger demands on small nations than on large ones, notably the Soviet Union.

Vance strongly denied any attempt to apply a double standard. But he was questioned at length on the dispute resulting from the State Department's warning last Thursday against Soviet attempts "to intimidate leading Soviet dissident Andrei D. Sakharov.

President Carter on Sunday said neither he nor Vance had cleared the Sakharov statement and indicated it should have been handled differently, but he added, "We are not going to back down" on it. Critics charged that was a rollback, nevertheless, to minimize offense to the Soviet Union at a time when Vance was discussing a visit to Moscow for nuclear arms negotiations.

Vance confirmed yesterday that he will go to Moscow in March, at a date to be announced.

While reiterating Carter's "deep concern" about human rights, Vance sought to rule out any automatic public declarations in what he termed the "very complex area" of human rights.

"We will speak frankly about injustice, both at home and abroad," he said. "We do not intend, however, to be strident or polemical, but we do believe that an abiding respect for human rights is a human value of fundamental importance, that it must be nourished."

Vance said the administration will speak out "from time to time" on this subject, but he also said it would do so without being "intrusive in an improper way."

The Soviet Union has charged that the Sakharov statement was an unwarranted intrusion into its domestic affairs. Vance said yesterday, "I respect Mr. Sakharov very deeply," but he did not repeat the Thursday warning against any Soviet breach of "accepted international standards in the field of human rights.

Vance's basic, low-key approach, in contrast to the style of his predecessor, Henry A. Kissinger, was evident from the moment he entered the conference room. Unlike Kissinger, alwasy preceded by security guards, Vance unpretentiously ambled into the room a few minutes early, before most reporters realized he was there.

Vance made the following points on other major subjects:

Middle East - It is "critically important that progress be made this year in the Middle East" and his trip to the region, which begins Feb. 14, will begin to explore the groundwork for new negotiations in Geneva. Vance said there will be consultation with the cochairman of the Geneva conference, the Soviet Union. He also said he has 'some thoughts" on the 'critical" question of the legitimate interests of the Palestinian people . . ."

Cuba - Although Kissinger stressed the removal of 13,000 or more Cuban troops from Angola as a condition for normalizing U.S. relations with Cuba, Vance said, "I don't want to set any pre-conditions at this point . . ." He said that "the presence of any outside forces is not helpful to a peaceful solution" in Angola, but "I think that this is a matter that should be settled by the Africans themselves."

Nuclear agreements - He asked Brazil and Germany, and Pakistan and France, to hold off on their controversial agreements for Brazil and Pakistan to acquire advanced nuclear technology. Both projects raised American alarm about this potential for use in nuclear weapons. Vance said, "We would hope that in each of these cases ways could be found to not proceed with reprocessing plants."

Cyprus - Talks between the Cypriot president, Archbishop Makarios, and Turkish Cypriot leader Rauf Denktash last week seem "generally constructive," Vance said. Vance expects an announcement later this week of a special envoy on the Greek-Turkish-Cypriot dispute. Washington attorney Clark M. Clifford, Defense Secretary under President Johnson, has been offered the post.