U.S. arms control director Eugene V. Rostow yesterday accused some Republican conservatives in the Senate of attempting to take over nuclear arms control policy by challenging key presidential appointments and endangering the Reagan administration's credibility in negotiations with the Soviet Union.

Following a White House decision Monday against renominating Robert T. Grey Jr. as deputy director of the Arms Control and Disarmament Agency (ACDA), Rostow said "the next few appointments" in the vacancy-riddled agency will demonstrate whether "we retain control in the agency and have personnel in the agency representing a continuing point of view."

In coming weeks, as the administration prepares for the resumption of nuclear arms reduction talks in Geneva, Rostow said, the agency will not be hampered by unfilled positions. For now, he said, he is optimistic that the Senate conservatives who have blocked Rostow's choices to ACDA positions "are not having any effect on the formation of policy" and that "policy is being made in accordance with the president's ideas."

Rostow said that he did not wish to state his accusation toward the Senate conservatives in "naked" terms, but added that a minority of Senate Republicans led by Sen. Jesse Helms (R-N.C.), who successfully scuttled the Grey appointment and one other major appointment last year, are seeking a level of influence in arms control policy not warranted by their numbers.

He said the threat of such disproportionate influence could promote a "fear response" from the Soviet Union "over the credibility of American guarantees or lead to extreme nationalism" to counter a greater perceived threat from the United States.

"The Soviets can try to exploit the doubts, and they get very anxious at the thought that any extremist American group might take charge of American nuclear policy," Rostow said.

He said that the difficulty of his job has been to chart a course between "people who want an agreement with the Russians at any price . . . and people opposed to having any agreement at all."

Rostow called the loss of Grey "a sad comment on the political process," but sources on Capitol Hill and in the administration said yesterday that the battle over Reagan administration foreign policy appointments is likely to continue.

While some sources had suggested that abandonment of the Grey nomination in the face of strong opposition from Helms would ease opposition to other key appointments held hostage during much of 1982, aides to several conservative senators involved in the fight indicated otherwise.

They and administration officials said it now appears certain there will be a major confrontation in the Senate later this month over the expected renomination of Richard Burt as assistant secretary of state for European affairs.

Burt is seen by several conservatives as a potential moderating influence on a hard line toward the Soviet Union in arms negotiations.

Yesterday, an aide to Helms said he wasn't aware of any deals relating to the White House's decision to abandon Grey.

"There have been any number of so-called deals announced unilaterially . . . ," he said. "But there never were any such deals, they just said, 'Here's the deal.' "

"I doubt very seriously whether Grey bought them much," said an aide to Senate Majority Leader Howard H. Baker Jr. (R-Tenn.). "The community opposing Grey and the community opposing Burt are not entirely the same."

Saying the current intention is to force the fight over the Burt nomination as soon as the Senate reconvenes, the aide added:

"At some point we've got to move on. The president and the secretary of state seem committed to the nomination, and we're going to do our best to get him through."

An aide to Sen. Malcolm Wallop (R-Wyo.) said yesterday that administration officials are "whistling in the dark" if they think opposition to Burt has abated. They have broadened the challenge to Burt to include "security" grounds stemming from an article he wrote in 1979 as a reporter for The New York Times.

That article revealed the existence of a U.S. spy satellite code-named Chalet that could be reprogrammed to monitor communications signals emanating from Soviet missile tests.

As a follow-up to a classified letter protesting the Burt appointment sent to Secretary of State George P. Shultz last summer by Sen. Barry Goldwater (R-Ariz.), Republicans on the Senate Intelligence Committee have stepped up efforts to marshal additional opposition to Burt by arranging for classified briefings to members interested in hearing a "damage assessment" report compiled by the CIA on the 1979 article's impact to U.S. national security.

One intelligence committee staffer maintained yesterday that former CIA director Stansfield Turner had recommended in a letter that the Justice Department take action against Burt for the 1979 article.

But Turner, reached in Arizona yesterday, said he did not.

"My concern would not have been with Burt, but with who told him the classified information ," Turner said, adding that he did not remember the specifics of the letter, which was classified. "But I did not in any way ask for an investigation or punishment of Burt."

In addition, Goldwater was reported to have told Intelligence Committee members that former CIA deputy director Bobby R. Inman had reservations about Burt.

But Inman, in follow-up conversations with Sens. Daniel Patrick Moynihan (D-N.Y.), Charles H. Percy (R-Ill.) and Joseph R. Biden Jr. (D-Del.), said nothing in Burt's past should impede his confirmation by the Senate, according to sources familiar with the discussions.

They said Inman felt Burt should be judged only on his performance in the State Department.

Biden said he thinks an extended Senate floor fight over Burt is "probably inevitable."

He added, "I'm not going to be able to get Burt through. Howard Baker is going to have to make the fight . . . because this is, effectively, Republicans fighting Republicans."