White House officials are checking whether they have enough support to force a vote by the full Senate on William Bradford Reynolds' nomination as associate attorney general, which was rejected by a Senate committee last month.

Some administration officials are also considering the possibility of installing Reynolds in the Justice Department's No. 3 job as a recess appointee after the Senate adjourns Aug. 2, sources said. They said some department officials believe that if the nomination were forced to the Senate floor, and, as expected, encountered a filibuster, a recess appointment could be justified.

"If they do that, this place would shut down," said one high-level Senate official. "They'd go absolutely berserk. It would be an open declaration of war."

Senate Majority Leader Robert J. Dole (R-Kan.) is taking a formal head count of Republicans to gauge the chances of approving a "discharge petition" to bring Reynolds' nomination to the floor.

"If the president wants us to push Brad Reynolds, then we're going to help," Dole said yesterday.

Republicans and Democrats said the alternative -- a recess appointment -- would be an unprecedented affront to the Senate Judiciary Committee, which turned down the nomination after three weeks of contentious debate. Committee members said they voted against Reynolds, now head of the Justice Department's Civil Rights Division, because he repeatedly misled the panel in sworn testimony and had a lax four-year record of enforcing civil rights laws.

Under the rarely used procedure of naming a presidential appointee while Congress is in recess, experts say, Reynolds could serve as associate attorney general without confirmation during the 99th Congress, which runs through 1986.

Attorney General Edwin Meese III told reporters last week that he considers the Reynolds nomination to be "still before the Senate. No decision has been made with regard to the next step."

Reynolds did not respond to inquiries yesterday. His spokesman, John Wilson, declined comment on the possibility of a recess appointment, saying, "Some people who oppose him are floating that story."

Meese and other top department officials were at an American Bar Association conference in London and could not be reached.

White House lobbyists have been calling senators in recent days to assess the depth of support for moving Reynolds' nomination to the floor. This would require a majority of voting senators to approve a discharge petition, an unusual device to force an issue from a reluctant committee to the full Senate.

"They're not necessarily trying to raise the Titanic," a Senate official said of the administration. "They're just trying to find out how deep it's buried."

GOP officials cautioned that the Reynolds nomination could tie up the Senate for weeks, while Dole is trying to deal with the federal budget deficit and other key legislation.

"It's not a high priority," one said of the nomination. "We've known all along it would be . . . tough getting the votes. The White House would have to make a . . . convincing case that the votes were there."

A Democratic senator said recently that he told White House officials that resurrecting the Reynolds nomination is "a no-win proposition for the Republicans."

"It would be a source of embarrassment to the Republican senators to make them walk that plank," the senator said. "Either you vote against your president, or you vote for a guy who's painted as being terrible on civil rights."

But there may be some support for Dole's efforts. Judiciary Committee Chairman Strom Thurmond (R-S.C.) would support an effort to bring the nomination to the floor, according to a spokesman.

On June 27, the committee voted, 10 to 8, against the nomination, with Republicans Arlen Specter (Pa.) and Charles McC. Mathias Jr. (Md.) joining all eight Democrats in opposition. Two subsequent votes on reporting the nomination to the floor -- either with no recommendation or a negative recommendation -- failed on a 9-to-9 tie.

President Reagan and others have complained that Reynolds' critics opposed the nominee on ideological grounds and raised questions about his testimony as a way to explain their votes against him.

While the nomination has remained in dispute, the administration has not moved to fill several other vacancies in the Justice Department. Among those widely reported to be in line for top jobs are Herbert E. Ellingwood, chief of the Merit Systems Protection Board, to head the Office of Legal Policy; Charles J. Cooper, a deputy to Reynolds, to head the Office of Legal Counsel, and Carolyn B. Kuhl, now a deputy in the Civil Division, for Reynolds' civil rights post.