A South Carolina federal judge, the Reagan administration's top choice to succeed Robert H. Bork on the federal appeals court here, has asked that her name be withdrawn from consideration in an 11th-hour move that may jeopardize the administration's chance of filling the important vacancy.

U.S. District Court Judge Karen L. Henderson, named to the federal bench in South Carolina in 1986, told administration officials this week that she had changed her mind about joining the influential D.C. Circuit, according to sources familiar with the nomination.

The sources said that initially Henderson had indicated interest in the job and that the Federal Bureau of Investigation and the American Bar Assocation had launched background checks, which precede formal nomination by the president. They said she withdrew for family reasons, including her mother's health.

Sen. Strom Thurmond (R-S.C.), whose strong backing of a nominee from his state was expected to smooth Henderson's path, said in a statement that he is "sorry that Judge Henderson has withdrawn her name from consideration, but I fully understand her feelings."

Thurmond said he recommended that she succeed Bork and believed that she would make an "outstanding judge" on the appeals court. "However, Judge Henderson told me of her desire to remain as a district court judge in South Carolina and remain with her family," he said. "I realize family concerns should be heavily weighed in this decision."

Henderson's withdrawal comes nearly two months after Bork announced that he was stepping down to speak publicly about the process in which the Senate rejected his nomination to the Supreme Court. His resignation was effective Feb. 5.

At the time of Bork's announcement, administration officials said they needed to move quickly if they were to fill the vacancy. They promised a swift nomination to the 12-member court, now split 6 to 5 in favor of conservatives.

However, officials conceded in recent weeks their difficulty finding a nominee ideologically compatible with the administration and able to win Senate confirmation.

Among other issues, some Democrats on the Senate Judiciary Committee have complained about the administration's failure to nominate more minorities and women to the federal bench, particularly in the District.

In addition, the D.C. Bar has protested the fact that the Reagan administration and its predecessors have named few local lawyers to the appeals court.

Given those constraints, the Justice Department has focused primarily on women as possible replacements for Bork. Among those who had been under consideration are Kimba Wood, a New York lawyer whose nomination to a federal trial court there is before the Judiciary Committee; Pamela Rymer, a federal trial judge in Los Angeles, and District lawyers Judith Richards Hope and Carla Hills.

Staff writer Al Kamen contributed to this report.