James D. Swinson managed to hold office as Fairfax County's sheriff for 16 years despite the overwhelming tendency of local voters to elect Democrats.

When he retired in 1980, the florid-faced former farm boy with a fondness for branding his opponents "knuckleheads" was something of a political legend in the county.

Now, six years after he stepped down from politics and public office, Swinson has again entered the fray. Late Tuesday, he won an apparent victory in his race to unseat Benton K. Partin as chairman of the Fairfax Republican Party.

The race, which will be settled at a convention April 12, has been marked by controversy and polemics, but it is not the first time Swinson, a 73-year-old retired Marine lieutenant colonel with a disarming belly laugh, has found himself in the middle of a fight.

After a federal court order forced the desegregation of Virginia jails in 1969, Swinson was criticized for being the last sheriff in Northern Virginia to comply. He said he had kept the races apart for their "comfort and safety."

The same year, he was criticized for running the only jail in the Washington area where juveniles who misbehaved faced solitary confinement on bread and water.

In his four terms in office, Swinson had four chief deputies, three of whom quit or were fired under what courthouse observers called "strained" circumstances.

In the last six months of 1978, three prisoners in the county jail, all of them black, died. Outraged Democratic critics called for his resignation.

Swinson further angered his detractors when, in his own defense, he told a reporter: "Going through the door of my jail is not a ticket to eternal life. People have died there before and people will die there again . . . . "

After an investigation and two lawsuits alleging wrongful death, the county sheriff's office agreed in out-of-court settlements to pay the families of two of the dead inmates.

It is this record that Party Chairman Partin has said would be an "embarrassment" to the county GOP if Swinson were elected chairman.

In the current contest for leadership in the county GOP, Swinson has portrayed himself as a moderate who would welcome participation from all ends of the Republican ideological spectrum.