Fairfax County's Republican Party, veering to the political center after four years of right-wing leadership, yesterday ousted party chairman Benton K. Partin at a fractious county convention and replaced him with James D. Swinson, a 73-year-old former county sheriff.

Swinson, a moderate who has described himself as a "tough old goat," appealed for unity and said he would "broaden the [party] base so as to make it a viable force."

Partin, after conceding defeat, said, "The so-called moderate position is out of step with the Reagan platform and the Republican Party grass roots."

The dramatic election of Swinson came after five hours of parliamentary maneuvering at a convention that attracted more than 2,000 county Republicans to Lake Braddock Secondary School in Burke.

The vote represented the culmination of a bitter struggle between Partin's New Right conservatives, who took over the party leadership in 1982, and party moderates, including most of the county's local GOP officeholders.

The bitter Fairfax County feud between the moderate and conservative wings of the party reflects the turmoil that has divided the Virginia GOP after the resounding defeat of all three of its statewide candidates in November's elections.

The Fairfax battle escalated after the state elections, with each faction of the Fairfax GOP accusing the other of contributing to that defeat. The Fairfax County Republican Party organization is the largest in the state.

Swinson said yesterday that he will serve as party chairman for only one two-year term.

Partin, a 60-year-old retired Air Force brigadier general, conceded after a preliminary ballot to select the convention's temporary chairman. Both sides agreed that vote would be a test of strength.

Swinson won the vote with a 12 percent margin that apparently convinced Partin he had no hope of winning.

After the vote that elected state Del. Vincent F. Callahan, a Swinson supporter, as temporary chairman, Partin huddled with his lieutenants in a corridor outside the main convention hall, then strode to the platform.

As he stepped to the microphone to concede, applause rippled through the bleachers and ended with a standing ovation by the delegates.

"I concede this election to Mr. Swinson," said Partin. "I would like to meet with him and his people to chart the course of the party." Swinson, embracing supporters and doffing his red and white cap to the crowd, made his way to the podium. The two men shook hands, framed by scores of blue signs advocating a vote for Partin and red signs supporting Swinson. The delegates roared.

Nearby, a Partin delegate was dejected. "I feel hurt," said David D. Perko of Centreville District. "I really believed in that man."

Maurice Dawkins, a black minister who ran unsuccessfully for lieutenant governor last year, took the microphone and led the delegates in chants of party harmony: "We will best the Democrats! We will work together! We are all Republicans!"

But despite the show of unity, many Republicans acknowledged that the strife within the party is not likely simply to dissipate.

One reason is that Partin still has a substantial following among the county GOP's rank and file.

Nearly half the county Republican committee consists of party workers loyal to Partin, and some grass-roots groups of Hispanic and Asian American Republicans say they have a special allegiance to him.

In addition Swinson is not generally held in high regard by Partin's supporters. As county sheriff from 1964 to 1980, Swinson was the subject of frequent storms of controversy for his handling of -- and remarks about -- the county jail.

Swinson drew fire from his mostly Democratic critics in 1978 when three black inmates died while under his custody at the Fairfax County jail. He further enraged critics when he told a reporter: "Going through the door of my jail is not a ticket to eternal life."

As the convention ended and delegates filed out of the hall, Swinson crossed the floor to greet Partin: "Ben, I hope you'll keep the party apparatus in place."

Replied Partin: "You're the chairman, Jim."