Virginia state Del. Vincent F. Callahan of McLean has been ousted by his Republican colleagues as minority leader in the House of Delegates, an action that some Northern Virginia legislators said reflects the increasing tensions between rural and urban lawmakers in the General Assembly.

Callahan, the senior GOP legislator from the Washington suburbs, was replaced in the leadership post by Del. Raymond R. (Andy) Guest Jr. of Warren County, in a secret vote Tuesday night that participants said Callahan lost by one or two votes.

"It's another sign of the potential escalation of the economic tension between urban areas and rural areas and it is an issue this General Assembly is going to have to resolve or it's going to find itself running the risk of being paralyzed," said state Sen. Wiley F. Mitchell, a Republican from Alexandria.

Urban legislators last year rewrote the appropriations formula for state highway funding, an issue that angered many rural legislators and one they have vowed to revive during the 1986 session.

Del. Thomas M. Moncure Jr. from Stafford County, who was one of the leaders in Callahan's defeat, said that many of the 33 Republicans in the House wanted a more activist leader and denied that urban-rural conflicts prompted the ouster. "It wasn't us versus them," Moncure said. "We feel like we need to be more active."

Moncure said Guest, whose family operates a 2,000-acre cattle farm and has been a member of the House since 1972, will be a more active leader because he sits on the state Republican Central Committee and Callahan does not.

"I don't buy any of that," said Del. Robert E. Harris, a Fairfax Republican. "He [Callahan] works quietly. You've got to work quietly with these Democrats. You can't go in there and try to bash them. I hope that doesn't portend a barking dog mentality."

Harris and others said that Callahan's downfall may be linked to the Democratic sweep of the state's three top offices in the Nov. 5 elections. "It seems like there was a backlash of frustration over elections this year," said Harris. "I really felt it was tragic that a guy who's given unselfishly was treated that shabbily."

Callahan, who presided yesterday as chairman of a legislative task force, appeared bewildered by the action that removed him from the post he has held since 1982.

"They're getting someone just like me [in Guest]," said Callahan, a newsletter publisher and a member of the House since 1968. "Philosophically and temperamentally we're similiar. He's laid back, well respected, and very rarely speaks on the floor."

Guest has played a relatively minor role in the 100-member House. Some colleagues said yesterday he is best known for his ill-fated effort in the 1978 session to win support for parimutuel betting on race horses.

Moncure said he and other legislators began planning the ouster two weeks ago at a meeting in Charlottesville. But Callahan said yesterday that such a move "was not even remotely alluded to" during the 7 1/2-hour closed meeting. Callahan first learned of the planned coup Monday night. He said he then called Guest, who confirmed it.

"He got blindsided," said Fairfax Del. James H. Dillard II, a Republican. "I don't think he knew what was happening to him."

Dillard and other Northern Virginia legislators said they learned of Guest's candidacy when his supporters began soliciting their votes this week. All of Guest's major supporters were "rural guys," said Harris.

While Guest said he is no more likely to be "brash and argumentative" than Callahan, there may be "a difference in style. Maybe I have a lower boiling point, maybe I speak louder, from calling cows."

The latter is a reference to the Guest cattle farm near Front Royal. Although he lists himself as a farmer, Guest is a native of New York City who holds a degree in economics from Yale.

He said he always thought he'd end up in politics: His grandfather, a Manhattan lawyer, served in the administration of President Franklin D. Roosevelt, and his father was in the Virginia Senate, as a Democrat.

Guest said he hopes a weekend conference of Republican leaders in Staunton will have "a minimal amount of bloodletting and beating about the head and shoulders. Let's get it out of our system with as little acrimony as possible."

Callahan won't be there. He previously had accepted an invitation to visit Taiwan, along with legislators from other states, as a guest of that nation's government.

After Guest was elected minority leader, Callahan was nominated for assistant minority leader, the position previously held by Guest. "There was stone silence," said Harris. "Then Vince nodded yes. People figured they had had enough blood for that day."

Late yesterday, longtime friends Callahan and Guest sat down in Callahan's office on the sixth floor of the General Assembly Building, where legislative leaders are provided spacious offices overlooking downtown Richmond. Asked if he'll miss the leadership position, Callahan peered out the window toward the backside of the aging John Marshall Hotel. He smiled weakly and said: "It's not much of a view anyway."