Virginia Gov. Charles S. Robb yesterday withdrew his proposal to build a prison in Fairfax County in the face of a virulent controversy that threatened to become a major issue in the congressional campaign of former Democratic Rep. Herbert E. Harris II.

Robb made a special trip to Northern Virginia at the request of Harris and other local Democratic leaders to assure residents a prison will not be built over their objections. At a hastily called press conference at National Airport, the governor accused Harris' opponent in next Tuesday's election, Republican Rep. Stanford E. Parris, of demagoguery in making the three-month-old prison proposal public this week.

Parris, facing a difficult reelection challenge from the man he beat two years ago, on Monday released a July 15 letter from Robb to President Reagan suggesting a 107-acre parcel in southern Fairfax as a possible site for a state prison.

Parris labeled the proposal "dangerous and dumb" and criticized Robb, Virginia's top Democrat, for failing to inform local officials about the letter.

"It didn't occur to me there would be or should be any public reaction," said Robb, who called his letter a "preliminary step to a preliminary step." Robb said the state long has recognized the need for a prison in Northern Virginia, where a large share of the state's inmates come from, but he said none would be built in a community hostile to the idea.

As two of Parris' top aides gleefully looked on, the governor told reporters the state might resubmit its request later after "full consultation" with local officials. He defended the southern Fairfax site as a reasonable one to consider for a prison, but said he wanted to "take it out of the political arena" because Republicans had made it a "bogus issue" in a close congressional race.

"I don't think the subject would have come up if it weren't the week before the election," Robb said, adding he stopped on his way from New York City to Richmond "so as not to permit this kind of demagoguery surrounding what is a very serious issue."

"It is refreshing to see the governor admit in the light of day that maybe it didn't seem like such a hot idea," said Parris' administrative assistant, Dick Leggitt, who said Parris learned of the proposal last week.

"I'm satisfied he's pulled it back for the time being, which hopefully will be enough time for us to kill it permanently."

The object of the controversy is a 107-acre portion of the Fort Belvoir military reservation at the corner of Telegraph Road and Route 1 in southern Fairfax which the federal government declared "excess property" in 1978.

A spokesman for the federal General Services Administration, which controls the land, said the State Department has expressed interest in building a facility there to train security personnel.

The GSA spokesman said the State Department has not yet decided it wants the land and will be asked to make a decision within about 30 days. Parris said State Department officials told him they no longer are interested in the parcel. If State gives up its claim, the GSA spokesman said, the government could select the local or state government with the best proposal for its use and then negotiate a fair market value for its sale. Any prison site would have to be approved by the state legislature.

Fairfax Board Chairman John F. Herrity said the state was threatening to turn southern Fairfax, which already houses the District of Columbia's prison at Lorton, into "a concentration camp" and said he was not pacified by Robb's response. "It sounds like a political shell game to me," said the Republican official, who unsuccessfully challenged Harris in 1978. "Now you see it, now you don't, maybe you'll see it again--after the election."

Herrity and Democratic Supervisor Sandra L. Duckworth, whose Mount Vernon district includes the proposed site, both said they received dozens of calls yesterday complaining about the proposal. Duckworth, who was miffed that Robb never informed her of his letter to Reagan, said she was referring all callers to his office in Richmond.

Robb said he would certainly have consulted with Herrity, Duckworth and other state and local officials before proceeding with the proposal, which he said would not materialize before 1986. He particularly faulted Parris for holding a press conference before consulting with his office, which he said was "unprofessional at the very least."

Responded Parris aide Leggitt: "We obviously got the governor's attention."