Smithsonian Institution officials bowed to pressure from wildlife groups and Congress yesterday and canceled the National Zoo's scheduled hunt of native American white-tailed deer at its animal preserve in the Virginia Blue Ridge Mountains.

Instead, zoo officials said they plan to spend from $400,000 to $600,000 to relocate deer and install chain fences and gravel barriers at its Front Royal preserve to make three deer-proof compounds. That way brain and lung parasites carried by the deer can be kept from infecting the preserve's expensive and exotic animals, officials said.

In the next two years, Virginia game wardens will try to resettle at least 200 of the deer, outfitting them with radio collars to see if they can survive outside the park, a possibility zoo officials called "questionable" because of failed deer relocation attempts elsewhere.

"It's not a matter of backing down," said Larry Taylor, a spokesman for the Smithsonian, which oversees the zoo. "The Smithsonian has said from the beginning it was not wild about sponsoring a hunt."

Still, animal groups were ecstatic over the cancellation, which follows a three-week furor over the scheduled hunt. "This is great that sanity has prevailed," said Lewis Regenstein, vice president of the Fund for the Animals, a Washington-based organization. "All this embarrassment to the Smithsonian could have been avoided."

The planned rifle and bow-and-arrow hunt originally had been scheduled to trim the overpopulated deer herd, which numbers 1,500, because they were considered a danger to the more exotic animals, such as reindeer and antelope, who share the preserve. The zoo had allowed a hunt in 1981 that attracted virtually no public attention.

But this year when wildlife advocate Ann Cottrell Free noticed a sign advertising the hunt during a chance visit to the preserve, wildlife groups were alerted. The organizations blasted the hunt as an inhumane "slaughter" and Rep. Sidney Yates (D-Ill.), chairman of the appropriations subcommittee that oversees the Smithsonian, held a hearing last Thursday on the issue.

"A majority of the subcommittee opposed the hunt," Yates said yesterday. "We had a conference this morning with the Smithsonian and the hunt was canceled. I think their new plan is a very constructive and imaginative proposal."

Smithsonian officials were not completely confident, however, that the relocation plan would work well. "There's frequently a high mortality rate," said spokesman Taylor. "We're not at all certain this will be a long-range solution."

Yates said the 12-foot fences, which will make deer-proof compounds out of 1,500 of the preserve's 3,150 acres, will cost about $350,000. That money will be added by the subcommittee to the zoo's regular 1983 fiscal budget in a Nov. 18 session, he said. It has not yet been determined how much the resettlement costs will be and where the money will come from, he said.

Phillip S. Hughes, undersecretary of the Smithsonian, estimated the cost of the combined projects at $400,000 to $600,000. "We hope to get some help from Congress ," Hughes said.

John Grandy, a vice president of the Humane Society of the United States, said his group would be willing to help resettle the deer. "We're pleased the Smithsonian came to its senses," he said. "Now we're going to get together and figure out how to get the deer out of there."

Hughes said he hopes the resettlement will be done with the help of the Virginia Game Commission and the Virginia Polytechnic Institute and State University.

The zoo's conservation and research center contains about 350 mammals and 370 birds of endangered and exotic species. They are kept for breeding and research purposes.