Plans for saving the herd of fast-multiplying deer on the Smithsonian Institution's game preserve in Virginia were set back yesterday when a House subcommittee rejected a $650,000 proposal advanced to keep the deer from death at the hands of hunters.

Although the controversial hunt scheduled by the Smithsonian to trim the herd, considered a danger to more exotic animals at the preserve, has been canceled for this fall in the wake of protests by wildlife groups, the Appropriations Interior subcommittee vote casts doubt on the future of the deer.

"We'll do some thinking about it," Phillip S. Hughes, Smithsonian undersecretary, said last night. At this point, he said, "that's about all we can do."

In a Nov. 9 announcement of the cancellation of the hunt at the Blue Ridge Mountain preserve, Smithsonian officials said they planned to spend several hundred thousand dollars to relocate deer and segregate them from the more exotic species. In an experiment, at least 200 of the deer were to be resettled, wearing radio collars to see if they could survive outside.

Rep. Sidney Yates, (D-Ill.) proposed adding $450,000 for the fences and $200,000 for the study to the Smithsonian budget. An aide, who called yesterday's 5-to-2 rejection a surprise, said Yates has not yet decided whether to bring the plan up before the full committee.