U.S. Postal Service inspectors in eastern Tennessee have been investigating whether Rep. James H. Quillen (R-Tenn.) bought thousands of commemorative Olympic postage stamps after they were recalled March 11.

Rebecca C. Masters, Quillen's press secretary, said in a telephone interview from Tennessee yesterday that Quillen is a "long-time stamp collecter" who did buy many of the Olympic stamps in "early March." She said he had been interviewed by postal inspectors, but said he bought the stamps before he knew they had been recalled.

President Carter ordered the stamps withdrawn from circulation because of the U.S. boycott of the Moscow summer Olympic Games -- a response to the Soviet invasion of Afghanistan.

The recall was described at the time as a collectors' paradise, because the sudden scarcity of the stamps could have made those in circulation worth more than their face value. But last Wednesday the Postal Service suddenly announced that the stamps were being put back into circulation, thus reducing the value of any being hoarded by collectors.

Postmaster General William F. Bolger said in the announcement that the sale would resume as "a tribute to the fine men and women of the U.S. Olympic team who sacrificed months and years of training" because of the boycott.

Quillen's press aide said the congressman, who faces opposition in Thursday's Republican primary for the first time in years, bought four sheets of 31-cent stamps and 16 sheets of 10-cent stamps in the Johnson City Post Office in early March.

He also purchased five sheets of 31-cent stamps, 10 sheets of 10-cent stamps and 20 sheets of 15-cent stamps from "a collector" in Knoxville about the same time, the aide said.

There are 50 stamps on each sheet, putting the face value of the ones Quillen's aide said he bought at $419.50.

After the recall, Washington area dealers said, there were brisk sales of the stamps for 3 to 10 times the face value. Now that the stamps are being circulated again that price is certain to decline.

Masters said Quillen paid for the stamps with his own money, rather than office funds. Asked if the purchase of thousands of stamps wasn't unusually large, she said, "Obviously you don't know much about stamp collecting. It's not at all unusual for a collector to buy them in large volume."

She declined to identify the collector in Knoxville who sold Quillen many of the Olympic stamps. "There's an investigation going on, and we wouldn't want to do anything to impede that," she said. She added that Quillen hasn't heard anything from the U.S. attorney's office in east Tennessee, which is studying the postal inspector's report.

Richard Harris, an assistant in the U.S. attorney's office in Knoxville, refused to comment on Quillen's involvement in the case. "We don't comment about investigations down here," he said.

Quillen, 64, is a nine-term congressman from the First Congressional District in Tennessee's mountain country. He is the ranking minority member of the House Rules Committee, and was in prominent view at last month's Republican National Convention in Detroit because of his position on the podium as parliamentarian.

Quillen was a member of the House Committee on Standards of Official Conduct. He is known in his district as a wealthy man who has a prosperous insurance business.

This year, for the first time in a decade, he is being opposed in the GOP primary by N.M. (Bill) Bays, a Johnson City businessman. The campaign has been enlivened in the past week by the trial of a longtime Quillen friend who was charged with paying men to tear down Bays' campaign signs.