Gerald P. Carmen, chief of the General Services Administration, yesterday labeled as a "nonstory" the fact that he has twice received deferrals in repaying a low-interest Small Business Administration loan, and insisted that his performance in Washington should be judged by how well the GSA operates.

"I was surprised such a nonstory became a story," he told reporters at a breakfast meeting. "You fellows could follow black limousines around town and get a better story."

Carmen spent much of the hour-long session maintaining that he has made substantial management improvements at GSA and complaining that the press makes it difficult to attract good people to Washington because it reports on private affairs of public officials.

Since he took over at GSA, he said, the agency has survived a 16 percent budget cut while improving its performance; has reduced the time it takes to complete a "leasing action" from 366 days to 220, and has trimmed its personnel roster by 6,000 people, "only 160 of them involuntarily."

He defended the transfers of 17 top mid-management officials to jobs out of town. "Any administration is entitled to have people in key slots to carry out the programs you campaigned on," he said at one point. Later, when asked if there was a political motive for some of the transfers, he said, "I'd like to refute the idea we did it for anything other than good management reasons . . . . If I'm managing a $6 billion business and make the decision somebody should be transferred, I think that's all there is to it."

He also proved that he had mastered Washington jargon: at the GSA, he said, "Curves are better, tracking is better, time frames are better."

Carmen's public affairs office has peddled stories on good management practices at GSA actively, emphasizing the strength of Carmen's leadership. But no matter how much Carmen wanted to talk about the GSA yesterday, reporters insisted on asking about his SBA loan.

Articles Tuesday in The Washington Post and The Baltimore Sun reported details of the deferrals Carmen has received on the $405,000 he owes the SBA for property in Manchester, N.H. That figure is the total principal and interest remaining on a 6.5 percent loan he received in January, 1975, SBA officials said.

The Sun also reported that Carmen has sold a second parcel of Manchester property for $177,000 to the local housing authority for a federally funded urban renewal project. He was chairman of that housing authority in 1970 and again in 1973, his last year as a member, and the sale was described yesterday by GSA spokesman Herbert Koster as a "taking, under eminent domain."

Carmen also said yesterday that he is considering selling the property on which he has received the loan payment deferrals. "Yes," he said. "I'll have more to say on that next week . . . in the near future."

John Moffitt, special assistant to the SBA's regional administrator in Boston, said that sale of the property was listed as the reason the second deferral was sought."Obviously if he is going to sell the property personally, it is far better for the government than if he was going to liquidate it," Moffitt said.

The most recent deferral, for six months, expires June 30. Carmen's next payment would be due July 24 unless a new deferral were approved, and no deferral has been sought, Moffitt said.

"I applied for deferrals because they were available to any citizen," Carmen said. He was asked if the loan deferrals did not contribute to an impression that the Reagan administration is of and for the rich.

"I am troubled by the system," he said. "The major thing that controls this government is the press . . . . We've got problems--the press, voters, others--we're going to have to figure out some way not to put our personal lives in the position we're in now . . . . "

When he came to town, he said, the GSA was "perceived as the worst in government. Test and see what kind of job I've done . . . . The things we're trying to do, the things we believe in, are right for this country. People voted for it."

Carmen was a top GOP organizer in New Hampshire during the Republican primary in 1980, and is credited with having provided significant help to President Reagan.