The federal government has collected more than $135 million from "hard-core" loan defaulters -- most with college student loans -- by keeping their 1985 income-tax refunds in the first use of the Internal Revenue Service to recover unpaid government debts, the Office of Management and Budget said yesterday.

Of 248,000 people whose refunds were withheld, the largest amount -- $9,600 -- came from a Washington physician, according to Richard Hastings, chief of debt collection for the Department of Education.

Hastings said others whose refunds were withheld include a television sports announcer, a baseball player and a government attorney, but he said none could be identified further.

Education Secretary William J. Bennett said $116 million has been collected through tax-refund "offsets" of 217,000 people with defaulted student loans, bringing to an estimated $210 million the total recovered by the department this year.

Bennett said the $90.4 million collected last year was the largest such amount previously recovered.

OMB Director James C. Miller III, who appeared with Bennett and other officials at a news conference, said the program "shows we are deadly serious about collecting debts owed to Uncle Sam." He said the message to debtors is "you can run but not hide."

Miller said the new withholding has recovered $8 million for the Veterans Administration in education-benefit overpayments, $6 million for the Small Business Administration in disaster home loans, $5 million for the Department of Housing and Urban Development in mobile-home improvement loans and $1 million for the Agriculture Department in non-farm rural-housing loans.

The pilot program was authorized by Congress for two years as part of the 1984 Deficit Reduction Act. Next year, Miller said, the Public Health Service will join to recover education loans, the Defense Department for overpayments to former personnel, the Justice Department for criminal fines and penalties, and the Treasury for administrative overpayments.

This year's collections amount to about 10 percent of the $1.3 billion in debt referred to the IRS as the "collector of last resort," Miller said. In addition, about 50,000 people voluntarily paid $30 million in debt when notified that the IRS would keep their refunds, he said.

The federal government is owed $24 billion in nontax debt, including $4 billion in student loans.

Although critics have charged that using the IRS to collect debts might reduce voluntary compliance with tax laws, Bennett said "no evidence" supports this claim.

"This collection is something of a vindication for those nine out of 10 student-loan recipients who do repay their loans on time, in full," Bennett said.

While some defaulters owe large debts, Hastings said the largest number of the 2.5 million people with delinquent student loans dropped out of colleges or proprietary schools after borrowing relatively small amounts.