The Reagan administration plans to cut the budget of the Institute of Museum Services by 98 percent, congressional sources said yesterday. The $292,000 budget proposal for fiscal 1986 would effectively put the agency out of business.

The budget request marks the third time the Reagan administration has attempted to phase out the agency, which was created in 1977 to help the nation's museums -- roughly 5,000 of them -- pay for operating expenses such as heat and lighting. The institute's current budget is $20 million.

Around the country, the reaction from museum patrons was swift and predictable.

"I think it's extremely unfair," said Joan Mondale, wife of the former vice president and an arts patron and enthusiastic supporter of the agency. "Museums are public institutions, repositories of our culture. They depend on the government because the private sector has not been able to keep pace." The agency last year made more than 700 grants to museums around the country.

"It doesn't surprise me," said Lawrence L. Reger, director of the American Association of Museums, a Washington trade group that represents nearly 2,000 museums. "We're absolutely convinced that Congress will reject that out of hand."

Reger and others said the agency's grants are important for reasons that go far beyond the relatively small percentage they constitute of museum budgets.

"What it does is help the museum in the community. It can go out and say, 'We've got this much, now you do your share.' That's the hardest kind of support in the world to raise," Reger said. "It's relatively easy to raise money for new acquisitions. But who wants to pay for the heat and light and toilet paper? This federal program has turned around the ability of museums to raise that kind of money."

The full budget, which the administration will send to Congress on Monday, will also call for cuts in the budgets of the federal endowments for the arts and humanities of approximately 11.7 percent and 20 percent, respectively, and in the case of the arts endowment actually represent slight increases over earlier Reagan proposals for the current year.

The Office of Management and Budget declined to comment on the agency budget request until it has been released to Congress. Institute director Susan Phillips also would not comment on the figures. Phillips was appointed by Reagan two years ago to replace Lilla Tower, who in 1982 had raised the ire of Democratic arts patrons in the House and Senate by rejecting more than 60 grant applications and refusing to reconsider them because they were improperly filled out. The rejected institutions included the Folger Library, the Adler Planetarium in Chicago and the Denver Museum of Natural History.

In 1982 and 1983 the Reagan administration proposed eliminating the agency, but each year congressional arts patrons instead increased its budget, doubling it in four years. During this period the administration thinned the agency's advisory council by not appointing new members when some old terms expired.

"They've been trying to put us out of business one way or another for a few years now," said George Seybolt, former chairman of the agency's advisory council.

Yesterday, some arts administrators and lobbyists predicted that Congress again would come through to save the agency, although it is widely believed that this year's congressional preoccupation with deficits may mean less largess than in the past.

"I'm very worried about the action but I'm very confident that we'll be able to turn it around," Reger said. "We want to be treated equitably. If other agencies are going to be cut, we want to participate, but we want to be treated equally. They should have learned their lesson from last time. We've never tried to say anything more than we want to be treated equitably."