The Reagan administration today will ask Congress to cut the budget of the National Endowment for the Arts and the National Endowment for the Humanities by 11.7 and 10 percent respectively, put the Institute of Museum Services out of business and increase the budget of the Smithsonian Institution by 6.5 percent. The federal allowance for the John F. Kennedy Center for the Performing Arts would remain at its present level.

Congressional arts patrons, in a departure from the usual script, yesterday reacted by predicting that, for the first time since President Reagan took office, government support for the arts and humanities in fiscal 1986 may have to be trimmed to help reduce the federal deficit.

"You can't just consider the arts in a vacuum," said Rep. Sidney Yates (D-Ill.), chairman of the House subcommittee that appropriates the endowments' funds. "If all the social programs are cut, the arts should be cut as well. And if the social programs are cut, the military should be cut, too."

With the exception of 1981, when the Reagan administration sought to cut the federal arts budget by almost 50 percent, the annual battle between the White House and Congress over endowment money has had all the drama of a minuet. Each year the president has asked for cuts and Congress has not only denied that request but actually increased budgets. This year may be different, however.

"The deficit seems a more overwhelming problem than it has in the past," said Sen. Claiborne Pell (D-R.I.), a longtime supporter of the arts and a sponsor of the legislation that created the endowments 20 years ago. Pell said Congress might agree to a compromise to maintain current funding levels, with adjustments for inflation.

The Reagan budget would reduce the budget of the National Endowment for the Arts to $144.5 million, a figure half a million dollars higher than the White House proposal for the endowment last year, but significantly less than the $163.7 million appropriated by Congress this fiscal year.

The arts endowment is the largest supporter of the arts in the country, awarding thousands of grants each year to individual artists, performing arts groups and cultural institutions. The reduction would affect the endowment's programs to varying degrees. The programs for opera and musical theater reportedly would sustain the largest cuts.

Opera and musical theater performers and lobbying groups have said they will fight both the cuts and their allocation.

Under the administration's proposal, the budget of the humanities endowment would be cut from $139.5 million to $126 million. The humanities cuts would be administered across the board for the most part. The humanities endowment awards grants for scholarly research and education in fields such as history, literature and philosophy.

Only the Smithsonian Institution came out ahead. The administration proposal increases the Smithsonian's budget by $15 million, from $230.7 million to $245.7 million. Smithsonian Secretary Robert McC. Adams said that the increase was the amount necessary to operate and staff Smithsonian projects already under construction. Those projects include the Museum of African Art and the Museum of Near Eastern and Asian Art, both being built on the Mall. "I don't think it will be found, upon close inspection, that we have been singled out for any special treatment," Adams said.

The budget would keep federal funding for the Kennedy Center at its current level of $4.5 million.

The Institute of Museum Services was not so lucky. Under the president's plan, its budget will be cut by more than 98 percent, from $20 million to $292,000, a cut designed to phase out the eight-year-old agency. The institute awards most of its budget to many of the nation's museums to help pay for such operating expenses as heat and lighting.

Both Pell and Yates said they would fight administration proposals that would close the institute. "That doesn't make sense to me at all," Yates said.

"It's a very small agency that does a very useful job," Pell said. "Not for just the National Gallery and the Metropolitan Museum of Art, but for little museums all over the land."

Pell said public awareness of the arts as well as support for federal funding has increased dramatically since the endowments were created. This year, he said, "I would hope that across the country those who love the arts will come forward. We in Congress are there to reflect the national interest."

Endowment officials have declined to comment on the figures until the budget is presented to Congress today. Appropriations subcommittees in both the Senate and the House will then begin preparing for budget hearings that begin in April.

Although arts lobbyists and patrons have already begun their protests, not everyone finds the cuts unacceptable. "Will the arts survive if there are 11.7 percent cuts? Yes, there's not the slightest doubt," said music critic and publisher Samuel Lipman, a Reagan appointee to the NEA's advisory council.

"We've had a dreadful period where we've gone to pure advocacy and every discussion comes down to one word -- 'more,' " said Lipman, who was in Washington this weekend at the advisory council's quarterly meeting. "What you wind up with are entitlement programs, as in you're entitled to what you got last year and more. It's understandable for bureaucratic reasons, but it's not the way you budget."