A disgruntled art collector is suing the Whitney Museum of American Art in New York City, alleging breach of contract and demanding the return of works he donated in 1978. Art experts describe the suit as the first of its kind, and some say it could affect the way the nation's art museums and donors do business.

"It could have an electrifying effect on the practice of other museums," said John Henry Merryman, a specialist in law and the visual arts at Stanford University Law School.

Henry M. Reed, a New Jersey insurance broker and art collector, filed suit in federal court in Manhattan in November charging the Whitney Museum with failing to fulfill its promise of a timely exhibit and catalogue for his collection. That collection consists of an estimated 200 paintings, notebooks and letters of American avant-garde painter Morgan Russell.

"Our position is that a contract was created," said Reed's attorney, Martin Bressler. "Henry Reed is 63 years old and he wants an exhibition."

Lawyers for the Whitney dispute Reed's claim and say his gift was made unconditionally. The museum is planning a Russell exhibition in 1987, they add, but it is under no legal obligation to show the works. "The museum's position is that Mr. Reed gave them the collection, period."

Morgan Russell (1886-1953) was one of the central figures in the development of a pre-World War I style of painting known as Synchromism, which art historians consider an important, if short-lived, antecedent to the Abstract Expressionism that has dominated American painting in this century.

Lawyers for both sides put the appraised value of the collection at several hundred thousand dollars. And each side has filed a motion asking that the case be decided without a trial.

Among the documents filed in the case are a series of letters between Reed and Whitney Museum director Thomas N. Armstrong III, according to lawyers for Reed and the Whitney. Those letters provide a rare glimpse at the dickering that goes on behind the scenes when a private collector seeks to donate his paintings or money to a major museum.

It is often in a museum's best interest to receive a proposed gift with as few restrictions as possible, experts say, while a donor often will request a catalogue, exhibition or some other assurance that his collection will not be relegated to the museum's basement. Experts differ as to how often the arrangements for such gifts are subjected to legal scrutiny or even set down on paper.

Stanford's Merryman, who has written extensively on law and the visual arts, says that the refined atmosphere of the museum world sometimes works against the best wishes of donors. "There is something ungentlemanly, something sort of crass about saying, 'I trust you, but let's get it down on paper.' That almost never works to the benefit of the donor."

Michael Botwinick, director of Washington's Corcoran Gallery of Art, disagreed yesterday, saying that while gentlemanly agreements had produced some celebrated art scandals years ago, donor agreements have of necessity become more formal in recent years.

"As the world has become more complicated, museums in general have tried to come to a much clearer understanding of what their donors want," Botwinick said. "More and more things are handled in a straightforward way."

Merryman, however, and one other New York art lawyer, who declined to be named, described the art world as permeated by an informality that often causes problems for museum, art dealer and donor alike.

Merryman said that a victory for Reed could have wide-ranging consequences. "Just drawing attention to the problem will make museum directors a little more sensitive," he said, and likely to ask themselves, " 'Wait a minute. Are we vulnerable?' "

Most experts agree that for collectors, the donation of art works that have been carefully assembled over a lifetime is an experience far more complicated and emotional than the receipt of a charitable deduction.

"Collectors are frequently very enthusiastic about the works they have accumulated," said Merryman. If Reed is like most collectors, Merryman said, he probably believes that Russell isn't adequately appreciated and that a catalogue and exhibit could help restore him "to his true place in the pantheon of artists."

In fact, Reed has been collecting Russell paintings and memorabilia for at least 15 years and has written a book on the subject. Reed has said he intends to donate the collection to another musuem once it is returned to him. Neither Reed nor Armstrong could be reached for comment.

The question in the case, lawyers say, is whether the assurances Reed received about exhibitions and catalogues were wishes or legally binding conditions.

"It's a very simple matter of what constitutes a contract," says Reed attorney Bressler. "But the fact that we are dealing with a museum, and the fact that very rarely before has the sanctified museum ground been defiled by saying, 'Hey, that's what you promised,' make the case seem more complicated. You can attribute this to the great piety of the museums."

The court records include letters in which Reed told museum officials that he wanted an exhibition by 1981 or 1982 for his collection, and that the exhibit should be accompanied by a scholarly catalogue. "Should you have any reason to believe that your people are less than enthused about this proposal as I have outlined it," Reed wrote in July 1978, "I should know that without delay so I can proceed in a different direction."

That the museum accepted the collection and issued a press release that discussed plans for an exhibition and catalogue is evidence of a contract, according to Bressler.

Museum attorney Raymond L. Vandenberg said yesterday that Armstrong's correspondence with Reed was extended as a courtesy to keep the donor informed of the museum's plans and does not constitute a contract. "People make inquiries from time to time," Vandenberg said. "Just because plans are revealed, that doesn't mean a commitment is made."