The Whitney Museum of American Art in New York has agreed to relinquish a collection of works by Morgan Russell as part of the settlement of a lawsuit brought by the donor. The paintings and papers will be moved next week to the Montclair (N.J.) Art Museum.

Russell (1886-1953), an American, was a founder of the pre-World War I Synchromism art movement, a colorful and textured abstract style. The collection, valued in the hundreds of thousands of dollars, is important in the history of the 20th-century abstract art movement.

The dispute began after Henry Reed, a Montclair insurance agent and art collector, donated his Russell collection to the Whitney in 1978. Last February he sued for its return, charging that the Whitney had breached the gift's contract because it had not published a catalogue or exhibited the works.

"I am prevented by the terms of the agreement from saying anything other than a settlement has been reached, satisfactory to both sides," said Martin Bressler, Reed's attorney, who confirmed that the transfer will take place next week.

Robert J. Koenig, director of the Montclair museum, said he was very "happy," and added, "We plan an exhibition to open in 1988."

The controversy is significant because it involves the control donors have after a gift has been made. Over the years, donors have protested that museums have accepted artwork only to keep them sequestered in dark basements, away from the light of critical and public attention. In some cases donors have complained because museums have sold acquisitions and have used the money for building maintenance and to buy other artwork.

Reed said yesterday he was pleased because the Montclair museum would do what he had expected the Whitney to do: "conduct in-depth research toward exhibiting the collection and publishing a scholarly catalogue in 1988."

The collection includes six oils on canvas, 50 or so watercolors and abstract studies and about 200 drawings. "The papers are a real revelation," Reed said. "His notebooks from 1911 to 1913 include notes on theory and composition and color studies. The correspondence is from leading figures in the art world, including Leo Stein, the art collector and Gertrude's brother ; painter Robert Henri; Russian/French poet Blaise Cendrars; and French critic and poet Apollinaire. Stein said of one Russell drawing, 'This is the birth certificate of American Synchromism.' "

Russell exhibited with the cofounder of Synchromism, Stanton Macdonald-Wright, in Munich, Paris and New York. After the first World War, he went back to painting in a more realistic style and, after his conversion, began using Catholic motifs. He died in 1953.

The Whitney's lawyer, Raymond L. Vandenberg, was out of town yesterday, his office said, and could not be reached. The office of Thomas Armstrong III, the Whitney's director, said he was in a meeting and would not be able to return a call.