Years ago publishers discovered they could wrap four paperbacks by the same author in cellophane and a rubber band and sell them as a gift set for $2 during the Christmas season. With time, things have become more complicated -- and expensive. The cellophane "shrink wrap" has given way to elaborate cardboard boxes featuring four-color art, which alone, according to one marketing executive, cost more than a single paperback book did 20 years ago. Boxed sets can run as high as $100 (for Scribners' four-volume Dictionary of the History of Ideas ). But among this season's new titles, Penguin's limited-edition George Eliot Gift Set , at $40, is the top of the line.

The boxed-set concept is being continually reevaluated for cost and marketability by both the publishers and the book stores. What will the public buy? And how should it be presented?

The newest wrinkle is Walden Books' "Box Your Own" campaign, to be launched near the end of this month in the chain's 675 stores. Walden has printed up 908000 blue and gold collapsible boxes of varying sizes and elasticized gold bands to encircle the book-filled box. Customers will be urged to choose from a store's entire stock of mass-market paperbacks to make up a gift package.Walden is making a few suggestions: A "historic saga" box would include The Year of the French, The Establishment and Love and Honor . A "suspense" set might range from Smiley to Travis McGee. The box is free.

According to Dave Culley, Walden's merchandise manager for mass-market paperbacks, store managers complained last year that a number of prepackaged boxed sets did not sell. Apparently there was some customer resistance to preselected material.

"Customers like to participate in the gift buying," said Culley. "This gets them involved."

Fawcett, too, is trying to find new ways to explore the gift market, according to Belle Blanchard, director of advertising, promotion and publicity. They are looking at other gift-buying times of the year to capitalize on, like Mother's Day. Last May, an Erma Bombeck Box Set ($9.75) was available with a free Mother's Day cartoon card featuring a harassed mommy, complete with squalling baby and carpet slippers. The caption, guaranteed to raise the hackles of any feminist, reads, "Happy Mother's Day to the only woman I know who had the courage to wash my gym clothes." For Father's Day, Fawcett pushed a John D. MacDonald boxed set, with card.

The trend in boxed-set marketing appears to be away from quantity. Jack Romano, Bantam's marketing director, says, "Five years ago we were overproducing.Anyone who could get together four or five books made a boxed set. We are publishing fewer now and much more selectively."

Last summer Publisher's Weekly wrote that the number of boxed sets has declined since last year, dropping from 247 sets to 227. But the number of new titles has increased from 50 in 1979 to more than 70 this year.

What Romano calls the Cadillacs have always held their own: the Tolkien trilogy, boxed sets for juveniles, sets by James Herriot. Science fiction and fantasy have also traditionally sold well.

"Popular fiction is much chancier," says Romano, "unless it's Michener."

"We view boxed sets as promotional material," says Ling Lucas, publicity manager for Berkeley Books. "They are great gift ideas which encourages readers to buy several books rather than single volumes. Portability is also an important factor in their popularity: A person can read all of Doyle's Sherlock Holmes mysteries, but not be encumbered by a huge single volume with impossibly small type."

But, Lucas warns, "Without some new packaging methods or materials . . . the cost of producing such sets may well become prohibitive."

Lucas says he has high hopes that breakthroughs will come soon.