For years auctioneers have said that mammoth artworks will not sell on the block because buyers like to try them out at home. Last night, Sotheby's proved the rule wrong. A dangling Alexander Calder sculpture fetched $852,000, setting a record price for a work of American sculpture.

Lucy Havelock-Allan, Sotheby's contemporary art specialist, was delighted: "Just today a dealer came in and said, 'You can't sell that.' Well, I could and I did."

"Big Crinkly" (1971)--a festive red, blue, yellow, pink, black and white stabile, 24 feet high and nearly as long--brought the top price in the sale of post-World War II art and sculpture. It toppled the previous record of $715,000--bid in 1982 for Frederic Remington's "The Norther." And it contributed to a new record total for a contemporary art sale: 74 works for $5.4 million. (The previous, also at Sotheby's, was $4.6 million for 86 items in November 1983.)

Though no other major records were set, high prices were posted tonight for works by Abstract Expressionists Mark Rothko, Richard Diebenkorn and Willem de Kooning.

Only a few disappointments--primarily the Washington Color School art by Morris Louis--marred the sale. A Louis veil painting, which was expected to sell for $300,000, failed to bring the minimum. Bidding for Louis' "Wine" (1958) reached $240,000, and Andre Emerich, the dealer who handles the estate of the late Louis, thought it had sold. "Missed it by an ace," said Havelock-Allan.

Louis' "Red Go" (1962), one of his important small stripe paintings, was bought for $165,000. Another Louis striped canvas, "Color Reverberations" (1961), expected to bring $180,000 to $220,000, failed to sell at $166,000. The artist's "#147" (1962), another vertical stripe painting, brought $49,500.

"There was a general Louis hangover," said Havelock-Allan, referring to the successful Louis sale Tuesday night at Christie's that set an auction record for the painter, $462,000.

The second highest price in tonight's price game went for de Kooning's "Bolton Landing" (1957). It sold for $847,000. Phillip Block, a spokesman for the seller, the Chicago-based Inland Steel Co., explained after the sale that the firm was worried about security for the picture. It was displayed in the executive quarters, said Block, and "some nut might deface it." He also noted that steel industry revenues have been weak.

A smaller Calder stabile, "Two White Dots" (1973-74), brought $187,000.

Diebenkorn's "Ocean Park No. 46" (1971), an abstract, was sold for $330,000 to the Texas Gallery in Houston on behalf of a private Texas collector.

Four Rothkos also brought high prices, particularly some of the small works on paper. "Three Blacks in Dark Blue" (1960), an oil painting depicting pools of black floating on a dark blue background, fetched $374,000. The winning bidder was not identified.

An untitled Rothko work on paper (1950), shimmering with pink and magenta fields of color against a yellow background, fetched $104,500--nearly double its estimate, and a new record for a contemporary work on paper.

Nearly a thousand bidders crowded the main salesroom, and Sotheby's tightened security measures. Reporters were roped off from the main floor, making it difficult to determine who bought what.

Sotheby's chairman, A. Alfred Taubman, made a rare appearance on the floor of the salesroom, briefly entering the bidding for the second Calder. But he stopped short of winning it; both Calders went to unidentified buyers.

One of the evening's more exciting moments came when six 6-foot-long fluorescent strips--executed by conceptual artist Dan Flavin and on display in front of the salesroom--were lit. Against a cool white glow, a bidder's war ensued between Swiss dealer Thomas Amman and an unidentified friend of British advertising magnate Charles Saatchi.

"The Nominal Three (to William of Ockham)" (1963), which was expected to go for $10,000, sold after nearly three minutes of heated bidding for $31,900.

Everyone applauded.