"The Signal of Distress," a Winslow Homer oil described by the painter, 90 years ago, as "a fine picture . . . a scene in mid-ocean . . . I expect to do well," yesterday became the second most expensive American painting ever sold at auction when it brought $1.7 million in the Manhattan salesrooms of Sotheby Park Bernet.

The previous auction record for an American picture is the $2.5 million fetched last October by the long-lost "Icebergs" of Frederic Edwin Church. Prior to that sale, the record had been held by "The Jolly Flatboatmen" of George Caleb Bingham, which sold for $980,000 in June 1978.

The 24-by-38-inch Homer oil, owned in the 1950s by Cornelius Vanderbilt Whitney, was offered for sale by an anonymous private collector. After less than three minutes of bidding, it was sold to Lawrence Fleischman, president of New York's Kennedy Galleries, who will pay an additional $170,000 buyer's commission to the auctioneers. The previous record for a Homer, set in April 1978, was $275,000.

Scholars here expressed surprise at the sale price, noting that the work, despite the painter's claim, is not one of Homer's finest.

Other works of art offered in yesterday's sale of "American 18th century, 19th century and Western paintings, drawings, watercolor and sculpture" also did better than expected. "We had estimated that the 148 lots offered would bring a total of between $2.3 and $3.3 million," said a Sotheby official. "The actual figure was $5,957,750."

"The Sleigh Ride," an 1852 canvas by the little-known William Tylee Ranney (1813-57) -- which seems to be part genre scene, part luminist landscape -- brought $680,000 (the pre-sale estimate had been $250,000 to $350,000). George Catlin's "Battle Between Sioux and Sauk and Fox (Eastern Dakota)," a varient of a painting in the National Museum of American Art, fetched $280,000, approximately twice the pre-sale estimate. "The Thirsty Trapper" (1850) by Alfred Jacob Miller sold for $170,000; "Pack Horses From Rim Rock Ranch" (1931) by Frank Tenney Johnson brought $170,000; Albert Bierstadt's "Gathering Hay, New Hampshire" (1867) sold for $105,000, and "Repairing Ships Gloucester Harbor" by the luminist painter Fitz Hugh Lane sold for $95,000, more than tripling the pre-sale estimate. "Cotopaxi," a seven-inch-high Church, expected to bring $20,000 to $25,000, sold for $90,000.