Paperback rights to "The Brethren" -- the recently published investigation of the inner workings of the Supreme Court by Bob Woodward and Scott Armstrong -- were sold at auction yesterday to Avon Paperbacks for a reported $880,000.

The auction was held yesterday in New York by the book's hardcover publisher, Simon and Schuster, and took eight hours of phone calling to complete. Bidding started at $750,000 and involved at least three paperback houses according to one source involved in the sale.

"The Brethren," which details the politicking among Supreme Court Justices, had risen to the number one non-fiction spot on the local best seller list after one week, and is expected to do well on the national list.

"I think it's an important book and will last a long time," said Walter Meade, publisher of Avon, a mass-market house which has bought the reprint rights to such books as the phenomenally successful novel, "The Thornbirds" and "The Final Days" by Woodward and Carl Bernstein, which sold for $1.5 million.

"There's something about the Supreme Court that touches everyone's lives," said Meade. "No one alive has forgotten about the integration of schools order and/or the confusion of the abortion decision."

Meade said the paperback is scheduled to come out in January of 1981, but may be produced earlier.

The auction reportedly was calm compared to the frenzied bidding which often characterizes paperback sales of hardcover best sellers.

"It's not really a mass-market book," was the explanation of one representative from a paperback house which did not bid on "The Brethren."

"I don't think the entire paperback industry was eager to take this on," said Meade. "Some of the houses have had a tough time this year. But we didn't have to worry."

The record high bid for paperback rights to a non-fiction book was made a little over a year ago when Fawcett paid some $2 million to reprint Linda Goodman's "Love Signs."