Camalier and Buckley, a 50-year-old jewelry and gift retail company in Washington, agreed yesterday to pay up to $180,000 in cash refunds or offer up to $1.6 million in discounts on future purchases to customers who suffered losses from a failed mail-order business operated by a separate New York business under the Camalier and Buckley name.

The unusual case centers on Camalier and Buckley Mail Order Inc., a defunct firm in Westchester County, N.J., which began a mail-order catalogue business in the 1970s under an agreement with the Washington firm to use the retailer's name.

Camalier and Buckley had been one of the first area firms to start a mail-order division in 1947, but operations had grown so complex 30 years later that the New York firm was selected to run the catalogue business under a license agreement.

The New York firm became insolvent in 1979, and some 4,000 customers throughout the country did not get either the merchandise they ordered or refunds. New York State won a judgment against the Westchester firm, but was unable to collect any funds.

Subsequently, New York Attorney General Roberts Abrams sued the Washington company, charging that the retailer was liable for acts of the mail-order business and responsible for making refunds. Camalier and Buckley has eight stores in D.C., Maryland, Virginia and Georgia.

In an agreement announced yesterday by Abrams and Camalier and Buckley President F. Davis Camalier, affected consumers have the option of receiving a cash refund of up to 72 percent of their loss or a substantial discount on merchandise offered in future Camalier and Buckley catalogues -- a mail-order operation that the Washington company now has resumed with two catalogues a year.

In a interview, Camalier said, "We've been victimized, as have these customers, and we want this case behind us."

Camalier emphasized that his company wants to turn the affected consumers into "our best customers," by offering discounts of up to 50 percent on purchases of up to $300 worth of goods from the current catalogue, discounts of one-third in the upcoming fall catalogues and a 25 percent for 1982 or 1983 catalogues.

"Mail-Order Inc. tarnished the 50-year reputation and good will established by our company," he added in a letter being sent along with notice from New York State to the customers involved.