The Reagan administration yesterday surprised women's rights advocates by joining them in a Supreme Court challenge to a widely accepted system of calculating retirement benefits under which women tend to receive smaller amounts each year than men.

The Justice Department, in a brief filed at the Supreme Court, said the practice violates federal civil rights laws.

The system, used in annuity plans across the country, is based on actuarial tables showing that women live longer. The money women pay into the annuity plans during their working years thus has to last longer, the insurance industry argues.

While the industry contends that any change in approach would revolutionize their business, women have attacked it in court cases across the country and have lobbied the government vigorously to win support for their position. White House officials, the Department of Labor and the Equal Employment Opportunities Commission all reportedly became involved in the debate.

Until yesterday, the women thought they had lost their effort and had expressed concern that the administration would actually oppose them.

Marcia D. Greenberger, managing attorney with the National Womens' Law Center, said she saw yesterday's action as an effort by the administration to improve its low political standing with women.

U.S. Solicitor General Rex E. Lee, who signed the brief along with William Bradford Reynolds, chief of the Justice Department's Civil Rights Division, said it underscored earlier statements by the administration that equal rights for women should be protected not through a constitutional amendment but through enforcement of current laws. "This is one of those laws," Lee said in an interview.

The EEOC was already a party in the case supporting Diana L. Spirt, a professor at Long Island University. Spirt sued under the job discrimination provisions of the Civil Rights Act of 1964 challenging an annuity plan provided to university employes by the Teachers Insurance and Annuity Association and College Retirement Equities Fund.

Those two organizations manage pension programs at 85 percent of all private colleges and universities and more than 40 percent of all public colleges and universities involving more than 400,000 employes. They, like other annuity managers, pay reduced payments to women based on mortality rates.

Spirt won her case in the 2nd U.S. Circuit Court of Appeals. The Supreme Court, already considering the identical issue in a case from Arizona, was asked by the two organizations to review the Spirt decision. Yesterday the government also asked for a review, but sided with Spirt on the crucial issue in the case.

"Whether a woman contributes a greater amount of her compensation than a man for an equal benefit or contributes an equal amount for a lesser benefit, the use of sex-based actuarial tables in calculating periodic benefits results in the same discrimination," the brief said.

The practice "cannot be justified" by a statistical showing that women as a class are treated fairly, the government said. The Civil Rights Act "protects individuals" and it is "simply irrelevant . . . that women as a group receive total annuity benefits equal to men as a group. The violation consists of the payment to each individual woman of smaller monthly benefits than are paid to a similarly situated man."

The administration did not take a firm position on what the courts should do, though it said the order in the Spirt case prohibiting use of mortality tables for people who retire after May 1, 1980, was not "excessive."