The Justice Department sided with white firefighters in Memphis yesterday, asking the Supreme Court to overturn lower court rulings that prohibited the fire department--under court order since 1980 to hire more blacks--from laying off newly hired minorities because of a budget cutback.

In a brief filed in the Supreme Court, the Civil Rights Division argued that the firefighters hired under the affirmative-action plan were "only beneficiaries of affirmative action and not victims of employment discrimination," and therefore "have no basis claiming any seniority in addition to what they have actually accrued on the job."

The Justice Department position, which has been largely spelled out in similar cases in Boston, Chicago and New Orleans, is that only victims of personal racial discrimination have any right to relief from past discriminatory practices of the department. The department also opposes quotas as a remedy for past discrimination.

The lawsuit against the fire department started in 1977, when Carl Stotts, a black fire captain, filed a class-action lawsuit charging the department with racially discriminating in hiring and promotion.

The department never admitted wrongdoing, but in April, 1980, it entered into a court-approved settlement that set a 50 percent interim hiring goal and a 20 percent interim promotion goal for qualified minorities.

In May, 1981, however, the city announced budget cutbacks to be made on the basis of city-wide seniority. The city estimated that, as a result of the budget cuts, 52 persons in the fire department would be laid off, 61.4 percent of them recently hired minority firefighters.

The federal district court issued a preliminary injunction against the layoffs and ordered the department "not to apply the seniority policy proposed insofar as it will decrease the percentage of black" employes. That decision was upheld by the appeals court.

Yesterday's brief, signed by Solicitor General Rex E. Lee and William Bradford Reynolds, chief of the Civil Rights Division, urged the Supreme Court to overturn lower court rulings because the minorities involved in the case are "non-victims of discrimination who can receive relief only at the expense of 'bumping' incumbent white employes."