he Justice Department today asked the 5th U.S. Circuit Court of Appeals for a delay in submitting its brief in the massive civil rights suit against the Texas prison system amid signs of a possible agreement between the department and the state that would diminish the government's role in the case.

The outcome of the negotiations could signal how far the Justice Department intends to go in reversing its once-activist role in prison litigation. During the Carter administration, the department aggressively pursued the Texas prisons case, one of the longest civil rights trials in history, by intervening in behalf of the plaintiffs against the state and the Texas Department of Corrections.

Last December, U.S. District Court Judge William Wayne Justice ruled that the overcrowded Texas prisons violated the constitutional rights of the inmates, and later he ordered sweeping changes in the system, the largest in the nation.

Justice's order was accompanied by a wrenching description of life inside the Texas prisons. Among his most controversial remedies was single celling for all inmates.

With the appeal briefs originally due to be mailed to the 5th Circuit in New Orleans by Monday, William Bradford Reynolds, head of the department's Civil Rights Division, and members of his staff met in Washington for the second straight day with lawyers from the Houston firm of Fulbright & Jaworski, which has been retained by the Texas attorney general's office on the prisons case.

Early this afternoon, the department telephoned the 5th Circuit to request a week's delay in submitting its brief. It was quickly granted. A department spokesman said the delay had been sought "to see if we can settle" the case. The department will file a brief regardless of the outcome of the talks.

"We still don't have anything firmly in hand," said Texas Attorney General Mark White. "This thing goes at different speeds, but I'm still hopeful. They've spent a lot of time trying to work things out, and if they can, I think it will be beneficial to both sides. As long as they're making progress, it's good to have more time."

Since President Reagan took office, Texas Gov. Bill Clements, a political friend of the president, has been seeking an accommodation that would eliminate or neutralize the Justice Department's role in the case.

He has met several times with Attorney General William French Smith on the prisons case and other litigation affecting the state. But it wasn't until this week that negotiations began in earnest.

Texas officials said it is unlikely that the government would back out of the case, as Clements had hoped, but the agreement could affect the strict remedies and timetables set out in Judge Justice's order.

White said the state was seeking recognition from the federal government "that great progress" had been made in correcting the deficiencies in the system.

The attorney for the plaintiffs protested that any agreement could undermine gains already made by the prisoners.

"I think it would be lamentable for the Justice Department to back away from this case, having come into it saying they had an interest in protecting the constitutional rights of citizens locked up in Texas," said William Turner, who is not a party to the talks.

"I'm not aware that the state is now willing to do things that it has refused to do up until this moment," he added. "And unless the state is willing to do things . . . the government is selling out."

Today's meeting was the third this week betwen the state and the Justice Department. The first was held Monday, with White in attendance. White did not attend meetings on Thursday or today, calling them "mechanical . . . in anticipation of an agreement."

"Many of the differences have been narrowed," White said. "It's not the spread that existed 60 days ago, or even a week ago."

White praised Reynolds' role in the negotiations. "He's not hoisting a white flag," White said, "but at the same time, he's not some dreamy-eyed fool who thinks prisons are supposed to be a big playground."

Already this year, the Justice Department has changed its positions on two other lawsuits involving Texas. One involved bilingual education, the other education of the children of undocumented workers.