The long struggle to clean up the violent and overcrowded Texas prison system has turned into a war of nerves between increasingly hostile combatants. Now the state has targeted a new enemy: the court-ordered master who is overseeing implementation of a federal judge's ruling.

Texas has made no secret of its contempt for the ruling of U.S. District Court Judge William Wayne Justice, who a year ago ordered sweeping changes in the prison system to correct what he said were constitutional violations caused by serious overcrowding. The ruling came after one of the longest civil rights trials in U.S. history.

The state agreed to improve medical care for prisoners and to some other changes ordered by the courts, but has steadfastly resisted other proposed remedies. It has carried its objections to the 5th U.S. Circuit Court of Appeals, and in some instances has simply dragged its feet in carrying out Justice's order.

Now, in a case that is to be heard in Houston later this month, the state is seeking to have the special master, Vincent Nathan of Ohio, dismissed, claiming that he and his staff have "repeatedly abused their offices flagrantly and egregiously" by encouraging violence in the prisons, failing to report instances of inmates carrying concealed weapons, setting up a clandestine mail system and offering special favors to prisoners who cooperate with him.

Nathan, in a rejoinder filed in federal court in Houston, denied the state's accusations, and charged that officials of the Texas Department of Corrections (TDC) have inhibited his work, tampered with mail between the inmates and the special master's staff and threatened prisoners for cooperating with his staff.

Nathan, a Texas-born attorney who lives in Toledo, is a recognized expert in the complexities of prison reform who has served as a master or consultant in Ohio, Georgia and New Mexico.

But Texas sees him as a scourge. Almost from the time of his appointment the relationship between his staff and the state has been poisonous.

At one point, Gov. Bill Clements falsely told reporters that a federal grand jury was investigating Nathan's conduct. When Nathan demanded an apology Clements responded that it was Nathan who owed the state an apology for "the furor he has created."

Among the most explosive of the state's assertions is that Nathan and his staff have contributed to an increase in prison violence by their conduct. The state's brief says that between 1975 and late 1979 prison disturbances occurred about every 19 months. This rose to one every 5 1/2 months during the long prison trial.

Since the arrival of the special master, the brief says, the rate of disturbances has increased to one every 19 days.

The state cites several instances in which disturbances in the prison followed by a few days the visit of one of Nathan's staff. In one instance, in November, 1981, inmates set fire to papers, linen, blankets, sheets, mattresses and other articles a few days after the visit of one of Nathan's monitors. The state claims the monitor told the inmates of disturbances at other prisons.

Nathan's lawyers say that in all the instances cited by the state the staff of the special master had done nothing improper and contend that the fires set in November were done with the help of the building tenders, or inmate guards.

The role of the building tenders is at the heart of much of the ongoing dispute. Justice ordered the elimination of the building-tender system after testimony showed that the building tenders intimidated other inmates and received special favors from prison officials. But the state has been reluctant to comply with the order.

Nathan's future is to be decided at a hearing at which the state's compliance with the building tender order also will be discussed. It is likely to be another flash point in the prison reform struggle here, rather than a resolution to the problems.