In the face of mounting criticism over its failure to haul violators of environmental laws into court, the Environmental Protection Agency has replaced its top enforcement official, devised a plan to refer more cases to the Justice Department and is considering revamping its enforcement program.

William A. Sullivan Jr., who was appointed EPA enforcement counsel last June, resigned Wednesday. His resignation had been expected, since he lost out in a turf battle with Robert M. Perry, the agency's general counsel. Two weeks ago, Perry was put in charge of EPA enforcement, over Sullivan. Within days Perry stripped Sullivan of all his authority.

Sullivan's resignation coincided with a meeting in Quantico, Va., of Perry, Carol E. Dinkins, assistant attorney general for the Justice Department's Land and Natural Resources Division, and their staffs. The officials met to hammer out procedures for bringing more violators to court in an effort to deal with what one EPA attorney called "our terrible public relations problem."

Perry, a former attorney for Exxon, and his aides are also considering revamping the enforcement structure in Washington--for the third time since President Reagan took office. Critics charge that Perry wants to reorganize the office to parallel the structure of the general counsel's office so that the two can be merged and the enforcement attorneys eventually eliminated. But EPA officials deny such charges, insisting that any changes would be intended to improve enforcement.

Under Sullivan, an attorney who used to run a firm that advised governments in areas with steel mills, the EPA enforcement office has been the subject of sharp attacks by congressmen, environmentalists and former EPA staffers. The number of civil enforcement cases that EPA sent over to Justice has dropped from 252 in 1980 to 79 in 1981, a 69 percent decrease.

But one EPA source said that Sullivan was only "doing what he was told" and that he was being made "a scapegoat." A former EPA enforcement official said it is "obvious they don't want any enforcement. Sullivan didn't get the message. He was still trying to do some enforcement, although he couldn't do much because they had taken all his technical people from him. That's why he's gone." Sullivan could not be reached for comment.

EPA sources say Sullivan's assistant, Peter Broccoletti, has also been asked to leave. His operating style--which included dress codes and lectures--"is an embarrassment to EPA," according to one EPA source. Broccoletti could not be reached for comment and a Perry aide said he was not aware that Broccoletti had been asked to leave.

The plan worked out between Justice and EPA--known as the Quantico Guidelines for Enforcement Litigation--is an attempt to coordinate and streamline civil litigation procedures.

Perry and Dinkins are to meet regularly, as well as with staff here and in the regions. Case development teams will be established when a potential civil enforcement case arises. However, clear-cut violations will be referred directly to Justice.

In addition, a new streamlined referral process, still in the development stage, should increase the number of cases sent to Justice, according to the EPA.

The guidelines said that regional administrators and heads of individual program divisions--like air and water--will play "key roles" that are "still being clarified." Critics had charged that those officials were not well suited to head enforcement efforts. The guidelines also said states would be "given the opportunity and incentive to initiate enforcement cases," although the incentive was not detailed.

Dinkins said she agreed that the small number of cases referred to Justice had given the Reagan administration a bad image on enforcement, although she argued that in judging the effectiveness of enforcement, the number of cases was less important than the types of cases.

Critics seem to agree that the problem is at EPA and not Justice and, in fact, generally praise the work of Dinkins and the attorneys in her division.

Dinkins said one reason for the low referrals is "that EPA has reorganized several times." She added, however, that the latest reorganization should improve enforcement efforts.