A federal judge yesterday told the State of Maryland and the Environmental Protection Agency to try to reach a compromise that would permit Montgomery County to build a regional sewage treatment plant at Dickerson.

In negotiations that began immediately after a meeting with District Court Judge John Lewis Smith Jr., the opposing sides began looking for a way to make Dickerson what Montgomery says should be the "cornerstone" of region's solution for future sewage treatment.

The compromise, some Maryland officials said, is likely to involve Montgomery's acceptance of a plant smaller than the 60-million-gallon capacity one the county has been insisting on building up to now.

EPA attorney John Varnum, defending the agency decision last year not to provide federal money to build a 60-million-gallon sewage plant at Dickerson, said in court yesterday that the biggest plant Montgomery could justify is one handling 42.5 million gallons.

While the lawyers for both side began their negotiating after the court hearing, James B. Coulter, Maryland's secretary for natural resources, said in an interview, "If they (EPA) recommend a 42.5-MGD plant with a continuing planning process under which it could be expanded, I'd tell Montgomery to accept it."

A smaller plant would also tend to meet one of EPA's other objections - that the originally proposed plant at Dickerson was too costly. A full-sized plant there could cost more than $400 million. EPA funds would cover 75 per cent of the cost.

Maryland went to court to try to force the EPA to continue evaluating a full-size plant at Dickerson instead of rejecting it - as the agency did in August, 1976 - before an environmental impact statement had been completed.

But in reserving judgement yesterday and asking the two sides to sit down and talk. Judge Smith indicated he doesn't want to impose a solution except as a last resort.

Montgomery, Prince George's County and the District of Columbia - a recent ally of suburban Maryland on the need for the sewage plant - have insisted that all efforts to deal with regional sewage problems hinge on the building of the Dickerson plant.

EPA has not flatly disputed that strategy, but has urgued that a 60-MGD plan is bigger and more expensive than necessary. A memo from former EPA regional administrator Daniel J. Synder III to then EPA administrator Russell E. Train on April 5, 1976, said:

". . . EPA would be in an impossible position with both OMB (the Office of Management and Budget) and GAO (the General Accounting Office) if we were to go ahead with Dickerson."

Montgomery County and Maryland, on the other hand, have argued that the EPA insisted on all of Dickerson's expensive features in discussions on the plant's design.

Dickerson's cost could be reduced not only by trimming the size of the plant but by lowering the quality of proposed treatment. Because effluent from the plant would enter the Potomac River above suburban Maryland's and the District's waer intake pipes, the EPA stipulated that the facility would have to remove nitrogen from sewage. This is the most expensive part of advanced treatment.