The federal judge overseeing the Prince George's County school desegregation case has given both sides a deadline of Aug. 29 to agree on how the school system should go about eliminating what the judge called vestiges of segregation.

Both sides--the school system and the NAACP--still disagree on significant points, according to correspondence they have filed in court within the last month.

If the two sides cannot reach an agreement by the end of the month, Judge Frank A. Kaufman said, he will issue an order of his own telling the school system what to do.

Last June, Kaufman, who presided over the first desegregation suit filed against the county in 1972, ruled that some segregation still exists. He then recommended seven steps to remedy the situation, including having the court again take jurisdiction over matters of desegregation. The court had relinquished control over the school system in 1975 after busing to achieve racial balance began.

Kaufman also said the school system should be held to guidelines in which no school would be less than 10 percent or more than 80 percent black. The judge said the travel time for a bused student should be no more than 35 minutes one way.

In July, Joseph M. Hassett and John C. Keeney Jr., attorneys for the NAACP, wrote Kaufman proposing that black composition of all schools be within 20 percentage points of the average percentage of black students countywide. That average is currently 53 percent.

If the NAACP guideline was adopted, the student enrollment in each school could be no less than 33 percent or no more than 73 percent black.

The attorneys also said the NAACP would oppose "any inflexible 35-minute one-way transportation time" and wants that guideline to be flexible enough to "ensure stable desegregation."

In a letter filed with the court yesterday, George D. Solter, attorney for the school system, said school officials would oppose any measures that were not in Kaufman's original set of recommendations, including the more stringent guidelines for racial balance proposed by the NAACP.

Representatives of the two sides declined to say yesterday how far along they are in the negotiations over the final order or whether they think they will come to an agreement by the judge's deadline.

"There are still differences. We just don't know" if an agreement can be reached, Keeney said.

"If discussions on an agreement are going to continue, they should be centered on what the judge has already said and not on any new areas that the plaintiffs are trying to create," said Brian J. Porter, spokesman for the county schools.

Both sides, however, said they agree with the judge that school assignment changes made since 1975 should not be automatically reversed "unless there are current affirmative reasons for doing so." The judge said such an action might disrupt the stability of some schools.

But the NAACP lawyers told the judge they also think the school system should be required to determine whether there is reason enough to change school assignments in some areas "to bring the virtually one-race schools closer to the . . . guideline."

The NAACP also wants the school system to be required to adopt a "racial impact statement" whenever decisions are made concerning school closings, busing and any reorganization of a school's educational program.