The judge presiding over the perjury trial of former White House aide Michael K. Deaver cautioned defense lawyers yesterday to trim the number of witnesses they plan to call or expect a delay in the start of testimony.

U.S. District Court Judge Thomas Penfield Jackson made the comments after prosecutor Whitney North Seymour Jr. complained that the defense had listed 183 witnesses compared with his 57.

"Most of them we don't know anything about," Seymour said at the end of the fourth day of jury selection. " . . . We're really swamped."

"He's got a point," Jackson replied, directing the lawyers to meet out of court in an effort to reduce the witness list.

If they are unable to agree, Jackson said, it would be unlikely that a jury could be seated by Monday and that testimony could begin Tuesday. Trial of the case against the former White House deputy chief of staff is expected to last three to five weeks, the judge has said.

Seymour, the independent counsel who brought the case against Deaver, said he intends to call all of the witnesses he has listed, many of them high-ranking officials in the Reagan administration.

The judge and the lawyers are expected to spend today completing their questioning of about 100 potential jurors summoned for the case. As of yesterday, 82 had been brought into court for questioning and seven of them had been excused.