Already some people in New York City are wondering if the Knicks traded the wrong guard. The team seemed eager to unload Ray Williams to New Jersey because, it was generally agreed, he played out of control.

Now Michael (Sugar) Ray Richardson is directing the offense and the same cries are surfacing at Madison Square Garden after the Knicks' fifth loss in eight games.

"Sugar is killing us," one of the Knicks was quoted as saying. "He knows some of the . . . he's pulling makes no sense."

Coach Red Holzman, who has the patience of a saint, yanked his all-star guard late in the third quarter Saturday night after the Knicks fell behind Philadelphia and inserted his fourth guard, Reggie Carter, to stabilize the offense.

The Knicks replaced Williams with two veteran scoring guards, Randy Smith and Mike Newlin, but neither is a ball handler. If Richardson can't do the job, the team will never match its 50 victories of last season.

Records at this point of the season can be deceiving. The Knicks' victories have been over lowly New Jersey (twice) and dreadful Dallas; the Bullets' four losses have been to the league's three top teams: Boston (twice), Philadelphia and Milwaukee.

There is no doubt here that Richardson is one of the most talented guards in the NBA. However, if he can't blend in with Maurice Lucas and the new guards, his value will greatly diminish.

In his first two starting assignments for the Nets, Williams made a mere five of 27 field goal attempts.