The World Jewish Congress and a New Jersey congressman have accused White House communications director Patrick J. Buchanan of favoring abolition of a Justice Department office that tracks down Nazi war criminals in this country.

Rep. Robert G. Torricelli (D-N.J.) described Buchanan's criticism of the Office of Special Investigations as "absolutely outrageous" and called on President Reagan to ensure that Buchanan is not allowed to interfere with its investigations.

As a syndicated columnist before he joined the Reagan administration, Buchanan wrote two articles denouncing the OSI for accepting Soviet-supplied evidence in proceedings against accused ex-Nazis.

In addition, in a 1982 television interview with Allan Ryan, former head of the OSI, Buchanan said, "You've got a great atrocity that occurred 35, 45 years ago, okay? Why continue to invest . . . put millions of dollars into investigating that? I mean, why keep a special office to investigate Nazi war crimes? . . . Why not abolish your office?" Buchanan said he saw no "singularity" about the Holocaust that would justify maintaining a special prosecution office.

The World Jewish Congress, an umbrella organization of several Jewish organizations, said Buchanan's appointment to the White House "marks the first time that a person known to favor the termination of the OSI program has attained a senior position in the White House." In a statement, the WJC added that Buchanan's appointment meant that East European emigre groups trying to shield ex-Nazis "will have a direct line to the White House."