American Jewish leaders have begun to blame a significant increase in reported anti-Semitic incidents on growing ignorance about the Holocaust. The new atmosphere, they say, is symbolized by a California group that contends that the Nazi gas chambers never existed.

For the last two years, in a small office near the fast-food shops and used-cars lots of the Los Angeles suburb of Torrance, a 35-year-old British publisher has enjoyed surprising success in compiling attacks on the "myth of the Holocaust" at his Institute for Historical Review.

The Holocaust is about as real as the emperor's new clothes," said the publisher, Lewis Brandon. He has published five issues so far of the Journal of Historical Review, carrying this message to at least 3,000 subscribers, including some libraries, and provoking outrage from Jews all over the world.

To American Jews like Mel Mermelstein, a Long Beach businessman who lost his mother and two sisters at Auschwitz, the journal is a vile insult. But to many Jewish leaders it is a dangerous sign of ignorance, at least partly responsible for the doubling of anti-Semtic incidents reported to the Anti-Defamation League of B'nai B'rith across the nation last year.

"For the first time in the United States," said Rabbi Maryin Hier, dean of the Simon Wiesenthal Center of Holocaust Studies, "a pseudo-intellectual group has arisen, who endorse the view that the Holacaust never occurred. One can never take away thier PhDs, which they have earned. Young people are very impressed, they are susceptible to the journal."

Hier said one woman told him she was not worried about the Journal now, but wondered what would happen in 20 years, "when all the Holocaust survivors, the eyewitnesses, are dead."

Mermelstein's response has been to file a $17 million lawsuit aganist the institute, including a demand that the institute pay him its offered $50,000 reward to anyone who can prove "that even one Jew was gassed in a Nazi concentration camp, as part of an extermination program."

Only one of the 11 PhDs listed as a member of journal's editorial advisory committee is a historian. He is James J. Martin, who Brandon says took his degree at the University of Michigan (in 1949, according to records) and taught American history for 25 years at several colleges.

Brandon said Martin is a "recluse" living in an unspecified mountain area and declined, through Brandon, to be interviewed. Several other PhD holders on the committee are scientists, such as Reihnard K. Buckner, a physicist at California State University at Long Beach.

Articles in the journal are written in cool, turgid prose. They espouse what Brandon calls the "revisionist" argument that millions of Jews were arrested by the Nazis and many suffered death, but through the ravages of disease, malnutrition and allied bombing and not through any planned extermination campaign.

The final solution for the Jewish question, Brandon said, was expulsion, to Russia, to Madagascar, to Palestine. He estimates that 350,000 Jews died in captivity, in contrast to the figure of 6 million deaths generally accepted by historians during the last three decades.

"I would feel that any deaths due to war are to be regretted. We are a pacifist organization," Brandon said. "But to prevent future wars, we have to tell the truth about past ones. We feel that the rate of deaths for the Jews was no greater than that for other groups."

Mermelestein, 54, who runs a business in Long Beach selling pallets to the lumber industry, presented Brandon with his claim for the $50,000 reward in the form of an affidavit of his experiences in the Auschwitz and birkenau camps in Poland in 1944, when he was 17.

"The last time I saw my mother and two sisters was when they were driven into what I later discovered to be the gas chamber," Mermelstein said. He described buildings used as gas chambers and pits filled with buring bodies.

"I've been back to Auschwitz 10 times since then, so I know the whole layout very well," Mermelstein said. He said after he wrote a strong letter to the Jerusalem Post decrying efforts to deny the truth of the Holocaust, someone at the institute wrote him and dared him to prove his claim that the gas chambers existed.

About 3 million jews are said to have died at Auschwitz alone, according to most histories of World War II. Rudolf Hoess, a supervisor at the camp, testified after the war that Zyklon B, a cyanide gas, was used to kill Jews and other camp inmates judged unsuitable for labor.

Brandon, however, contends that Hoess was testifying under duress and that the evidence for the gas chambers would not hold up in an American courtroom. The cyanide gas used only to kill insects in the inmates' clothing and the chambers found at the camp were simply places where bodies of those who died of other causes were taken, he said.

When Brandon declined to pay Mermelstein the reward and announced a special institute panel to hear evidence in November, Mermelstein sued in Los Angeles County Superior Court for breach of contract and "injurious denial of established fact."

Brandon received a similar claim for the reward, as well as a separate $25,000 reward to prove the authenticity of the Diary of Anne Frank, from Simon Wiesenthal, The austrian credited with hunting down dozens of Nazis who escaped Germany after Adolf Hitler's fall. The institute contends that Frank's diary was rewritten by her father and perhaps a second person. Brandon and Wiesenthal launched a test correspondence.

Brandon warned Wiesenthal that all affidavits had to be attested to by the author in person and that "hearsay evidence is not allowed." Wisenthal responded by saying he would offer a report from a Dutch expert who had examined the original of Anne Frank's diary kept in the Royal Institute for War Documentation in Amsterdam.

He said he would have the expert's signature verified by the Dutch ministry of justice and the U.S. Embassy. First, however, he requested that a California Supreme Court justice be found to hear the evidence.

Brandon replied to Wiesenthal that enlisting a Supreme Court justice "is not feasible" because the same judge might eventually have to hear the case in court. He also rejected Wiesenthal's offer of a report from a Dutch expert.

"We honestly -- if immodestly -- believe that American rules of justice are the best in the world." Brandon said. "We insist that the same rules apply to this case as apply to a normal American forgery case."

At that point, Wiesenthal cut off the correspondence. "I can understand your motives very well," he said. "You have made your decision; from the beginning you have presented Anne Frank's diary as a forgery. You are afraid to lose face after a verdict from an independent arbitrator . . . . I am sure that your conduct would be no different in the case of the gas chambers."

Brandon, in an interview, said he took the failure of Wiesenthal to pursue the matter as a sign of the weakness of his evidence. "It's our reward. The grounds for giving a reward are never determined by the applicants."

The institute director said Brandon is his pen name. He would not confirm nor deny that his real name, the one used in court papers served on him in the lawsuit, is David McCalden. He declined to be photographed, because, he said, "I an not very good looking" and because of concern about threats toward him and his organization. He says he has visited Germany for one day to pay a social call on a friend.

The founder of the institute is Willis Carto, treasurer of the Liberty Lobby, but Brandon says the institute has no ties with the Liberty Lobby.

The PhDs on the advisory committee are Walter Beveraggi Allende, an economist at the University of Buenos Aires; Austin J. Apt, and English professor retired from La Salle College; Buchner; Arthur R. Butz, and electrical engineer at Northwestern University; Robert Faurisson, an expert in document appraisal at the Niversity of Lyon, France; Martin A. Larson, an economist in Washington; Revilo P. Oliver, a classicist at the University of Illinois; Wilhelm Staglich, a retired judge in West Germany; Charles Webber, professor of German at the University of Tulsa, and Andreas R. wesserle, with a degree in goverment and urban planning and retired from Marquette University.

Brandon said the institute exists on a $100,000 annual budget from journal subscriptions at $20 a year and donations.

The Anti-Defamation League has reported 377 "anti-Semitic episodes," ranging from arson to anti-Semitic graffiti, in only 1980, compared to only 129 reported episodes in 1979. Part of the increase, the league said, could be attributed to improved reporting procedures, but it concluded that "anti-Semitism is a virulent social disease."

Asked his attitude toward Hitler, Brandon said, "We regard him with the same disdain that we do other megalomaniacs such as Churchill, Roosevelt and Stalin . . . . But we can find no practices that the Nazis followed that we did not follow also."