A New Jersey judge today upheld the right of the Educational Testing Service to use statistical methods to check for possible cheating in college entrance exams and to invalidate scores of suspected cheaters.

The decision by Superior Court Judge Richard S. Cohen in New Brunswick came on a court challenge by four New Jersey students accused of cheating while taking the scholastic aptitude test (SAT) last May.

The landmark case is the first court test of the procedures adopted by the service, which administers 5 million tests a year nationwide.

The students, who were graduated this spring from Millburn High School, said they would appeal the decision.

The Princeton-based Educational Testing Service (ETS) checked the students' scores after another student accused them of cheating. ETS used the K-index, a statistical method of comparing answers to determine the rate of correlation between tests taken by different students.

Of 85 questions on the verbal portion of the test, all four students agreed on 42 correct answers and three or more agreed on 17 incorrect answers, according to their attorney, George B. Gelman. Their scores ranged between 490 and 630 out of a possible 800.

During the trial, Gelman called the K-index, developed by Frederick R. Kling, "a piece of statistical garbage." The method, he contended, was based on the probability of students' getting the same incorrect answers. He contended that his four clients, from similar social and economic backgrounds, would tend to make the same mistakes.

The four students took the test at 6 a.m. on May 1, 1982, earlier than the rest of their classmates, in the office of their tennis coach so that they could play in a tournament. In a deposition, the coach said he had remained in the room during the test and did not believe cheating took place.

However, the question of whether the students cheated was not at issue during the three-week, non-jury trial, which dealt with whether ETS had proper cause to invalidate the students' scores based on the K-index. The suit sought a permanent injunction barring ETS from invalidating the scores and reporting its findings to the colleges the students--Steven Haskin, James Denburg, Richard Becker and Lloyd Berkowitz--plan to attend this fall.

Robert J. Salomon, executive vice president of ETS, said today "Our test security procedures are fair," and that when a score is questioned, "Sanctions taken are the mildest possible." Students, he said, can retake the test, supply more information, cancel the score, submit the issue to independent arbitration or send the file to the college to decide.

However, Gelman argued that the burden of proof should be on ETS to prove the students cheated, rather than on the students to prove they didn't.

"We were shocked and surprised at the result" of the trial, said Jill Haley, Gelman's associate. "We believe ETS procedures are unfair."

Wendy Nardi, an ETS spokesman, said that in the year ending June 30, 1,827 test scores were challenged by ETS, but of those, only 596 were judged to be potentially invalid by an ETS board of review. More than half of those tested canceled their scores voluntarily while the rest either elected to be retested or submit additional information.

Last year, ETS invalidated the advanced-calculus placement scores of 14 Latino students from an East Los Angeles high school, rousing protests from the Mexican-American Bar Association and other Hispanic groups.

Twelve of the students retook the tests and passed again with high scores.