Half a million students seeking federal education grants have been rejected by a computer programmed to catch cheaters. The government is delighted.

Unfortunately, no one is sure whether the students were really cheating or were merely sandbagged by a complex application form. Furthermore, there is apparently no way to find out.

Some colleges catering to low-income students have reported a 20 percent decline in enrollment, blaming much of it on the change in the Basic Educational Opportunity Grant program.

"In trying to get the 5 percent that were beating the system, some of us believe they have really hurt the other 95 percent that need these grants," said John E. Terrell of the American Association of Community and Junior Colleges.

The Health, Education and Welfare Department doesn't look at it tha way. The program's 13 percent rejection rate, said Secretary Joseph A. Califano Jr. in a recent Washington speech, is "a major achievement." The figures are "astounding" and will result in "a staggering saving" of $300 million to $500 million this year, he said.

The HEW computer, reprogrammed last February in the wake of student loan default disclosures, rejected one-quarter of the 3.5 million aid requests received for the recent fall semester, or about 876,000 applications, according to Leo Kornfeld, deputy commissioner for student financial assistance in the Office of Education.

These were returned to the applicants for "correction," he said; about hal were resubmitted -- some of them several times -- and have finally been approved. That leaves roughly 455,000 students who applied for aid and didn't get it. Those who did get aid under the program, nearly 3 million students, represent about one-third of all U.S post-secondary-education enrollees.

Marilyn Nixon, 24, or Kansas City, Mo., was one of the early rejects, even though she had received a $481 BEOG grant to study business and fashion at Penn Valley Community College in Kansas City the previous semester.

"They sent the form back in March and said I hand't checked a little box on the back," she said in a telephone interview. The box authorizes release of application information to other aid agencies and was one of 100 "edit checks" and changes newly programmed into the computer, Kornifeld said.

Nixon then got the form back again, along with a letter naming half a dozen errors she had made in listing her taxable, nontaxable and adjusted gross income, which Kornfeld said are the most common mistakes. There are seven pages of instructons for the two-page form, many in prose reminiscent of income tax regulations.

"The wording wasn't very clear... I didn't really understand wht they wanted for adjusted gross income," Nixon said. But she sent it off again. Back it came a third time.

"This time there was a letter with a long list of things that might possibly be wrong, but it wasn't specific about my form. I redid it again, and it was wrong again." By now school had started, so Nixon borrowed money, got two months behind in her rent and let the utility bills go in order to stay in school.

Finally she appealed for help to Penn Valley and was given the toll-free number of the BEOG processing center, where an official helped her fill out the form yet again, line by line. Last week she was finally accepted for another $481 grant.

Not everyone is persistent enough to fill out the form five times. "We've lost a third of our BEOG grant recipients," said Penn Valley President James S. Manilla. As in many urban community colleges, he said, half his 6,000-member student body depends on some kind of financial aid.

Other educatiors said that low-income and moderate-income students would be the most likely to have difficulty with the complex form.

"It's the least sophisticated kids who are getting kicked out," said Jerold Roschwalb, director of government relations for the National Association of State Universities and Land Grant Colleges. "We're reasonably certain that at least 50 percent of the rejects were really eligible."

There is no way, however, to know for sure. "That kind of information is almost impossible to get," said Peter Voigt, who is in charge of policy planning for student aid programs at the Office of Education. "You can't go to an individual and say basically, 'Hey, did you cheat? It's just impossible to say what is an error and what is someone trying to play games with the system."

Kornfeld said there was "no question that complexity [of the form] leads to mistakes." Secretary Califano, he said, has ordered the form simplified but not at the expense of accuracy in the data. Some "modifications" will be made but no major overhaul is expected, Kornfeld said.

The BEOG rejection rate was the subject of what Roschwalb called "a semiheated exchange" during hearings and debates on the coming education budget. President Carter sought -- and Congress agreed -- to raise the income ceiling from $15,000 for a family of four to $25,000, a move expected to increase the number of applicants from 3.5 million to about 5 million next year.

Full funding for an average grant of about $1,000, the administration argued, would require $3.1 billion in fiscal 1980, a full $1 billion more than current appropriations. The Office of Management and Budget, however, is reported to have called for only $1.56 billion on grounds the new computer program had saved roughly $500 million this year and could be expected to save a comparable amount next year.

Kornfeld said he expects the new figure to be about $2.9 billion, apparently a compromise. "Basicaly we're very pleased with the progress we've made in administering these programs more effectively," he said. "There's nothing wrong with the computer system."