A few years ago, a group of social scientists took some perfectly ordinary children and assigned them to teachers who had been told they were subnormal.

The test was designed to show that teacher attitudes are a major factor in children's school performance. The researchers were right: the children did badly.

The project was a success. Its results were widely quoted, and it may have prevented other children from being type-cast.

But there is a recurrent nightmare among researchers these days that the children in the test will come back to haunt them, claiming they were irreparably damaged. Were they? If so, are they entitled to compensation? And if they are, who pays and how much?

This case is only one in a Pandora's medicine chest that scientists worry about under a new federal rule on compensating victims of research.

The regulation, in effect since Jan. 2, orders research institutions to tell their subjects what kind of compensation is available if they should be injured taking part in a project.

The problem, researchers say, is that the rule does not define injury and does not set up any means of making compensation. It only requires that the human guinea pig be told whether compensation is available.

"It's so ill-defined, so open-ended that in our litigious environment it's really making people caustious," said John Sherman of the Association of American Medical Coleges. "Being told there is no compensation, a person may just be completely scared away from whatever ordinary project he's asked to take part in."

Even worse, people may decide they were hurt when it otherwise might never have occurred to them. Suppose someone develops depression in the wake of answering a questionnaire about family food-buying habits. There are no guidelines to decide whether the events are connected, whethe depression is an injury, and if so, whether compensation is warranted.

"You're promoting liability -- the notion in people's minds that they might be injured," said Jerold Roschwalb of the National Association of State Universities and Land Grant Colleges.

Further questions arise. Women who took the fertility drug DES 20 years ago now find their daughters coming down with uterine cancer. Are the women who took the drug subject to compensation, even though they weren't hurt themselves? Is there a statute of limitation on researchers' liability? If so, when should it start -- when the experiment begins, when it concludes, when damage is discovered even though it may be years later, or when the damage begins, whether or not it is discovered?

It is not always even clear who the victims are. Robert S. Gordon of the National Institutes of Health recalled that thousands of men who years ago took an experimental heart attack prevention drug developed gall bladder trouble later at a rate 50 percent higher than other wise expected.

"That means that if 150 men got gallstones, 50 of them would not otherwise have them, but which 50? How do you tell? I don't know how the lawyers could possibly figure that out."

The basic idea that victims of research should be compensated is univesally supported. In the past, research subjects generally found institutions willing to pay for their injuries, but that might have been because few claims were made.

A January 1977 study by a Department of Health, Education and Welfare task force found that 3.7 percent of 133,000 research subjects, or nearly 5,000 people, suffered some kind of injury in the study period. Nearly four-fifths of those were classified as "trivial," while only 1 percent, or about 50 persons, were fatally hurt or permanently disabled.

The study looked for horror stories of injured persons left out in the cold, but apparently it found none. The injury rate, it said, was about what all of us run in the course of daily life. The numbers, in fact, are not high enough to allow insurance companies the sort of data base they need in order to write policies.

Only one institution, the University of Washington, currently holds insurance covering possible injury to its research subjects. The Aetna Life and Casualty Co. policy pays only actual hospital, wage loss and other costs associated with an injury, and it cost the university a premium of $35,000 last year, according to Diana McCann of the Human Subjects Office. In seven years of operation, it has paid off 15 persons, the largest amount being $1,500, she said. The university has an estimated 35,000 research subjects each year.

Still, McCann said, the new federal rule is troublesome. "This policy renews annually and may change, or we may go to self-insurance to save money... We can't honestly tell subjects we will compensate them in the future, or that we won't do it, either."

There is general agreement that, since it is federal money being spent in these research projects, presumably for the public benefit, then federal funds, ought to back some kind of injury compensation insurance. But federal officials are understandably wary of involvement, for the same reasons of insufficient data that worry everyone else.

"At the moment," Roschwalb said, "we have human subjects who are better informed but still uncompensated."