A new Census Bureau report estimates that by the end of the 1970s there probably were fewer than 5 million illegal aliens in the United States, a figure well below the estimate of 8 million to 12 million that some studies and government officials had offered in the past.

The Census Bureau report, which is based on an analysis of more than a dozen major studies, also estimates that fewer than half of those here illegally are from Mexico, another figure well below other estimates.

While those who authored the census report characterized their estimates as "cautious speculations," staff members and officials of the President's Select Commission on Immigration and Refugee Policy, for whom it was prepared, called it probably the best available analysis yet conducted of the undocumented foreign population.

"It's the best analysis done to date," said Dr. Lawrence H. Fuchs, the commission's executive director. "It's the best summary of studies)."

Nonetheless, the report comes at a time when the national debate over illegal immigration has reached new intensity because of the fears of many that the illegals are using services without paying taxes and taking jobs from Americans now threatened by a recession.

"Those figures are practically snatched out of the air," said Roger Conner, executive director of the Federation for American Reform, one group concerned about immigration trends. "Every other indication is that the flow is increasing."

"All those figures are pretty squishy," said Phyllis Eisen, an official with Zero Population Growth. "The number [of illegal aliens] is probably growing."

But others saw the report as merely confirming what they already knew.

"We have always argued that the figures being given out by Congress and Immigration and Naturalization Service were far too high," said Al Perex, an official with the Mexican-American Legal Defense and Education Fund.

Perez said that Mexicans have long suffered the brunt of what he called "hysteria" triggered by the higher estimates.

"The figures people have thrown out in the past create a hysterical reaction by the American public," he said. "They think they're being swamped by brown hordes. That ends up being translated into action by Congress, which tightens border restrictions, penalizes employers who hire illegal aliens, and tighten regulations affecting government benefits that should be available to all."

Perez cited a lawsuit filed recently by the Federation for American Immigration Reform against the Census Bureau as one example of what fears of large illegal immigration can do. That suit argues that there may be so many illegal aliens in some sections of the country that the 1980 cencus could be tilted in favor of some states, forcing redistricting of congressional districts if many of the illegals are counted.

Among those critical of the report's conclusions yesterday was Gen. Leon F. Chapman, former Immigration and Naturalization Service commissioner, who estimated in 1975 that there were between 4 million and 12 million illegal aliens in the country.

"The estimate we made at the time was based on the estimate of illegal aliens in each [INS] district by the district directors," said Chapman. "My judgment is based on the collective judgment of theirs. I still put more faith in their judgment, because they're the ones out there, they're the more knowledgeable."

Recently, however, there has been a tendency to reduce estimates of the number of illegals here Leonel J. Castillo, who recently left the INS as commissioner, estimated shortly before he stepped down that there were about 4 million illegals in the country.

The estimated numbers of illegal residents of the country has played a major role in the continuing political debate over whether to restrict immigration. In recent polls as many as 91 percent of those interviewed urged an "all-our" war against illegal immigration.

Some of the report's conclusions about migration patterns also differed from a number of popular conceptions. Among other things, the report said that Mexicans are probably less likely to remain in the country than non-Mexicans because , some studies indicate, a large number of Mexicans work as farm laborers and return home when harvest seasons end.

More likely to take up permanent residence illegally are those from such places as Europe, South America and the Philippines. Those illegals generally entered the country legally -- with proper visas -- only to vanish into the population when their visas expire.