Court-ordered busing for school desegregation has created an "incredible surge" in white enrollment in private schools, which has increased racial separation between "predominantly minority public schools and predominantly white private schools" in many cities, a Rand Corp. researcher told a Senate subcommittee yesterday.

David J. Armor, a sociologist who termed busing "the most unpopular, least successful . . . national policy since Prohibition," said mandatory busing plans have stirred such great opposition that not only have "white flight" to the suburbs accelerated, but in recent years there has been "an increasing reliance on private schools to flee" from busing.

In some cities the switch by whites to private schools has been so great, Armor said, that it has reversed the long-term decline in white private school enrollments. For example, in Los Angeles, a 20,000 increase in private school students occurred between 1978 and 1980, the first three years of a major busing program which has just been discontinued. In Boston the proportion of whites in private schools rose from about one-third before using to just over one-half in six years.

Armor said he feels strongly that discrimination should be combatted, but he said court-ordered busing should be curbed because in many cities it has generated such white flight that it has served "to increase rather than decrease, racial isolation."

At the hearing, conducted by the Senate judiciary subcommittee on the Constitution, Armor was sharply challenged by Julius Chambers, president of the NAACP Legal Defense and Education Fund.

Chambers was a lawyer in the Charlotte, N.C., desegregation case, which produced, in 1971, the Supreme Court's first major busing decision. He said the busing was necessary to end state-imposed segregation, and has led to clear gains for black students. Even though white enrollment has declined in public schools and increased somewhat in private ones, Chambers said after nine years of busing whites still make up 65 percent of the students in the public school system, which includes both city and suburban areas.

"No one has offered a viable alternative" to busing, Chambers declared. If Congress tried to ban busing, he said, "we would simply sanction segregation throughout the country."