A federal judge yesterday ordered the Reagan administration to take steps to cut off federal funds for higher education in Virginia and eight other states if they fail to make "substantial progress" in the next 18 months to desegregate their colleges and universities.

The ruling by U. S. District Judge John H. Pratt marks the first time in a decade-long battle over college desegregation that a federal judge has ordered the Department of Education to enforce specific deadlines.

Of the nine states mentioned in the order, Virginia is the only one that has already had a revised plan provisionally approved by the education department's Office of Civil Rights. That plan, submitted by Gov. Charles S. Robb last January, would cost the state $16 million over three years.

Pratt ordered Virginia to show "substantial progress" in carrying out the new plan by September 1984. If the state fails to meet that goal, Pratt's order would require the civil rights office to begun cutting off federal aid to the state.

Virginia's new plan--drawn up after the state's 1978 proposal was deemed a failure by federal officials--would increase the numbers of blacks students at traditionally white universities and vice versa.

Also included are building plans and academic programs to enhance the quality of education at Virginia State University and Norfolk State University, the state's two predominantly black institutions of higher education.

"Virginia is ahead of other states, but that is not to say that it is going to be easy to meet the goals in this plan," said Barry Dorsey, a spokesman for the Virginia Council on Higher Education. "We are probably in a much better position because we have already begun to develop recruitment plans for the fall."

Pratt ordered five other states--Arkansas, Georgia, Oklahoma, Florida and North Carolina, which have not submitted acceptable plans--to submit plans by June 30 containing "concrete and specific measures" to desegregate by the fall of 1985. The judge gave Texas 45 days and Pennsylvania and Kentucky until midsummer to draw up acceptable desegregation plans or face funding cut-offs.

All nine states will have to meet the September 1984 deadline for showing they had taken steps to ensure desegregation by the fall of 1985.

Pratt's order came after the NAACP Legal Defense Fund, which filed the original lawsuit in the case nearly 13 years ago, argued that without a court-supported threat of federal sanctions, the Department of Education and the states would not move quickly enough to achieve desegregation.

Elliott C. Lichtman, a lawyer for the fund, said the states failed to make progress during the years of protracted litigation and negotiation with the Department of Education and in some cases states had "gone backwards instead of forwards" in desegregating their school systems.

Education Department officials, who opposed court-ordered deadlines, declined comment yesterday on the judge's ruling.

A Justice Department spokesman said officials there were studying the order to see whether or not to appeal.