It was 361 years from the time the first black arrived as a slave in Virginia until a black student was admitted to the College of William and Mary -- the nation's second oldest college.

That was in 1966. Since then the pace of integration has been slow at this graceful, colonial campus, with its brick walks and clipped boxwoods. Last fall, blacks still made up less than 4 percent of the college's 4,760 undergraduates -- in spite of a much-heralded state plan that four years earlier had promised vigorous affirmative action.

With those numbers, William and Mary has achieved one of the worst desegregation records of all of Virginia's 13 traditionally white state colleges. By comparison, black students last year made up 13 percent of the freshman class at the University of Virginia, which, like William and Mary, is considered one of the state's top academic institutions.

Black enrollment is one measure of compliance with the state's desegregation plan. On others, William and Mary hasn't done much better. Last year, there were exactly three black professors on a faculty of 372. And black enrollment at the college's professional schools actually dropped between 1976 and 1980 -- from 20 to 17.

All are well below the objectives William and Mary was to meet under Virginia's desegregation plan. By this fall, William and Mary was supposed to enroll a total of 149 black freshmen; in fact, the number will be about 50.

In a report given to the university last spring, the college's Affirmative Action Advisory Committee found that only 8 of the college's 21 faculty departments bothered to keep records on their minority recruitment efforts, making it "nearly impossible" to determine if affirmative action plans had been followed.

Furthermore, the report noted that affirmative action guidelines were regularly waived on all temporary faculty appointments -- which constitute the majority of all appointments.

Professor Michael Faia, former chairman of the committee, says these facts only show that the college has hardly made affirmative action a top priority. "If the original objective meant anything to you, you'd get upset when it is not achieved," said Faia. "I just don't think anyone here was pushing affirmative action. It is not rampant racism; it is just inertia."

Thomas Graves, president of the college, has instituted changes recommended by the committee, agreeing that more needs to be done. "We have not made anywhere near the progress I had hoped," he said recently.

To find out why the college was having trouble recruiting blacks, the school surveyed the black students accepted at William and Mary who chose to go elsewhere. Of the 108 blacks the school accepted in 1981, 63 turned down William and Mary. Of those, 20 went to the University of Virginia, and another 11 went to Ivy League schools.

Some were lured away by better financial packages, particularly for middle-class students; others were wary of spending four years tucked away in Colonial Williamsburg; some noted the lack of certain programs -- engineering or communications, for instance; but several cited the small number of blacks on the faculty.

"Not seeing any black professors, that makes it hard," said Elizabeth Young, a senior this year who plans to go to graduate school at a college where blacks are more visible. "And I don't buy this thing that there aren't enough blacks qualified."

The college hopes to attract more black students with a reorganization of its recruiting effort -- placing responsibility for minority enrollment directly under the admissions department. Some here argue, however, that the problem is deeper than that.

"Basically we are a traditionally white institution with all that that encompasses," said Dale Robinson, director of William and Mary's affirmative action programs. "There have been significant efforts to change that image, but it is still there."