After three decades of major growth the number of students at American colleges is likely to drop about 15 percent by the mid-1990s with steeper declines likely in Washington and Maryland, a Brookings Institution scholar says in a new report.

David W. Breneman, a senior fellow at Brookings, said the decline will stem mostly from the nationwide drop in births in the 1960s and 1970s and a decrease in high school graduates that is starting to take place now.

Even though many college officials hope to offset these trends by enrolling more "nontraditional students," such as older women, minorities, foreigners, or high school dropouts, Breneman warned that such hopes are misplaced.

The numbers from all these groups already have increased strikingly since the mid-1960s, he said, making it unlikely that further substantial growth will occur.

"It's a very difficult 10 to 15 years that the colleges are facing," Breneman said at a briefing arranged by the Association of Governing Boards of Universities and Colleges, which issued his new report.

"I'm afraid there is a lack of planning in a great many institutions," Breneman continued. "They hope that each year they'll make it, that it won't happen to them . . . Sure, it is possible for any one institution to beat the odds. But overall some very sharp reductions will have to occur."

Breneman said the most important factor in determining college enrollments is the number of high school graduates. Nationwide it has declined slightly -- less than one percent -- since 1979. But he said it is expected to drop by 5 percent next spring and again in 1984 because of the substantial decrease in births 18 years earlier.

In Washington and Maryland the drop in high school graduates already has begun. D.C. public schools report that last spring's total, 4,521, was down by 603 or almost 12 percent from two years ago. In Maryland, the most recent figures available show a drop of 1,050 or 1.6 percent from 1979 to 1981, when 60,893 received high school diplomas statewide.

In Virginia, the number of high school graduates has held almost steady at about 67,000 for the past three years.

To gauge the future for individual states, Breneman uses elaborate projections based on births, migration patterns, and graduation rates, that were prepared by William R. McConnell, of the Western Interstate Commission for Higher Education.

Compared to 1980, these show a 39 percent drop in D.C. high school graduates by 1987 and a 60 percent drop by 1994. For Maryland the report forsees a drop in high school graduates of 17 percent by 1987 and 35 percent by 1994. In Virginia, the projected declines are 16 percent by 1986 and 26 percent by 1994.

He said the overall drop in college enrollment nationally as in most states should be about 10 percentage points less than the drop in high school graduates because colleges are likely to have some success in their efforts at recruiting and retaining more students.

The colleges likely to have the most severe problems, Breneman said, are nonselective private liberal arts schools and the second rank of public state universities, many of which used to be state teachers colleges.

On the other hand, prestigious private and public universities, along with low-cost community colleges probably will be able to hold their enrollments, he said.

Breneman said the major expansion in federal student aid from 1978 to last year probably delayed the enrollment drop. Indeed, very slight increases have continued.