Budget director David A. Stockman, already under attack for his candid criticism of farm subsidies and military pensions, yesterday drew fire from the higher education community for telling a House panel that colleges care more about their finances than the welfare of their students.

"It's untrue, misguided, inappropriate, insensitive," said Robert L. Clodius, president of the National Association of State Universities and Land Grant Colleges, which represents most of the largest state-supported schools. "It's demeaning and it's a put-down to all the university presidents who view the education of young people as their highest priority."

Stockman, defending the administration's student financial aid cuts, which officials said would knock an estimated 1 million students off federal aid rolls, said Thursday: "Now you're going to get a lot of pressure from colleges on it . . . . They're not worried about the student; they're not worried about equity in America; they're worried about financing their budgets."

Reaction was swift.

"He's been talking a lot of nonsense lately, and I would put this under the same category," said Allan W. Ostar, president of the American Association of State Colleges and Universities. "It's outrageous, like the other things he said were outrageous . . . . For him to say this opposition to cuts is simply a self-serving ploy on the part of the universities is sheer nonsense."

John D. Phillips, president of the National Association of Independent Colleges and Universities, retorted: "I'm really gratified to know there's one person in Washington who has a sure finger on everyone's motivations and can speak so categorically."

Democrats on Capitol Hill also joined the battle against the Office of Management and Budget director.

"This is just another example of David Stockman shooting from the hip without any factual knowledge," said Rep. Augustus F. Hawkins (D-Calif.), chairman of the House Education and Labor Committee. "No matter what Mr. Stockman says, the proposed cuts in student aid represent another step in the administration's efforts to destroy the principle of equal access to higher education."

Sen. Claiborne Pell (D-R.I.), ranking minority member of the Senate's Education subcommittee, said, "I think it's wrong to say that our colleges and universities, which are the finest in the world, are not interested in educating our children. It will be difficult, however, for those colleges and universities to educate children who are unable to attend for lack of money."

The Reagan administration's proposed $15.5 billion education budget for the 1986 fiscal year includes plans to deny federally guaranteed loans to students from families earning more than $32,500, while placing a $4,000-a-year limit on the amount of aid available to any student.

University officials said yesterday that those cuts likely would cause a drop in college enrollment, affecting the already shaky financial status of some cash-strapped colleges.

"Certainly there's an element there in Stockman's quote I can't argue with," said Lawrence E. Gladieux, executive director of The College Board, an organization that sets standards for colleges and universities nationwide. "Institutions are worried about their survival and they do depend in part on federal assistance, on the backs of the students in the form of student aid."