Sen. Patrick J. Leahy (D-Vt.), chairman of the Senate D.C. appropriations subcommittee, suggested yesterday that officials of the District's new public university have an "edifice complex" and urged them to consider dropping plans for a $56.7 million downtown campus.

In a testy confrontation with university officials during a budget hearing, Leahy strongly questioned the need for the campus that the University of the District of Columbia wants to build just north of Mount Vernon Square at Seventh and K streets NW.

Instead, Leahy said, if the university needs additional classroom space, buildings should be constructed on vacant land next to the university campus at Van Ness Street and Connecticut Avenue NW.

University President Lisle C. Carter Jr. Strongly rejected Leahy's criticism. He said planning for the downtown campus has been under way since 1957, and that four blocks in the Mount Vernon Square area have already been purchased and cleared for construction.

"The planning for this has been going on so long," Carter declared, "and our classes still are scattered over 12 buildings downtown."

"We're an urban university. We serve students who commute from all over the city," said university trustee Marjorie Parker. "Downtown is where the University of the District of Columbia should have been all the time . . . Putting a campus at Van Ness wasn't the best decision."

The Van Ness campus, which already has received $68.3 million from Congress for construction of large modern buildings, was originally planned for the Washington Technical Institute. The Mount Vernon campus was supposed to house Federal City College.

The new University of the District of Columbia was created in 1977 by a merger of Federal City, Washington Tech, and the District of Columbnia Teachers College.

In May 1978, Congress approved $56.7 million for the Mount Vernon campus, but said none of the mony could be spent until a new master plan for the university was approved by the mayor, City Council and the appropriations committees of the Senate and House.

Only the Senate committee has not approved the plan, but Leahy said it would probably decide soon whether to do so.

Yesterday Leahy noted that enrollment at the University and its predecessor schools has been declining since 1975. But Carter said he expected enrollment to increase substantially in the early 1980s.

This year the university has a total of 13,647 students in its classes. About two-thirds are part-timers, and the full-time equivalent enrollment is 7,985.

Leahy suggested that the university make efforts to obtain the 11-acre tract next to its Van Ness campus, which is owned by the General Services Administration. But Carter said the State Department already is moving to get that property for foreign chanceries.

"You're putting all your eggs into one basket," Leahy complained."You're making no attempt whatsoever to have any fall-back position at Van Ness if we don't approve the campus at Mount Vernon."

Carter said university officials felt strongly that it would be best to have two major campuses, rather than having all its buildings at Van Ness. CAPTION: Picture, SEN. PATRICK J. LEAHY . . . notes "edifice complex"