Four years ago, bulldozers and wrecking crews began demolishing the homes of 183 families and 16 stores in the heart of Washington's inner city so that the new downtown campus of the University of the District of Columbia could be built there.

Today, the four-square-block site just north of Mount Vernon Square is a wasteland of trash, abandoned cars and temporary parking lots - testimony to the power of a key congressional-subcommittee that has held up funds needed for construction.

Aobut $9.7 million was spent to buy and clear the site, and another $4 million is being spent to convert the old D.C. central library on the square into the president's office and university library, but so far not a single dollar has been appropriated for construction.

The resulting delay has seen the Mount Vernon campus project become a tightly wound and emotional political controversy with racial and economic overtones. The delay also has threatened to pit the District of Columbia against the Senate subcommittee charged with overseeing the city's budget.

"Our feeling is we're talking about a self-government issue," said Ronald H. Brown, chairman of the UDC Board of Trustees. "Every responsible body in the District of Columbia has spoken [in favor of the campus]. If we as citizens can't decide where we want something located...if we can't make that decision by ourselves, we make a mockery of the limited self-government we have."

Congress approved $56.7 million to construct the Mount Vernon campus in June 1978. But those funds could be spent, it stipulated, only if the mayor, the city council and the House and Senate Appropriations Committees approved a new comprehensive plan for the university.

All have approved that plan except the Senate committee. Sen. Patrick J. Leahy (D-Vt.), chairman of its D.C. subcommittee, is not yet convinced that the city needs the downtown campus. He has asked university officials to consider abandoning the Mount Vernon site and housing the entire school in one location by expanding its $68.3 million Van Ness campus now under construction on upper Connecticut Avenue, NW.

Leahy's position has drawn the ire of university officials, students and alumni, who say a campus is needed downtown to be convenient to student housing and jobs. By contrast, they say, Van Ness is in a relatively remote, overwhelimingly white area that is alien to most of UDC's student body, which is 90 percent black.

Even the opening of the Metro subway line in 1981, which will put Van Ness 10 minutes from downtown, will not reduce the "psychological distance" of that campus for some black youths, they contend.

During the past few months, UDC students and alumni have rallied against Leahy and threatened to campaign on behalf of his opponent in next year's Vermont Senate race. In one article published on the front page of a Vermont newspaper, they called him a racist. Several weeks ago, they challenged him to a televised debate in his hometown, a challenge that Leahy accepted. No date has been set for the debate.

University administrators and Mayor Marion Barry have publicly kept their distance from the students and alumni activity, but the protests have irritated Leahy, who on many other issues, has been the District's principal defender on an often hostile Capitol Hill.

"I've tried to deal with the [D.C.] government fairly on this," Leahy said. "I've asked them to get the answers in and I hear, "Yeah, yeah, heay, I'm gonna do it." Then I have these people who claim to be acting on behalf of the university and give the appearance of acting on beahlf of the city up in Vermont, and they say I'm racist because I won't approve this."

In a spirited interview, Leahy repeated a threat he made to Barry - that he will, in effect, kill the project, if the actions persist.

"I'm catching enough flak for defending them [the city] on the summer jobs program, full Congressional voting representation, and [government-funded] abortions.... I'm just growing weary of sticking my neck out for the city," Leahy said. "If they're that insistent [on getting approval of the Mount Vernon campus], I'll move it as an amendment [on the floor of the Senate], and I'll vote for it."

Asked how many other affirmative votes he thought the proposal would get, Leahy responded, "About 12."

UDC President Lisle C. Carter Jr. was asked to comment on such a move. "I don't think it would be a helpful outcome," he said.

Since early 1978, when the General Accounting Office issued a critical report, the crux of the issue as seen by Leahy and other skeptics has been whether the univesrity will have enough students to fill all the buildings it wants.

Enrollment was down last fall for the third year in a row placing it about 20 percent below the peak reached in 1975 when the three public colleges merged to form UDC - Washington Technical Institute, Federal City College and D.C. Teachers College.

Last fall, UDC's enrollment totalled 13,647, three quarters of whom were parttime students. Using a formula employed by educational planners, UDC's 13,647 largely parttime students become the equivalent of a fulltime enrollment of 7,985 students. That is just 700 more fulltime students than the planned capacity of the Van Ness buildings now under construction.

If the two campuses were built as planned, UDC would have enough space for 13,000 fulltime students, or 60 percent more than are currently enrolled, according to the formula used to translate parttime enrollment to its fulltime equivalent.

Leahy and others cite the possibility that 599 large a facility might be built as reason enough for the univesrity's backers to consider alternatives.

University officials say, however, that because of inflation and changes in plans, the two campuses combined actually would support between 9,600 and 11,200 students. The officials say that, as of now, they anticipate an enrollment of 9,600 by 1983.

The original capacity figures were calculated on the assumption that Van Ness would house a two-year junior college program, according to university vice p esident Claude Ford, while the Mount Vernont campus would be the home of the four-year Federal City College and graduate school programs. As a result, the Mount Vernon campus would need more extensive library and labroatory space per student.

Now, he said, UDC trustees have adopted a different plan. Under that plan, all science and education programs - from junior college to graduate level - will be at Van Ness. All liberal arts and business courses will be located at Mount Vernon.

Ford said that major construction changes will have to be made in what has already been built at Van Ness so that it fits the new plan. He estimated that the changes, which are to be financed from funds to be spent on the Mount Vernon campus, will cost about $8.6 million.

Before Leahy and his committee staff members decide what they fell the committee should do, the Senator said he wants to study a new General Services Administration report currently being prepared.

That report will study whether 11 acres of federally-owned land adjacent to the Van Ness campus would be better used for expansion of the university or for the construction of new foreign chanceries, a proposal the State Department backs.

Whatever happens, Leahy and his subcommittee must act quickly.

On Sept. 30, the appropriation for the Mt. Vernon project will expire unless at least $1 of the funds is spent. An expiration would force the city to usher another request all the way through Congress, a prospect some city officials think might be difficult.

"The fact is, there is not a tremendous amount of support on the Hill for UDC, much less for the campus," said Ivanhoe Donaldson, Barry's general assistant. "Whether or not there's a UDC in the District of Columbia just doesn't ring any bells in Iowa." CAPTION: Picture 1, The proposed Mount Vernon Square campus site of UDC is a veritable junkyard. By Gerald Martineau - The Washington Post; Picture 2, A UDC sign frames a portable toilet.; Picture 3, An abandoned service station next to the proposed UDC campus site is for sale. By Gerald Martineau - The Washington Post; Picture 4, Debris fills the proposed UDC downtown campus site, cleared four years ago. By Gerald Martineau - The Washington Post