Following a series of secret high-level meetings and negotiations between congressional and District of Columbia leaders, an agreement was reached yesterday for construction of the long-stalled convention center at Mount Vernon Square in downtown Washington.

Under terms of the unusual agreement, Congress will give the actual go-ahead for the $110-million project if private investors agree to build millions of dollars in new hotels, restaurants and other facilities to serve convention visitors. The increased tax revenues from those enterprise would offset possible losses from the center's operations.

The broad terms of the agreement were worked out two weeks ago at a secret, informal meeting attended by Sen. Patrick J. Leahy (D-Vt.), Rep. William H. Natcher (D-Ky), D.C. Mayor Walter E. Washington, City Council Chairman Sterling Tucker and two mayoral aides.

Yesterday, it took the members of a Senate House conference committee, let by Leahy and Natcher - the chairmen of the Senate and House D.C. Appropriations subcommittees - just 48 minutes to ratify the agreement on the center and decide on the rest of the city's $1.2 billion budget for the fiscal year that began last Oct. 1. Washington and Tucker were invited back for yesterday's session.

Formal deliberations on the budget had been dead-locked since last Oct. 17, chiefly over the convention centre. The city has been operating under terms of a stopgap appropriation resolution passed by both houses of Congress.

The new agreement is subject to approval by the full House and Senate, where it probably will be considered next week.

City officials, along with business and union leaders, have portrayed the center as a generator of economic revitalization in the old section of downtown. Forecasts have ranged up to $100 million a year in additional business activity and the creation of 4,000 jobs.

The House, urged on by Natcher, approved the city's request last September to borrow $27 million from the U.S. Treasury to buy land for the center. The Senate, taking Leahy's advice, rejected the outlay. This led to the deadlock.

Leahy cast strong doubt on the need for the center and on several occasions demanded direct private investment in the center to supplement public spending on the project. Business leaders countered that Washington has no large corporations with large sums to invest in such a project.

City officials have said the biggest benefit from the center would be the so-called "spin-off development," such as the hotels, restaurants, bars and retail shops that are established or expanded to serve convention visitors.

With yesterday's agreement, Leahy backed away from demanding direct investment and settled for an arrangement under which the city must certify that private investors have committed themselves in advance to build such establishments. Only then would Congress release funds to start the center.

Under the arrangement, the new developments must generate tax revenues large enough to pay the entire cost of debt service and the center's operating losses within three years after the center is opened.

That would take about $8 million to $10 million a year, city budget director Comer S. Coppie estimated. That would include not only real estate taxes but also business taxes, sales taxes and income taxes paid by employes of the new enterprises.

Mayor Washington said it should not be too difficult to attain that figure.

The city already is levying two business taxes enacted last year by the City Council and earmarked for the convention center. One is an 0.9 percent surtax on the city's income tax on businesses. The other is an 80-cents-a-night tax on the occupancy of hotel rooms, which went into effect May 1. Together, they are expected to yield $6 million a year.

Under yesterday's agreement, revenues from these taxes will be used to pay the cost of the center's site and to pay operating costs of the center when it first opens and before it attracts a full calendar of convention.

The agreement also directs the city to seek ways to cut the $110-million estimated cost of the project.

Both Mayor Washington and Council Chairman Tucker, likely rivals in this year's mayoral campaign, praised the decision, both at the meeting of business and labor leaders invited to the District Building.

"I believe this is a good day for the city," Washington told the community leaders. "The whole government (executive and legislative branches) stood together . . . with a large segment of the community," Tucker said.

The agreement "in no way compromises anybody's responsibility, either on the (Capitol) Hill or down here (at the District Building)," Tucker said.

Several spokesmen for such groups as the Metropolitan Washington Board of Trade, the D.C. Chamber of Commerce, the Greater Washington Central Labor Council and the D.C. Bankers Association hailed the congressional action, and pledged support for efforts to get commitments for investments.

The agreement on the convention center overshadowed the action on the rest of the city budget, which contained at least one major disappointment for city officials.

The conferees agreed to grant a fedeal payment to the city of $276 million for the 1978 fiscal year - the sum the Senate had voted earlier - instead of the $300 million authorized by law that was sought by the city. The House had agreed to grant $295.4 million.

The main effect of this action would be to reduce the amount of money the city would carry over to help finance its 1979 budget. However, the city may seek some of the money in a supplemental appropriation request.

The conferees decided, despite a protest by Natcher, to hold up the start of work on a new $56 million campus for the University of the District of Columbia until Congress receives a master plan and revised estimates of future enrollment.

Natcher won approval for adding $1.2 million to the city's budget to keep 4,161 officers on the police payroll.The city had proposed a reduction of 186 officers by not filling jobs vacated by those who retire or quit.

The conferees also agreed to reserve $9.9 million from the 1978 budget to pay half the amount due Dec. 1, 1979, to pay off bonds floated 20 years ago to build the Robert F. Kennedy Stadium.

Leahy, in a speech to the Senate last Oct. 4, cited the city's experience with the stadium bonds in arguing against the convention center. The stadium, he said then, "was supposed to pay for itself," but never has been able to set aside any money toward paying off the bonds.

Leahy did not mention that yesterday. But Natcher, agreeing to the $9.9 million outlay, observed that the stadium "was placed on the back of the city," requiring the large payment now.