Declaring "it is too late to compromise." Sen. Patrick J. Leahy (D-Vt.) predicted yesterday that Senate conferees would hold firm next week and kill the District of Columbia's proposal for a $110-million downtown convention center.

Leahy, chairman of the Senate District Appropriations Subcommittee, said in an interview that he would attend the Senate-House conference committee meeting next Wednesday with the right to cast five votes - a clear majority of the Senate's nine-vote delegation - against the controversial project.

Leahy's House counterpart, Rep. William H. Natcher (D-Ky.), issued a terse statement, meanwhile, declaring that the House conferees "will endeavor to convince the Senate conferees that the convention enter project is necessary and should be approved."

Natcher, for nearly two decades the most powerful congressional figure on District budgetary matters, gave no clue of the strategy he plans to use to sway Senate votes his way. His adversary, Leahy, is serving his first year as chief SENATE OVERSEER OF THE District budget.

A joint conference committee of Senate and House members is formed to resolve differences between bills passed in differing forms by the two chambers. Each chamber has an equal voice in conference deliberations. In case of disputes. House and Senate conferees vote among themselves on their chamber's position.

The House version of the District budget bill for the 1968 fiscal year, which began last Saturday, includes $27 million in start-up appropriations for the center project. The Senate version omits the project. An attempt Tuesday on the Senate floor to add the center money to the bill lost, 65 to 25.

Although there was no House vote directly on the center, Rep. Robert E. Bauman (R-Md.), a foe of the project, forced a test vote on the related question of letting the city borrow construction funds from the U.S. Treasury. The borrowing authority came within nine votes of losing 196 to 187.

Leahy voiced his prediction of victory for opponents of the center after reading a news report yesterday that some supporters were hoping to rescue the project by some form of compromise - perhaps cutting; its size or moving its site away from southwest of Mount Vernon Square.

"I didn't know we were considering a compromise," Leahy said. Mayor Walter E. Washington said through a spokesman that he firmly opposes any compromise on the center.

Some days before the Senate vote on the question. Leahy said he passed work to supporters of the center that "I'd be happy to talk any time" on the question, but "nobody wanted to talk . . . nobody came to see me."

"Now that the Senate has spoken by such a resounding vote, it is too late to compromise," Leahy said. "There have been a lot of senators - senior senators - who voted on my side who have come to me and told me they would not be very happy if I have it away in conference."

Leahy said he would enter the conference next Wednesday with his won vote and proxies - the right to cast quentin N. Burdick (D-N.Dak.), Dennis DeConcini (D-Ariz.), Warren G. Magnuson (D-Wash.) and Daniel K. Inouye (D-Hawaii).

"In effect, as chairman, I will be speaking for (a majority of) the Senatt conferees," Leahy said.

Other Senate conferees, who supported the center project, are Sens. Charles Mac Mathias Jr. (R-Md.), Lowell P. Weicker Jr. (R-Conn.), Milton R. Young (R-N. Dak.) and Ernest F. Hollings (D-S.C.). Leahy said he expects Hollings on case of a dispute, to vote for the Senate majority's position, and not for his own personal views.

When House conferees were being named yesterday, Bauman said he had intended to propose a motion instructing the House conferees to refect the center. But the strength of he Senate vote against the project made that action unnecessary, Bauman said.

House conferees, in addition to Natcher, include these members who supported the center in the test vote on the Treasury borrowing power: Reps. Robert N. Giaimo (D-Conn.), Yvonne B. Burke (D-Calif.), Adam Benjamin Jr. (D-Ind.), Clair W. Burguner (R-Calif.) and Elford R. Cederberg (R-Mich.). House conferees who voted against the Treasury borrowing are Reps. George H. Mahon (D-Texas), chairman of the Appropriations Committee) Gunn McKay (D-Utah) and Jack Kemp (R-N.Y.). A conferee who did not vote on the issue was Rep. Charles Wilson (D-Texas).