A key Senate subcommittee yesterday voted to raise the District of Columbia's annual federal payment to a record level and to restore funds for a city-run gambling operation that had been removed from the budget by gambling foes in the House.

The two subcommittee actions were part of a compromise under which Mayor Marion Barry agreed to scrap his plans to rank potential police officers according to a lottery rather than on the basis of civil service test scores in an effort to hire more blacks and women. Angered representatives in the House had passed legislation forbidding such a lottery.

Barry and subcommittee Chairman Sen. Alfonse D'Amato (R-N.Y.), who appeared to have reached a surprisingly close working relationship, disclosed yesterday that instead of the lottery, the city will simply hire all of those among the 635 current police officer candidates who pass background checks and physical examinations.

The subcommittee's votes on gambling and the federal payment, which would be increased to $336.6 million, are only a preliminary step in Senate action on the city's $1.9 billion budget for 1982, which must still go to the full Appropriations Committee, the Senate floor and, most certainly, a House-Senate conference committee.

But they gave Barry, who had launched an intensive personal lobbying effort in the Senate over the past few weeks, a rare victory on Capitol Hill following a recent series of stunning setbacks in the House that prompted angry charges of congressional interference with the city's home rule government.

"This is a day of great joy for the citizens of the District," proclaimed Barry, as he stood side by side with D'Amato. "I'm just delighted to have such a good friend in Sen. D'Amato. I had kidded him that I didn't know what to expect from a conservative senator from New York. But we need more friends like this."

For his part, D'Amato strongly denounced those members of Congress -- most of whom are from his own party -- who "seek to impose humiliating controls on the city. Washington, D.C. should not be treated like some terrible animal.

"It's a tough enough job being the chief executive of a local or state government," added the former town supervisor of Hempstead, Long Island. "We have an obligation to make sure that we do not allow petty, partisan politics or philosophies to get in the way of managing a large urban city."

The action on legalized gambling was perhaps the most signficant victory for the city yesterday.

City voters had overwhelmingly approved gambling in a 1980 referendum. Three months ago, however, the House District Committee, with most of its members voicing distaste for gambling in the nation's capital, knocked out $628,000 for the city's new Lottery and Charitable Games Control Board to set up the lottery.

But D'Amato quickly moved to restore the money yesterday, noting that despite personal reservations about gambling as a source of revenue, the "people of Washington have spoken.

"I find it very hard for people to say they oppose the lottery in the District when you have it right here in the state of Maryland," D'Amato said later.

"If the Congress is so concerned about this, let them vote to prohibit gambling nationwide," he said. "But they wouldn't do that, because that would be a violation of states' rights."

The action now goes to the full Appropriations Committee headed by Sen. Mark Hatfield (R-Ore.), a longtime gambling opponent who introduced a resolution to overturn the city's gambling initiative.

But Hatfield, who had spoken with Barry by telephone two days ago, has agreed not to oppose the restoration of gambling funds as long as no federal funds are used -- an arrangement that would be acceptable to the city, according to a spokesman in Hatfield's office.

The subcommittee's vote on the federal payment, which is provided to reimburse the city for tax-exempt federal properties and the additional municipal service costs associated with being the nation's capital, is in accordance with recent legislation signed by President Reagan raising its maximum level to $336.6 million.

The House, however, had kept the payment at $300 million when it acted on the budget last month.

The subcommittee also voted yesterday to strike another amendment inserted by the House temporarily blocking a city plan to truck its sewage sludge to a landfill in Bucks County, Pa.

Instead, the panel approved language that the city may not send the sludge unless it receives the necessary state health permits. Barry said the amendment was much preferable to the House version because "we could not take sludge to Pennsylvania if the state doesn't agree, anyway."

Barry said the compromise agreement on the the politically sensitive police-hiring issue would avoid complaints from the city's mainly white police unions that the mayor's affirmative-action plans would lower standards dangerously in the department.

Rep. Stanford E. Parris (R-Va.), responding to complaints from D.C. firefighters and police officers who live in his congressional district, had introduced the amendment to block the plan.

Barry said yesterday that the city will simply "hire everybody" from a pool of 635 applicants who passed a recruitment examination last March, provided they clear background checks and pass physical examinations.

But city personnel officials, including personnel director Jose Gutierrez, said they were unaware of the mayor's comments and didn't know how his new plan would work.

The agreement raised the possibility that the city could wind up hiring more than the 200 new police officers whom Congress has pressured it to hire, though union and city personnel officials noted that normally, fewer than 50 percent of the candidates pass the physical and background checks.

The new plan to hire everyone also leaves open the crucial question of the order in which the new candidates will be selected, since new police officers usually are hired in stages, with about 40 at a time processed through the police academy.

"We could hire everybody, but what are we going to do with them," said Robert Storey, a special assistant to Gutierrez. "Obviously, he Barry knows something we don't."