Powerful Senate Republican leaders urged the Reagan administration yesterday to cut deeper into planned defense spending and to slash cost-of-living increases for Social Security and other federal retirement programs as part of its new round of budget cuts for the fiscal year starting next month.

Their proposed cuts, which aides said would total $16 billion to $17 billion as opposed to $10 billion in the administration's tentative plan, were outlined to two White House aides by Majority Leader Howard H. Baker Jr. (R-Tenn.) and six other party leaders in the Senate late yesterday.

Their budget initiative came amid indications the administration is preparing to back away from two of its commitments--one to a 1982 deficit no larger than $42.5 billion, the other not to cut into Social Security and other similar programs.

Even so, a source close to those at the meeting said the two presidential aides, White House chief of staff James A. Baker III and chief of liaison with Congress Max Friedersdorf, while indicating "some latitude" on defense spending, balked at cuts in Social Security or other so-called entitlement programs, under which money goes automatically to individuals who meet certain criteria. The aides reportedly expressed no qualms about the senators' proposals for cuts in other domestic programs, which more closely dovetailed with administration plans.

Baker planned to present the proposal to Reagan today on behalf of himself and the other senators, including committee chairmen Robert J. Dole (Kan.) of Finance, Mark Hatfield (Ore.) of Appropriations, Jake Garn (Utah) of Banking, Pete V. Domenici (N.M.) of Budget and Roger W. Jepsen (Iowa) of the Joint Economic Committee, as well as Paul Laxalt (Nev.), Reagan's closest friend in the Senate.

Although it reportedly did not satisfy every member of the group in all details, the proposal reflected what appears to be growing sentiment among congressional Republicans as well as Democrats that further budget cuts should be spread out among programs more equitably than the administration has planned.

Reagan has attempted to shield entitlement programs such as Social Security from the new round of cuts for 1982. His decision over the weekend to propose outlay cuts of only $2 billion for 1982 from the Pentagon's proposal for vastly increased spending was also interpreted as a victory for the military.

As outlined by a Republican aide, the senators proposed taking $4 billion to $5 billion from defense, $2 billion to $3 billion more than Reagan has sought. They would also take $2 billion to $3 billion more than the administration has proposed from discretionary domestic programs and pick up at least $2 billion and as much as $5 billion in savings from reducing cost-of-living increases for retirement programs.

Another proposal involved up to $3 billion in new revenues, basically from user fees. The administration had ruled out tax increases.

The senators also agreed to ask Reagan to set up a presidential commission on interest rates as a means of finding ways of bringing them under control, short of credit controls and other such steps that some lawmakers have been suggesting.

To achieve the retirement savings, Domenici has proposed limiting annual cost-of-living increases to three percentage points less than the annual increase in the Consumer Price Index (CPI). They now are pegged to the full CPI. Any change would affect more than 40 million retirees, including 36 million recipients of Social Security, 1.7 million civil service retirees, 1.4 million military retirees and 4.3 million veterans.

Some of the senators reportedly had misgivings about any entitlement changes, and it appeared from the initial White House reaction that the pension proposal might encounter the stiffest presidential resistance.

"They intend to abide by whatever the president wants," an aide to one of the senators said.

The senators made no mention in their proposal of a plan, suggested last week by Senate leader Baker and House Minority Leader Robert Michel (R-Ill.), to give Reagan the power to impound money already appropriated.

The idea did not go down well on Capitol Hill, where members are reluctant to relinquish any more budgetary powers. So the senators are urging that the cuts be made through the normal appropriations, authorization and budget process, sources said.

A huge stop-gap "continuing resolution" that Congress must pass to keep the government running after the start of the new fiscal year Oct. 1 has also been discussed as a vehicle for the cuts. But the senators reportedly view that, along with impoundment, as a last resort.

The House Appropriations Committee also gave the back of its hand to the idea of using the continuing resolution for budget cuts in unanimously approving a bare-bones, 30-day resolution pegged to spending rates already approved by Congress. Further cuts will be difficult and should be considered separately, warned ranking committee Republican Silvio O. Conte (R-Mass.), adding: "You can't make these guys go through the trenches again. They're too bloodied."

In a hint of softening on the goal of a 1982 deficit no larger than $42.5 billion, Commerce Secretary Malcolm Baldrige told reporters yesterday that the target is to keep the deficit below $50 billion. "Whether it's $42.5 or $48.5 or $50 billion doesn't make any difference," he said. "It's between that range and the $70 billion to $75 billion range that the crucial difference is."

White House officials denied that the 1982 target has been changed from the previous $42.5 billion to just somewhere below $50 billion, but Baldrige said he was not speaking as an individual.

"You don't think I would come here to [lunch with a group of reporters] and give you figures that my friend Dave [Stockman]would deny. I'm sure if you asked him he would say the same. He might say, 'No comment' but he wouldn't say these figures are wrong."

Baldrige said that Wall Street would be satisfied by a deficit below $50 billion because businessmen's biggest concern is making sure the deficit is decreasing. They don't believe the $42.5 billion figure, he said, "they believe what they've been reading, that it's going to be in the $65 billion to $70 billion range . . . They're reacting to the worst-case estimates."

The problem, Baldrige said, is that spending will be $9 billion to $15 billion over earlier estimates. "Barring a serious recession, which no one predicts," Baldrige said that is all the administration needs to trim in 1982.

"Most people underestimate the president's resolve and the administration's resolve," he said.

The new round of cuts, originally scheduled to be announced this week, will not be made public until next week, White House officials said. Reagan is still considering whether or not to go on television one night next week to explain the new cuts, according to aides.

At the White House, a group of 19 "boll weevils," southern Democrats in the House who supported Reagan's first budget cuts and his tax bill, emerged from a meeting with the president saying that Social Security and other entitlement programs should not be immune to 1982 cuts. White House deputy press secretary Larry Speakes refused to repeat a pledge that entitlements are safe for another year, saying the administration is working out its program with Congress.

"There are no plans in this administration to require any cuts in Social Security," Speakes said, but he left the door open to such cuts if they arise from the discussions with members of Congress.

The president hosted House Speaker Thomas P. O'Neill Jr. (D-Mass.) and House GOP leader Michel for lunch at the White House and got only chilly words from O'Neill on further budget cuts.