In a severe challenge to President Reagan's "Star Wars" missile defense plan, 46 senators yesterday demanded a huge cutback when the Senate Armed Services Committee sets budget ceilings on Pentagon programs next month.

The bipartisan coalition called for no more than a 3 percent after-inflation increase for the Strategic Defense Initiative (SDI) rather than the 74 percent rise Reagan requested fiscal 1987, the budget year starting Oct. 1.

"Our concern," the 46 senators said in a letter to the Armed Services panel, "is that the Strategic Defense Initiative has received excessive and inappropriate emphasis in the Department of Defense's budget."

"Not only are the goals of the research effort unclear, the need for accelerated funding for a long-range program such as SDI has not been demonstrated," they wrote. "We are concerned that the SDI program is being rushed to a premature development decision in the early 1990s in order to meet an unrealistic schedule."

In size and makeup, the Senate coalition represents one of the biggest challenges to SDI since Reagan proclaimed it on March 23, 1983, as an attempt to make nuclear offensive missiles obsolete. The senators' letter to committee Chairman Barry Goldwater (R-Ariz.) and the panel's ranking Democrat, Sam Nunn (D-Ga.), was clearly designed to influence committee decisions on the defense authorization bill when markup sessions scheduled for June 2 begin.

Last night, Rep. Charles E. Bennett (D-Fla.), second-ranking Democrat on the House Armed Services Committee, hailed the Senate drive as a "fine effort" and said he is putting together a similar coalition in the House in hopes of freezing SDI at this year's budget level without allowing for inflation.

If such a push develops, it appears that Reagan will soon be fighting for the centerpiece of his strategic program on two fronts at a time when both the House and Senate have set goals for his total military budget far below the amounts he requested.

Under the senators' proposal yesterday, the Pentagon's SDI account would rise from $2.76 billion in fiscal 1986 to just under $3 billion in fiscal 1987. Reagan is requesting $4.8 billion. The senators' $3 billion represents a growth of 3 percent plus a 4 percent allowance for inflation. If Reagan prevails, the SDI account, with Energy Department funds for nuclear research added in, would jump from $3 billion to $5.4 billion between fiscal 1986 and 1987, a 77 percent increase.

"It is difficult to conceive of a sound rationale for increasing the combined Department of Defense and Department of Energy SDI budget by 77 percent while the entire Department of Defense budget will be frozen at zero real growth and other vital military research programs are facing budget cuts," the senators said in their letter.

The senators also said they were pressing for a slowdown in SDI because of concerns that it was wasting money by going too fast, a fear that its flight tests would violate the 1972 antiballistic-missile treaty, and a belief that conventional military forces would otherwise be excessively slashed to meet House and Senate budget goals.

Said Sen. William Proxmire (D-Wis.), one of those who put together the coalition: "Challenger and Chernobyl have stripped some of the mystique away from technology."

"We could have gotten 51 senators a majority to sign the letter if we had a little more time" before the Memorial Day recess, said Sen. J. Bennett Johnston (D-La.).

Joining Proxmire and Johnston in forming the coalition were Sen. Lawton Chiles (D-Fla.), ranking minority member on the Budget Committee, and Republican Sens. John H. Chafee (R.I.), Daniel J. Evans (Wash.) and Charles McC. Mathias Jr. (Md.).

Signers of the letter included moderate conservatives Lloyd Bentsen (D-Tex.), Wendell H. Ford (D-Ky.) and Russell B. Long (D-La.) and liberals Edward M. Kennedy (D-Mass.), Patrick J. Leahy (D-Vt.) and Carl Levin (D-Mich.). Nine Republicans signed the letter: Chafee, Evans, Mathias, Mark Andrews (N.D.), Mark O. Hatfield (Ore.), Nancy Landon Kassebaum (Kan.), Arlen Specter (Pa.), Robert T. Stafford (Vt.) and Lowell P. Weicker Jr. (Conn.).