The Republican-controlled Senate, in its first vote of the year on President Reagan's military buildup, yesterday rejected on a tie vote a proposal to give him about two-thirds of the defense spending increase he wants for next year.

The 48-to-48 vote demonstrated the depth and breadth of congressional unease over the Pentagon's costly expansion plans, but it was clearly not the Senate's last word on defense spending as it heads toward major votes today and Thursday on the budget as a whole.

The proposal from Sens. Sam Nunn (D-Ga.) and Henry M. Jackson (D-Wash.), which was supported by Armed Services Committee Chairman John G. Tower (R-Tex.) as "the best of a bad lot" of cutback proposals, would have provided a defense spending increase of 6.5 percent after inflation.

Reagan requested a 10 percent increase, which the Senate Budget Committee cut to 5 percent.

Still pending are a budget substitute proposed by Majority Leader Howard H. Baker Jr. (R-Tenn.) and Budget Committee Chairman Pete V. Domenici (R-N.M.), which would put the increase at 7.5 percent, and a Republican moderates' proposal of a 6 percent increase.

The defense vote, followed by rejection of two proposals by Sen. David H. Pryor (D-Ark.) to cut spending for nuclear weapons production and foreign arms sales, came as Democrats indicated interest in the GOP moderates' proposal as a higher-tax, lower-deficit alternative to the Baker-Domenici proposal.

But it was unclear yesterday whether either of the competing plans would be approved, which led to renewed speculation that the Senate might not be able to pass a budget at all this year.

Majority Whip Ted Stevens (R-Alaska) said he would not "fall on my sword" if the Senate failed to pass a budget this year, although he indicated that he wants a budget passed and other Republican leaders reportedly are determined to get one enacted.

Earlier, Sen. William L. Armstrong (R-Colo.), a senior Budget Committee member, said that, "while it's very desirable to have a budget resolution, a bad resolution is not preferable to none at all."

These and other comments prompted complaints from Sen. Lawton Chiles (Fla.), ranking Democrat on the Budget Committee, who expressed fear that Republicans might bail out on the budget if they fail to get a tax-and-spending plan of their liking.

In the Democratic-controlled House, the Republican leadership was reported to be considering trying to scuttle a House-Senate conference agreement on the budget if it includes big tax and spending increases.

The Republicans would look instead to a "veto strategy" by Reagan to enforce budget discipline, leadership sources said.

Although Republicans were ineffectual in trying to stop passage of a Democratic version of the budget earlier this spring, it takes only one-third of the House plus one vote to sustain a veto, which would be within reach of the Republican leadership.

Some House Republican leaders are opposed to any tax increases. Minority Leader Robert H. Michel (R-Ill.) is understood to oppose any increases that might jeopardize indexing of tax rates for inflation.

Their strategy will hinge on the shape of the budget as it comes out of the Senate and, later, out of conference, sources said.

Both the House and the Senate Budget Committee have called for a $30 billion tax increase next year.

The Baker-Domenici substitute includes only nominal increases recommended by Reagan, while the Senate moderates' plan calls for $14.3 billion in new taxes this year and larger increases in the future.

Moreover, the moderates propose a $300 cap on the 10 percent income tax cut scheduled for July and repeal of indexing as one way of raising the revenue.

"In my opinion, it may be that the price we would have to pay to preserve the budget process isn't worth it," said Rep. Dick Cheney (R-Wyo.), head of the House Republican Policy Committee.

In the Senate votes yesterday, Pryor's proposal to cut back nuclear weapons production, which would have rolled back weapons funding to fiscal 1982 levels at a savings of $6.3 billion over the next three years, was defeated, 74 to 21.

Pryor said it would "slow their nuclear warheads' rate of alarming growth," but Tower described the proposal as a "blueprint for unilateral disarmament."

Another Pryor proposal, which was aimed at slowing slowed down foreign arms sales, at a three-year savings of $1.6 billion, was defeated, 59 to 18.