The Republican-controlled Senate yesterday overwhelmingly rejected President Reagan's plan to terminate more than 40 federal programs and endorsed an increase in spending for education as it began voting on a fiscal 1987 budget.

In both votes, the Senate at least tacitly supported tax increases that Reagan opposes to finance the programs and demonstrated continuing congressional support for domestic spending programs that the president wants to curtail or kill.

The votes underscored the difficulty of achieving a compromise under the rigid deficit ceilings of the Gramm-Rudman-Hollings budget-control law, especially in an election year.

The proposal to end programs called the bluff of many conservative senators -- who oppose defense-spending cutbacks and tax increases -- on the question of domestic-spending cuts that would be needed to meet the deficit target. But higher domestic spending resulting from the education vote may make it harder to get the conservatives' support for a final budget compromise.

Together, the votes further divided Congress and the White House on budget priorities, despite the hope of Gramm-Rudman-Hollings' drafters that the law would contribute to compromises aimed at avoiding automatic spending cutbacks that are required when deficit targets are not reached.

"A good start in the wrong direction," said Senate Budget Committee Chairman Pete V. Domenici (R-N.M.) in reaction to the Senate's 60-to-38 vote to raise taxes by $300 million to keep education programs from suffering losses from inflation and from earlier Gramm-Rudman-Hollings spending cuts.

There is "no surplus of votes" for a budget in any case, said Sen. Lawton Chiles (Fla.), ranking Democrat on the budget panel. He contended that additional domestic spending of any kind could make passing a budget more difficult.

It was Domenici and Chiles, however, who engineered the 83-to-14 vote against Reagan's proposed termination of more than 40 programs, including such popular ones as Amtrak, agricultural extension services, postal subsidies and small business loans.

Domenici had been chafing for weeks under criticism from both the White House and GOP conservatives, who contended that a budget prepared by his committee relied too heavily on tax increases and cutbacks in the president's defense request and dealt too gently with domestic programs.

As recently as Monday, Office of Management and Budget Director James C. Miller III fired off a letter to Domenici and other Senate leaders noting that the committee approved only three of 44 program terminations recommended by Reagan, 10 fewer than the Senate approved last year.

Earlier, 24 Republican senators, mostly conservatives, signed a letter that attacked both the defense and tax provisions of the committee budget, contributing to a month-long stall on the measure as Senate leaders groped for a way to get their support.

In an attempt to prod the Senate into action, Domenici tossed out on the floor an amendment to end the programs at a savings of $4 billion, which would be deducted from the committee's proposal for an $18.7 billion tax increase.

Domenici read through the list of targeted programs, pausing at times to note the small savings that would result from individual terminations, along with some of the political costs, such as loss of postal subsidies for charities, assistance to 4H clubs and relief for the hungry and homeless.

While he could not support the terminations, he said, he wanted to give those who supported them a chance to "come to the floor . . . and make a case."

Sen. William L. Armstrong (R-Colo.) arrived shortly, complaining that Domenici's proposal was designed to "embarrass" senators. "I think this is a sham," he said.

When the vote came, 11 of the 24 signers of the letter, including its organizer, Sen. Dan Quayle (R-Ind.), voted against the terminations. Several, like Quayle, are up for reelection this year.

No Washington-area senators voted for the terminations, although Sen. Paul S. Trible Jr. (R-Va.) had signed the letter.

The education proposal, proposed by Sens. Mark Andrews (R-N.D.) and Ernest F. Hollings (D-S.C.), who was a sponsor of the new budget law, would add $300 million to both spending and taxes this year. The measure contemplates nearly $3 billion in new spending authority over three years.

It would restore funds cut in the first round of Gramm-Rudman-Hollings reductions earlier this year and would provide for increases to cover inflation. This prompted some senators to worry that it would set a precedent for add-backs for other high-priority programs.

"If we're going to do that with education, it's Katy-bar-the-door," said Sen. J. James Exon (D-Neb.).

"Tomorrow we get a chance to break the bank," groused Domenici in reference to a proposal to add back as much as $4.6 billion for revenue sharing with local governments.

But the education amendment drew support from about one-third of Republicans, including most who are up for reelection this year, as well as nearly all Democrats.

The president's budget office had a mixed reaction to yesterday's votes.

OMB spokesman Edwin L. Dale Jr. dismissed the vote on program terminations as "a little innocent game-playing that can't be taken seriously." But, on the education vote, he said, "The Senate seems to be going in the wrong direction -- more spending and more taxes."