The Republican-controlled Senate yesterday refused for a second time to shut off a filibuster against a bill, backed by both President Reagan and Sen. Edward M. Kennedy (D-Mass.), to give the president broad new powers to veto spending items for individual programs.

After falling three votes short of the 60 necessary to limit debate, as they did last Thursday, backers of the line-item veto measure, including Senate Majority Leader Robert J. Dole (R-Kan.), indicated the effort was probably doomed.

"I think it's fairly certain we'll never quite reach 60 votes," said Dole, indicating that he will shelve the measure after one more attempt to shut off the filibuster today.

The bill would permit Reagan, on a two-year trial basis, to veto individual items in an appropriations bill by requiring that it be split into component parts before it goes to the White House. Now a president must sign or veto an appropriations bill whole.

Reagan made line-item veto authority a priority in his State of the Union address in February, wrote a letter in its behalf while hospitalized for cancer surgery and made a half-dozen telephone calls to wavering senators the past two days. Dole had described the cloture vote as a test of the post-surgery "sympathy" factor for Reagan in Congress.

But it ran up against powerful institutional reluctance to expand the presidency at the expense of Congress, even on such a sensitive issue as spending and deficits.

"The president could hold mortgages on the votes of every senator in this chamber," said Senate Appropriations Committee Chairman Mark O. Hatfield (R-Ore.) in leading the filibuster, a rare role for one of the chamber's top members.

The tally indicated that no more than 59 votes could be counted on for cloture. Supporters picked up the votes of Sens. Howell Heflin (D-Ala.), Arlen Specter (R-Pa.) and William L. Armstrong (R-Colo.), who was the one absentee Thursday. But they lost Sen. Robert T. Stafford (R-Vt.); and Sens. Paula Hawkins (R-Fla.) and Joseph R. Biden Jr. (D-Del.), supporters of the bill, were absent.

The surprise in both days' votes was Kennedy's support for the measure, reinforcing speculation about his image-polishing in preparation for a possible presidential bid in 1988. Kennedy called it "an idea whose time has come" to help control deficits and increase presidential accountability for spending.

Debate during the week-long filibuster hit on some of the most sensitive chords in Congress, ranging from its spendthrift reputation to its concern over losing power to the executive branch, especially its hold over the federal purse strings.

The issue hit the floor as House-Senate talks on cutting spending by more than $250 billion to halve deficits over three years teetered on the verge of collapse, providing helpful mood music for the president's campaign for expanded power to cut spending.

"Opponents of item veto ignore the fiscal fires that are now ablaze," contended Sen. Mack Mattingly (R-Ga.), who led the charge for a line-item veto. "They would appear to . . . ignore everything but protecting their little corner of the universe. They seem to be afraid that any change in the current system might somehow lessen their power and influence."

Hatfield and others countered that the move was more a bid for power and an invitation to political chicanery than a solution to the government's fiscal ills.

Hatfield described it as, among other things, "constitutional madness and a mindless affront to the concept of separate but equal branches of government . . . , an open invitation to political mischief . . . and a nasty swamp that serves as a breeding ground for lawsuits, confusion and escalated confrontation."

Reagan's record of using available tools for fiscal discipline, including the veto of entire appropriations bills and deferral and rescision of appropriated funds, became an issue for both sides.

According to Hatfield and other opponents, Reagan has vetoed only two appropriations bills in 4 1/2 years, a continuing resolution in 1981 and a supplemental spending bill in 1982.

Proponents argued that this was precisely why he needs new powers; the existing process doesn't work. By splitting bills into individual items, he can zero in on excessive or unnecessary spending, without holding programs hostage.

Opponents quickly exploited the argument. How would Republicans have liked a line-item veto power in the hands of Lyndon B. Johnson, asked Sen. Alan Cranston (D-Calif.).

Moreover, picking up an argument made by American Enterprise Institute scholar Norman Ornstein, Cranston said the result could be more spending, not less, if a president tried to win a lawmaker's support for one program by using a veto as carrot and stick -- and won.

As it stands, Hatfield argued, Congress has appropriated less in discretionary spending the past four years than Reagan requested: $2.237 trillion as opposed to $2.249 trillion.

With no attempt to defend Reagan on that score, Kennedy argued that line-item veto power would "instill a new and needed measure of presidential accountability in federal spending."

Said Kennedy: "The president pleads . . . that he fervently detests the deficits, but does not have the power to fight them fully. So let's give it to him. Let's help him live up to his own rhetoric."