The Republican-controlled Senate put aside legislation to expand the president's veto power over spending measures after a third attempt to end a filibuster failed yesterday by two votes.

Chief sponsor Mack Mattingly (R-Ga.) vowed to keep pressing for the "line-item" veto, one of President Reagan's top legislative priorities for the year.

But Senate Appropriations Committee Chairman Mark O. Hatfield (R-Ore.), who led the filibuster against Mattingly's proposal, vowed just as adamantly to block any such attempt, even if it means tying up one of his own bills.

Majority Leader Robert J. Dole (R-Kan.) abandoned the push after the 58-to-40 vote, two short of the 60 needed to break a filibuster. In two previous votes over the past week, proponents of the measure mustered 57 votes, and Dole predicted Tuesday that 59 would be a high-water mark.

"The die was fairly well cast . . . there was no way cloture would be voted at this time," Dole told the Senate after informing Reagan of the Senate's failure to act on the measure, despite letters and telephone calls from the president.

The bill would empower the president, for a two-year trial period, to veto individual items in appropriations bills by requiring that they be split into component parts before being sent to the White House. A president currently must sign or veto an appropriations bill, or any other bill, in its entirety. Congressional power to override a veto by a two-thirds vote of both houses would not have changed.

Proponents described the measure as a necessary tool to cut spending and thereby control soaring budget deficits. Opponents charged that it would vastly increase the power of the White House at the expense of Congress and would invite political "blackmail" over spending programs.

Mattingly indicated that he will try in September to attach the measure to an appropriations bill, possibly the omnibus continuing resolution needed by Oct. 1 to fund programs for which regular fiscal 1986 appropriations have not been passed.

Hatfield said he was prepared to fight on any front. "This issue will not pass on this floor," he said. "I want to say, as one senator only, if I have to stand here as one person, I will fight it on every issue and bring any matter pending before the Senate to a halt in order to stop this so-called mad piece of legislation [from] trying to throw the whole balance of power of this Constitution out of kilter."