The Senate yesterday averted a filibuster by agreeing to trim back a Carter administration proposal that would cut fuel oil and power costs along the East Coast at the expense of the rest of the country.

The measure, approved 53 to 40, could-cut in half a complex subsidy formula the administration had proposed under federal energy regulations that would have gone into effect Aug. 22.

It ends, at least temporarily, a spirited regional battle between senators from 17 eastern states and Michigan and the rest of the country.

Eastern senators, who earlier had threatened a lengthy filibuster to protect the subsidy, agreed to the compromise after two days of negotiations with Sen. J. Bennett Johnson Jr. (D-La.) and Energy Secretary James R. Schlesinger Jr.

Senators from New England states, which benefit most from the subsidy, accepted the compromise because "We felt half a loaf was better than none at all," Sen. Edward W. Brooke (R-Mass.) said after the vote.

Under Schlesinger, the Energy Department had planned to more than triple the subsidies for refiners and oil users in 17 eastern states, including Maryland and Virginia, and in Michigan.

Outraged by this prospect, Johnston accused President Carter of using the subsidy or entitlement program as "a $200 million slush fund . . . to reward friends and punish his enemies."

He told the easterners that he had the votes to pass an amendment to an Interior and Energy departments appropriations bill that would cut off the increases. After negotiating for the better part of two days, however, he introduced a compromise that would permit an increase in the subsidy of 50 percent, slated to end next July 1.The Carter administration had proposed a 100 percent increase.

To become law, the measure must be approved by the House and President Carter. Brooke said negotiators had contacted both House Speaker Thomas P. (Tip) O'Neill Jr. (D-Mass.) and Schlesinger. Both agreed with the concept of the compromise, he said.

It would save oil consumers in New England about $50 million during the next year, he said.

The subsidy or "entitlement" program is designed to offset the sharp difference in price between domestically produced petroleum and more expensive imported oil.

Since the East uses more imported oil than other regions, eastern consumers have been paying more for oil than consumers elsewhere.

The "entitlement" rules force refiners in the western two-thirds of the nation to pay subsidies to refiners, utilities and industries in the East to offset the higher price of imported fuel.

The District of Columbia, Virginia and Maryland benefit from the program. The Senate delegations from Maryland and Virginia split on the vote yesterday. Sens. Paul S. Sarbanes, (D-Md.) and Harry F. Byrd Jr. (Ind.-Va.) voted for the compromise. Sens. Charles McMathias Jr. (R-Md.) and William L. Scott (R.-Va.) voted against it.