The Senate dealt a major blow yesterday to the Carter administration's controversial U.S.-British tax treaty, resulting in an agreement to strike a key provision that leaves the fate of the pact in doubt.

After first rejecting the treaty altogether, the Senate agreed to reconsider it on Tuesday without the offending provision, which would have limited the reach of states in taxing British multinational corporations.

Although the amended version is considered likely to win enough votes for approval, it could effectively kill the pact anyway. Officials said it was not clear whether the British would accept the treaty without the key provision.

The compromise was engineered by Sen. Jacob K. Javits (R-N.Y.) after the Senate earlier voted to reject the treaty. The vote was 49 to 32 - a plurality in favor of the pact, but 5 votes short of the two-thirds majority needed.

Opposition to the provision involving state taxing practices was led by Sen. Frank Church (D-Idaho), a member of the Senate Foreign Relations Committee, who contended it would usurp the rights of states to tax as they please.

Before the Senate voted to reject the treaty in the 49 to 32 vote, the chamber defeated a Church bid to strike the state tax limitation provision.Both sides agreed the treaty might have passed had the Church measure been approved.

The treaty would have traded the limit on state taxing powers for some $375 million in tax breaks for U.S. investors in the United Kingdom, through a provision allowing Americans to claim special credits for taxes on stock dividends.

Proponents had argued it was unreasonable for states to exceed the limit in the treaty, which would have restricted them to taxing only that portion of a British multinational's business actually done in their state.

Under the compromise, states would be allowed to tax a British multinational on the basis of its total worldwide income, using a special percentage formula.

Only three states - California, Oregon and Alaska - currently use the more liberal taxing procedure, and the total extra revenue they receive as a result is estimated at about $40 million.