The controversial U.S. British tax treaty won formal approval in the Senate Foreign Relations Committee yesterday, but the measure still faces a tough challenge when it reaches the Senate floor.

The panel approved the treaty by voice vote after defeating, 10 to 5, a move by committee liberals to object formally to a provision that would restrict the way states tax British-owned corporations.

The liberals are expected to revive the issue when the treaty comes up for a floor vote, presumably after the Senate completes work on the pending Panama Canal treaty. The tax treaty has been delayed since mid-1977.

The liberals seemed heartened by the fact that a third of those voting on the panel supported their position. Lobbyists for state tax commissioners opposing the section on state taxation say they now have 30 senators on their side.

Nevertheless, while the outlook isn't at all certain, Capitol Hill observers say the liberals still have an uphill fight. Both sides concede that the issue is a difficult one for which to drum up much public support.

The tax treaty was negotiated last year, but has been bogged down in committee since than, in part because of possible conflicts with the Panama Canal treaty and in part because of opposition from some state tax officials.

Although the provision would result in revenue lossed only for California, Alaska and Oregon, tax commissioners from several states gave opposed the treaty - mostly to protest the "priciple" of controlling the tax laws of their states.

Liberals also are displeased with a second provision in the tax treaty that would allow U.S. oil companies doing business in Britain to treat as a U.S. foreign tax credit the payments they make under British oil tax laws.

Conservatives are pushing for approval of the pact, since it provides large concessions for U.S.-owned firms, whcih already are heavily taxed in the U.K. The Treasury hopes the treaty will serve as a precedent for other agreements.