The Foreign Office today instructed its envoy in Nigeria to lodge a formal protest against Nigeria's nationalization of British Petroleum interests and to request that it "reconsider" the decision.

With top British officials attending the Commonwealth conference in Zambia, senior officials here reacted angrily to Nigeria's attempt to use economic pressure to force changes in London's policy toward southern Africa.

"this is very serious", a senior minister in Prime Minister Margaret Thatcher's Conservative Cabinet said. "It is a question of whether Britain is going to be pushed around."

The Nigerian government said yesterday that it acted against British Petroleum only in retaliation for the oil company's decision, approved by the British government, to engage in swap arrangements that would send more crude oil to South Africa to replace oil it no longer gets from Iran. Nigerian officials said that this was not the beginning of a wave of other nationalizations by the Nigerian government.

But timing of the nationalization on the eve of the Commonwealth conference discussions about Rhodesia left officials here convinced that it also was a fulfillment of Nigeria's warnings to British diplomats and businessmen that it would use economic pressure to stop Britain from recognizing Zimbabwe-Rhodesia.

The United States, which has more than $1 billion in investments in Nigeria and exports about $1 billion worth of goods to Nigeria each year, also has been sent signals warning of possible retaliation in case of U.S. recognition of the new black-led government of Zimbabwe-Rhodesia.

After issuing the warnings in May, the Nigerian government excluded British companies from bidding on a $200 million port construction contract and told the British they might be frozen out of all future government contracts.

In July, a group of British businessmen, bankers and contractors doing business in Nigeria, including a representative of British Petroleum, met with a Foreign Office representative here to warn the Thatcher's Rhodesian policy put all of Britain's billions of dollars in investments and trade in Nigeria, black Africa's leading economic power, "very much in jeopardy."

The group also has reportedly been lobbying Conservative members of Parliament, some of whom have moderated their insistence on prompt recognition of the new Zimbabwe-Rhodesian government and the lifting of Britain's economic sanctions against Rhodesia.

Nigeria also has large amount of British currency in banks in Lagos and London. Sudden sales of large amounts of this sterling could sap some of the recent strength of the British pound in the world's money markets. But rumors that Nigerian sales of the pound might have helped cause its sudden six-cent drop to $2.25 yesterday were dismissed by government officials here today as unfounded. In any event, the pound's traded value rose again today to $2.27.

The loss of the more than 300,000 barrnls of crude oil that British Petroleum was taking out of Nigeria each day is a relatively manageable, if unwelcome, problem for British because of its steadily increasing harvest of oil from the North Sea. All of British Petroleum's Nigerian oil was being sold to other Common Market countries.

But the loss of more than 9 percent of its total crude supply was a serious blow to British Petroleum, 51 percent of which is owned by the British government. It has suffered production cuts in Iran after the revolution there and faces a planned cutback by its third major supplier, Kuwait, beginning next year.

Nigeria promised yesterday to compensate British Petroleum for the nationalization of its 20 percent interest in a major oil production partnership there and its 60 percent interest in an oil products marketing firm. But the confidence of private investors in British Petroleum has been shaken and its stock price fell sharply today.

Through swap arrangements approved by the British government this year, British Petroleum was to sell some North Sea Oil to the European Common Market in return for an equivalent amount of oil to be sold to South Africa. The oil going to South Africa, according to British Petroleum and British government officials, would not come from OPEC nations who banned shipment of their oil to South Africa and would not be Nigerian oil, as the Nigerian government charged yesterday.

The existence of these government-approved swap arrangements, which nevertheless provide new oil for South Africa, as publicly revealed here last month by Britain's former forergn secretary, David Owen, who is now energy spokesmen for the opposition Labor Party in Parliament. He raised in correspondence with his successor, Lord Carrington.

A british lobby group, the Anti-apartheid Movement, said today that it also had supplied detailed information about the swap arrangements to the Nigerian government as part of its efforts to end apartheid in South Africa.

The Thatcher government has formally declared its abhorrence of apartheid, but favors encouraging the white minority South African government to make changes rather than constantly critizing it or taking economic or political action against it.

Britain's trade with South Africa is half as large as its trade with black Africa.

This and Thatcher's determination to find a way to restore Rhodesia to legal independence through some modification of the new Salisbury government, rather than insisting on a settlement, between it and the Patriotic Front rebels backed by black Africa, are the policies that Nigeria is trying to force Britain to alter.