Nigeria nationalized the British share in a jointly operated oil company today in what appeared to be the start of a major drive to force Britain to change its Rhodesia policy.

The move came on the eve of the 41-nation Commonwealth conference in this southern African capital. Britain and the 14 African nations in the organization of former British colonies have been girding for weeks for a major clash over Conservative Prime Minister Margaret Thatcher's plans to lift economic sanctions against the breakaway British colony now known as Zimbabwe-Rhodesia.

The British delegation to the Lusaka conference, headed by Thatcher, called the Nigerian action an "extra-ordinary move." A statement indicated that the timing was designed to exert pressure on Britain to change its stance on lifting sanctions and recognizing the black-led government of Prime Minister Abel Muzorewa in Salisbury.

The government rejected what it said was a Nigerian explanation linking the move to the sale of British oil to South Africa.

"Taking into account the timming of this extraordinary move on the eve of the Commonwealth meeting, the British government can only suppose that the real reason for the Nigerian action is something entirely different," the statement said.

The takeover involves the 20 percent share of British Petroleum in the combined Nigerian-Shell-BP oil company. Nigeria already owned 60 percent of the company with Shell and BP evenly sharing the remainder.

Maj. Gen. H. E. O. Adefope, the Nigerian foreign minister, declined to comment on his government's move and said he would issue a statement after the opening of the conference Wednesday.

Nigeria has threatened Britain and the United States with economic retaliation if they lift sanctions against the Muzorewa government. Nigeria s the second largest exporter of oil to the United States and has surpassed South Africa as Britain's largest trading parter.

With its oil wealth and burgeoning economy, Nigeria is regarded as the weapon in attemping to prevent the West from accepting the Zimbabwe-Rhodesia government, which guarantees the 250,000 whites a privileged role in the population of 7 million.

It appeared that Nigeria may have begun a broad attack on the British economy. The pound dropped six cents to $2.25 on European exchange markets and some dealers blamed Nigeria, saying it was offering to sell upwards of 500 million pounds.

The pound dropped further on the New York market after the announcement of the BP takeover, which came too late to affect the rate in Europe.

The Nigerian announcement said BP had been told that it would be compensated, but there was no immediate indication of the amount.

Even though today's Nigerian actions were limited to Britain, the fallout of the nationalization could have an impact in the United States. Congress has been pressuring President Carter to lift sanctions against Zimbabwe-Rhodesia, a move that could conceivably lead to similar Nigerian action against the United States.

Although the United States has no formal representation at the Commonwealth conference, the Carter administration's policy of seeking improved relations with black Africa could be a major casualty of the meeting if Africa and Britain decide to go their separate ways over Rhodesian policy. If Britain lifts sanctions, it might be difficult for the politically weakened U.S. administration to resist moves in Congress to follow suit, observers here said.

In the last two days the Zambian press and one of the guerrilla factions fighting the six-year-old Rhodesian guerrilla war have called Thatcher a racist, indicating the growing African hostility toward her policies. In addition, Zambian President Kenneth Kaunda pointedly warned her against lifting sanctions in an interview published as she arrived yesterday.

British officials said, however that Thatcher was unfazed by the attacks.

Even visiting Queen Elizabeth was subjected to an anti-Rhodesian lecture by the major of Lusaka for which the government later apologized.

Holding the Commonwealth conference in Lusaka has multiplied Thatcher's problems because one of the guerrilla factions is headquartered here. The Rhodesian military has frequently attacked guerilla targets in Zambia over the last year, sometimes killing Zambians in the process.

When Thatcher and other Commonwealth leaders dine at Kaunda's residence Wednesday, they will be less than half a mile from guerrilla leader Joshua Nkomo's house, which helicopter-borne Rhodesian commandos destroyed in April.

Thatcher, clearly fearing that she could wind up with a diplomatic disaster in her first foray into Africa, has recently toned down her strong support for Muzorewa although still marking it clear that her aim is to lift sanctions.

Lord Carrington, the British foreign secretary who arrived with Thatcher last night, has said that the Lusaka conference will conclude British consultations, after which Britain will submit proposals to make the Zimbabwe-Rhodesia government acceptable to the international community.

Although not providing details, one government minister said Britain plans to put together "a bundle of pluses" to justify lifting sanctions and conferring recognition.

The "pluses" would likely include removal of former prime minister Ian Smith from the government, and dilution of some of the privileges the whites retain under the constitution.

Britain apparently hopes through such proposals to win enough moderate African support to be able to go ahead and lift sanctions.

One British official cautiously said, "We are casing the joint with a view for the Salisbury government.

He said at this point that Britain was not promoting any solution. "In fact," he said, "I would call this market research as a prelude to promotion." CAPTION: Picture, MARGARET THATCHER