Secretary of State Cyrus R. Vance raised the hope yesterday that mediation by Nigeria in the fighting in Zaire can resolve the serious problem that conflict has raised for the Carter administration.

It was learned independently that the private diplomacy now under way over Zaire embraces these significant potentials for U.S. policy in Africa:

If Nigerian mediation between Angola and Zaire can settle the socalled cross-border "invasion" of Zaire by Katangans returning to their original home province, the Carter administration would be spared escalating entanglement in Africa's newest conflict.

According to information reaching Washington, despite claims by Zaire, there has been no actual ground fighting between Zaire's troops and the estimated 1,500 Katangan gerdarmes. Some of Zaire's 3,000 troops sent into the zone of conflict have taken off their uniforms and deserted.

The heaviest rains in 18 years in Shaba Province (formerly Katanga Province) have halted all ground movement in the disputed region. The Zaire government's planes have conducted some bombing, and have displayed some bullet holes in planes to reporters, but President Mobutu Sese Seki is described as most reluctant to move larger forces into the region and risk trouble elsewhere in the country.

Zaire is short of ammunition, fuel and other military supplies, which it is urging the United State to replace in addition to the two planeloads of American equipment already dispatched to Zaire. Sending more U.S. supplies risks an escalation of congressional criticism, which so far has been limited.

If Nigeria is successful diplomatically in mediating the conflict, Angola, in turn, would expect the United State to move toward normalizing relations with it. At present this path is blocked, because of the continuing presence of some 10,000 to 15,000 Cuban troops in Angola. Improving U.S. relations with Angola also could help to improve U.S. relations with Cuba.

Nigeria, which has had strained relations with the United States until very recently, is display its prowess as peacemaker in Africa. Nigeria seeks recognitions as a central power in the continent.

Vance, in public testimony before the Senate Foreign Relations Committee yesterday, limited himself to saying that "Nigeria has agreed to assist in seeing if they can help in the resolution of the problem." He said, "I am happy to say I am encouraged today by what is happening in that area."

The United States, Vance stressed, believes that the problems of Zaire can best be settled by the Africans themselves.

Asked after the hearing what the United States is going to do about the pending "request for ammunition from Zaire," Vance said American policy at the moment is concentrated on trying to do something about "a political solition there." Vance said hopefully, "It may be unnecessary" to send further emergency military supplies.

State Department spokeman Frederick Z. Brown acknowledged yesterday that Vance met Monday with Nigerian Foreign Minister Joseph Garba, at Garba's request.

Sen. Dick Clark (D-Iowa), chairman of the Senate Foreign Relations Subcommittee on Africa, said yesterday that "this Nigerian initiative is extremely important."

"It would be a very poor policy," said Clark, "for us to become anymore deeply involved in Zaire through a (further) shipment of supplies, until we see what this Nigerian don't think we want any more superpower involvement" in Africa's conflicts.