Secretary of State Alexander M. Haig Jr., following up on President Reagan's offer of U.S. mediation in the Falkland Islands dispute, conferred yesterday with senior British and Argentine representatives, but U.S. officials cautioned that no formal American proposals had been made.

Argentine Foreign Minister Nicanor Costa Mendez, after meeting with Haig, said the United States "offered us assistance to see if we can solve our problem with Britain." He said he would transmit the offer to his government but added he was "confident" that the two sides would reach "an honorable and just peace" through negotiations. He also said that he would not describe the United States as a mediator in the dispute.

Haig also met separately with British Ambassador Sir Nicholas Henderson and Argentine Ambassador Esteban Takacs in an effort to probe for a possible compromise to avert an armed clash when a British naval task force nears the Falklands in two weeks.

The administration, which regards both countries as friends, is concerned that it could be forced to choose sides and be thrust into a situation that would have serious consequences for U.S. foreign policy goals.

For that reason, Reagan has instructed Haig to look for ways of satisfying Argentina's demand for sovereignty over the Falklands, which it calls the Malvinas, while allaying Britain's concern about the welfare and future status of the 1,800 British subjects who live on the South Atlantic islands.

However, as the statements by the two sides yesterday made clear, herculean work will be required to bridge the tensions and emotions generated by Argentina's seizure of the islands and Britain's stated resolve to regain them at all costs.

Henderson declined to say what he and Haig had discussed. But he insisted that Britain's position remained unyielding and said: "The first course of action is that the aggressors and usurpers should leave the territory to let its people live in a democratic society . . . . You had 52 hostages in Iran . We have 2,000 down there."

Costa Mendez later took exception to this remark, saying the islanders are free to leave and any who do will be compensated for any damages.

In an interview with The Washington Post, Henderson said: "We are ready to talk. But we certainly cannot agree to do so as long as Argentina is occupying the territory by force. You can't start halfway down the line. You have to go back to square one and start again from the situation as it was before Argentina's illegal action."

He said the British government does not think the United States should be asked to take sides on the question of who rightfully owns the Falklands. But, he added in reference to U.S. professions of neutrality, "It would be quite wrong and dangerous if the Americans said they were neutral about questions of aggression and self-determination in the Western Hemisphere."

Britain has taken the position that the United States, by voting in the majority for a U.N. Security Council resolution calling for Argentina to withdraw its forces, has supported it on these points.

But there is always the possibility that if the situation deteriorates, Prime Minister Margaret Thatcher's government, which has been Reagan's staunchest ally in Europe, will call for stronger expressions of American backing.

On the other side, Argentina's military government, which the administration has cultivated as a counter to Cuban influence in Latin America, has been making clear that it intends to maintain its hold on the Falklands and counts on the United States to dissuade Britain from resorting to drastic action. Costa Mendez was expected to underline that point in his meeting with Haig last night.