East and West reached a provisional final agreement at the European Security Conference today, paving the way for adoption of the Madrid accords updating the 1975 Helsinki document on military security, cooperation and human rights in Europe.

With 34 of the 35 participating states in agreement, a threatened filibuster by Malta, which is insisting that the final document include a reference to a Mediterranean security conference, appeared to be the only remaining barrier to setting dates for a concluding session.

All European countries plus the United States and Canada are participants in the conference, which opened 31 months ago.

Besides Malta's recalcitrance, the chief problem at today's meeting was reconciliation of language.

Max M. Kampelman, chief U.S. delegate at the conference, held bilateral talks with his Soviet counterpart, Anatoly Kovaliov, in the past 24 hours as well as with members of neutral delegations from Austria and Switzerland to resolve outstanding East-West divisions.

A traditional feature of meetings associated with the Helsinki accords has been a filibuster by Malta to push for its Mediterranean projects.

Delegates said plenary sessions next week would seek to reconcile Malta to the fact that it can expect a unanimous rejection of the proposal for a Mediterranean security conference.

Other conference participants view a Mediterranean conference as falling outside the scope of the European detente talks because it would involve North Africa.

Delegates to the Madrid conference are expected to meet next week to agree on dates for a final closing session to be attended by foreign ministers of the participating countries. This meeting, probably late this month or early next, would permit Secretary of State George P. Shultz to meet with Soviet Foreign Minister Andrei A. Gromyko.

In last-minute negotiations yesterday and today the United States gained assurances on key points related to human rights. These safeguards allowed the West to claim that the Madrid accords represented what Kampelman said was "a significant Western accomplishment."

Informal acceptance of the draft Madrid accords, framed by the neutral European nations and by host nation Spain, had been reached a week ago, but U.S. misgivings had prompted Kampelman to return to Washington for consultations with the White House, the State Department and members of Congress.

On his return to Madrid yesterday Kampelman said that he had the "green light" to negotiate and that while President Reagan "liked the thrust" of the Madrid talks, further clarification was necessary. The principal issue concerned a follow-up meeting to discuss "human contacts"--a term covering family reunifications, binational marriages, visas and other emotional issues.

Diplomats said the cornerstone of today's agreement was Kampelman's belief that the proposed human contacts meeting would not be linked to what the East Bloc terms "progress on detente" and that the meeting would not be a "second class" forum, although the mandate for it forms part of an appendix to the proposed Madrid concluding document.

Kampelman also got assurances that the decision to hold this meeting would be published along with the rest of the Madrid documents, the diplomats added. The meeting is scheduled to be held in Bern, Switzerland, in 1986.

The Bern human contacts meeting has been the final battleground for agreement to end the protracted Madrid conference. This has meant that, in its final stages, the Madrid negotiations have focused almost exclusively on the humanitarian aspects of detente that have consistently been the Western priority throughout the Helsinki process.

Western diplomats said a proposal for a human contacts follow-up meeting had been proposed by the United States in December 1980, shortly after the conference began, but the suggestion met with a head-on refusal by the Soviet Union. The final acceptance by Moscow of this key forum was viewed as a singular Western achievement.

The Bern forum is one of seven scheduled follow-up meetings.

Another is the first stage of a conference on disarmament in Europe, which will open in Stockholm in January 1984. It will address itself solely to so-called confidence-building measures such as guidelines for notification of troop movements.

Diplomats said that the Madrid accords will include agreement by the participating states to "encourage genuine efforts to implement the Helsinki Final Act" and acknowledgement that "governments, institutions, organizations and persons have a role to play" in that endeavor.

This was seen as an implicit recognition of the rights of dissidents, although diplomats conceded that prominent human rights activists in the Soviet Union remained in prison or in exile throughout the Madrid negotiations.