In the atmosphere of superpower vituperation created by the downing of the Korean Air Lines jet, the windup last week of the 35-nation conference devoted to the means of improving cooperation and security in Europe had a bizarre quality. Rarely can a major diplomatic conference have turned out less the way it was supposed to.

So irrelevant had the official closing document become--despite some declarations on religious freedom and the rights of trade unions applicable to the communist East--that the harried conference secretariat did not pass it out.

But then, the Conference on Security and Cooperation in Europe has been a strange creature from its beginnings a decade ago, little known, even in Washington, and less understood.

The Soviets initially pressed for a conference to affirm the status quo in Europe. The West only agreed when it was seen as a way of bringing what a senior Swiss official calls the "human dimension" onto the bargaining table of East-West relations--the flow of people and ideas across the ideological frontiers of Europe's postwar split.

U.S. critics on the right wing have reviled it as a sell-out to the Soviet Bloc because it gives de facto recognition to Kremlin dominance in Eastern Europe. Top American figures including Henry A. Kissinger and Gerald Ford and, last week, Secretary of State George P. Shultz have been urged to repudiate the whole business and pull out.

At its best, the "Helsinki process," as it is known for the site of its original meeting place, is portrayed by detractors as a vehicle for diplomatic yakety-yak in which the Soviets can be chastised for their wrongdoing in places such as Afghanistan and Poland. It is true that positive, concrete results of the deliberations--more ef-ficient methods for expediting business contacts, for instance--are hard to pin down.

Yet, for all the abuse, the Conference on Security and Cooperation in Europe maintains a distinctive place in the American diplomatic arsenal. It is the only international forum where the U.S. delegation consists of a joint commission of Congress and the executive branch. The commission, under staff director R. Spencer Oliver, churns out detailed reports on Soviet and Eastern European compliance (or in most cases, noncompliance) with a series of pledges on issues such as emigration, the free flow of information and advance warning of military maneuvers.

At any given time, Oliver said last week, gratified to be asked a question about the conference rather than the Korean plane, the commission has several thousand cases on its books, usually of people seeking to emigrate. In that sense, the delegation acts as "ombudsman" for reunification of families, an issue of some consequence in the United States as a nation of immigrants.

Moreover, the Helsinski final act signed in 1975, the Belgrade review conference that began in 1977 and the Madrid session that opened in 1980 have encouraged the peoples of Eastern Europe--Jewish "refusedniks" in the Soviet Union, Charter 77 members in Czechoslovakia and Solidarity trade union supporters in Poland--because the West has a vehicle for speaking directly to the Soviets on their behalf.

The Soviets routinely violate their pledges on human rights, said one diplomat who has been with the conference from the beginning, "but at least we have a text against which we can hold them accountable." Even out of the rubble of last week's ceremonial debacle, there remain plans for two conferences on human rights and human contacts for 1985-86, as well as a European disarmament conference scheduled for next year.

But in the years since the final act was signed, matters have gotten much worse in the areas of greatest U.S. interest. Emigration from the Soviet Union has slowed to a trickle, and dissidents in the U.S.S.R. and Eastern Europe have been largely dispersed through prison or exile.

Three years ago, when the Madrid conference convened, enthusiastic western supporters of the dissidents converged on the conference center to lobby delegates. Last week few were on hand. The best known was Avital Scharansky, wife of imprisoned Jewish activist Anatoly Scharansky, who denounced the conference results as empty.

The biggest demonstration outside the cavernous Palace of Congresses was by Koreans.

But defenders of the conference say the failures of East-West relations are not the fault of the Helsinki process. If the detente spirit of the mid-1970s had survived, they say, the record shows that the terms of the various accords would have provided a useful framework for broader East-West exchanges on all fronts, from movies to marriages.

In other words, this diplomatic institution--as the Soviet-American recriminations in recent days underscored--is a captive of general international relations, especially between the superpowers. When the relations are bad, as of late, the conference has shown it cannot improve them.

But there is also no conclusive evidence that through its major political contribution, keeping the defense of human rights in the forefront of political awareness, the conference is making matters worse.