The 35-nation Madrid Conference on Security and Cooperation in Europe broke up for a Christmas recess today with Western delegates satisfied that they had managed to flay the East for the invasion of Afghanistan and violations of human rights but apparently uncertain about the impact of their criticism.

In addition to their doubts about having any effect on the Kremlin, the Western delegates were decidedly pessimistic about the chances of strengthening detente when they reconvene here next month.

U.S. delegation cochairman Max Kampelman stressed to journalists that the West's ability to consistently raise the issues of Afghanistan and Eastern European dissidents during the past six weeks meant that the meetings had been "superbly successful from the point of view of the West." Kampelman added in a press conference: "Washington believes we could not have had a better meeting."

The protests over human rights violations formed the main thrust of the West's speeches over the past six weeks that the conference has been in session. The Soviet Union and its allies charged that such speeches constituted interference in their internal affairs.

In his closing address today to the conference, Kampelman said: "We have laid to rest forever the notion that the way a country treats its own people is its own affair alone, that such treatment is not a proper subject for international discussion, and that human rights has no effect on interstate relations or on international security."

"The first phase of the Madrid conference was devoted almost entirely to reviewing the implementation of the accords on military detente, economic cooperation and humanitarian principales that were agreed on by the 35 signatory states -- the United States, Canada and all European states except Albania -- at the first meeting in Helsinki, Finland in 1975. Delegates agreed that the discussion on implementation had been franker and tougher than at the previous review conference in Belgrade, Yugoslavia, two years ago.

But there was also consensus among Western diplomats that the impact of the recriminations had been limited where the East was concerned. Chief British delegate William Wilberforce said that after six weeks of talking, "serious doubts" remained about the East's commitment to honor the Helsinki bargain.

"Where do we stand?" Wiberforce asked the plenary session today. "It is with great regret that I have to say, 'No further forward, not even the smallest step forward.'"

Nearly 100 proposals ranging from calls to hold a European cultural forum to suggestions that rail fares be reduced for teen-agers have been tabled this week by the 35 delegations for discussion after Christmas. However, the main interest centers on proposals on human rights on a European disarmament conference.

The U.S. delegation has already served notice that it will refuse to commit itself to a NATO-backed French disarmament proposal until after the new administration takes office. But the proposal in any case is unacceptable to Moscow because it calls for confidence-building measures as a precondition to any direct discussion of weapons or arms budgets reduction.