BUDAPEST -- Hungary, the East Bloc's pioneer in reinventing the socialist economic system, is beginning to emerge as a testing ground for a riskier and more problematic task: the curtailment of totalitarian rule by the communist party.

Since Mikhail Gorbachev took power in the Soviet Union three years ago, Hungary's once daring dismantling of state economic management in favor of market methods has been established as a new orthodoxy among East Bloc economic managers.

Similarly, the party-directed style of gradual reform, employed here for 31 years by communist leader Janos Kadar, has seemed a precursor of what Gorbachev has sought to stimulate at home and in East Europe: "radical reform" that invites popular support yet is strictly controlled from above.

Yet as Moscow has embraced the Hungarian policy equation, its foundation has crumbled here to the point of collapse. Spurred by persistent economic stagnation and the atmosphere of openness encouraged by Gorbachev, a growing part of the Hungarian intelligentsia and some party elites have concluded that a basic rebuilding of the political system is necessary to preserve the country's stability.

The result is a process of debate, political mobilization and experimentation that is testing the limits of democratization under Gorbachev.

"The problem is how we can create possibilities for pluralism without a multiparty system," said Rezso Nyers, a communist Socialist Workers' Party Central Committee member who helped begin Hungary's reforms in the 1960s. "We must rethink the role of the party and the government and find solutions that change the structures we have had until now."

The search for change is similar to that in nearby Poland, the other East European country that has embraced the liberalizing ethic of the Gorbachev era. But while Poland remains polarized politically between a relatively isolated ruling party and opposition-dominated intelligentsia, Hungary is seeing the outlines of a potentially effective pro-reform coalition of dissidents, moderate intellectuals and liberal communists, including prominent Central Committee members.

In both countries, two central issues are beginning to dominate the liberalization debate. One is whether and to what extent the communist party can withdraw from part of its dominant role, yielding to a government that would follow the party's general lead but be a separate entity also responsible to an elected parliament.

While such a reform has also been endorsed by Gorbachev for the Soviet Union, the second issue is more sensitive: whether the party can coexist with social groups and mass movements outside its control. These independent organizations, reform proponents say, must play a role in preparing and implementing new policies, particularly the economic austerity measures necessary in both Hungary and Poland.

For now, the odds appear to be against major political reforms in either country. Conservative leaders of both the Polish and Hungarian parties are strongly resisting change, and Gorbachev appears to resist the notion that ruling communists should share the political initiative. "Only the party," he declared in a 1986 speech summing up the "lessons" of the Solidarity independent union in Poland, "can organize and direct the energy of the people."

The debates in both countries continue, fed by strong pressures inside and outside the ruling parties. In Hungary, the process is directed in part by a preliminary party program of political reform now being discussed in party cells and the prospect of a special party conference between now and June to consider the platform.

Party spokesmen say the program, which has not been published, contains a range of innovative measures, including steps to allow for the expression of minority views inside the party, to make party cells in other social organizations autonomous, and to give more independence to the government by ending the current practice by which a shadow party bureaucracy dictates decisions to government ministries.

Other party sources argue that the program is too modest and too vague, and party cell debates have reflected this view.

By itself, the party might not make much progress. But the usual political balance here has been changed somewhat by the emergence during the last year of strong independent groups of intellectuals and technocrats demanding reform.

At the center of the new trend is the Democratic Forum, a loosely organized movement founded last September for the purpose of promoting open, uncensored debates on key issues and forging a dialogue between proreform party elites, independent experts and opposition intellectuals.

Inspired by a circle of writers and intellectuals, known as "populists" for their interest in nationalism, the forum is imbued with the populists' traditional strategy of seeking accommodation rather than conflict with authorities. At the same time, the movement has insisted on political and organizational independence.

A session last month at a theater in Budapest crammed with more than 400 participants, produced an impassioned discussion of political reform and a statement calling for an increase in the power of the elected Hungarian parliament. Speakers invoked the names of Poland's Solidarity and Hungary's 1956 revolution leader Imre Nagy during the debate. One orator demanded the withdrawal of Soviet troops from Hungary.

Leaders of the forum, while renouncing such extreme positions, are pressing for a strict legal definition of the limits of the party's powers.

"The party's position is contradictory. It says it wants to yield to the government and parliament, but it also says it wants the 'leading role,' " said Csaba Kiss, a literary critic and forum activist. "What we need to do is define in the constitution the concrete range of prerogatives {the party} has and not allow it to be above the law."

The attitude of party authorities toward the movement has been ambivalent.

The leader of the communists' liberal wing, Imre Pozsgay, was the guest of honor at the forum's first meeting and published its first declaration in a newspaper he controls as leader of the communist-sponsored Patriotic People's Front.

However, Pozsgay and other prominent activists were prevented by the party Central Committee from attending the most recent meeting. The turnaround apparently occurred in part because leading dissidents had been invited and because the party wished to avoid the appearance that it had lost control over the debate on reform.

Party activists here suggest that the uncertain policy toward the forum reflects the communists' internal debate on reform, a struggle complicated by the competition to succeed Kadar. The 76-year-old leader is being nudged toward retirement by younger rivals, such as Prime Minister Karoly Grosz. Party sources say it is possible Kadar will announce his partial or full withdrawal from active leadership at the party conference.

Forum activists say the party will not find a new formula for stability that excludes them.

"The time has not yet arrived when we can determine what kind of change can be made in the long run," said Gyula Fekete, the vice chairman of the Hungarian Writers' Union and a chief forum organizer. "But for now, the economic and social situation is such that, without the sharing of responsibility between the authorities and society, it will be impossible to go on."