The expulsion from India last week of three key secessionist leaders by Prime Minister Rajiv Gandhi to try to force Tamil guerrillas back to the bargaining table with the Sri Lankan government has had little success.

The action has only sown confusion, disorganization and fear in the guerrilla ranks, driven one key rebel leader underground and triggered potentially explosive unrest among the rebel's Indian supporters in the southeastern state of Tamil Nadu, where 45 million Tamils reside.

Gandhi had finally gained greater cooperation from the government of Sri Lankan President Junius Jayewardene in talks begun several weeks ago in the Himalayan kingdom of Bhutan when he found the Tamil guerrillas recalcitrant.

The prime minister's problems were highlighted in the continued refusal of key rebels to meet with him in New Delhi and in a threatened demonstration in support of the deported Tamil leaders that has mobilized police forces throughout this state.

Although Gandhi today reversed the six-day effort to deport S.C. Chandrahasan, a leading Sri Lankan secessionist, police sources revealed that 2,500 persons had been taken into protective custody here in the past 24 hours to stem what the Indians call a "rail roko" demonstration Friday. A "rail roko" calls for shutting down all railroads in the state by having demonstrators mass on key track junctions.

Sri Lanka, an island nation 18 miles across the Palk Strait from southeastern India, has a population of 15 million, of which 12 percent are Tamils, who are usually Hindu. Most of the Sinhalese majority are Buddhists.

The Tamils, who maintain they are exploited and repressed in their nation by the Sinhalese, began to revolt in the late 1970s.

Indira Gandhi's government often turned a blind eye as Tamil guerrillas were given sanctuary, military training and, according to some Indian sources, political and economic support in and around Madras, Tamil Nadu's capital.

Rajiv Gandhi, Indira's son, has followed a policy of seeking negotiated border stability. He has been curbing the Tamil guerrillas' actions here and, through India's secret police, the so-called Research and Analysis Wing of the prime minister's office, pressuring them to accept the sort of compromise autonomy within Sri Lanka that he is trying to convince President Jayewardene is the price of peace.

The talks in Bhutan having foundered, Gandhi has managed to keep the chief Sri Lankan negotiator, Hector Jayewardene, the president's brother, in New Delhi to try convince him of the need for a more concrete proposal to the rebels. He has repeatedly summoned the chief rebel negotiators, only to be told that some of the key figures are not in Madras or even "reachable."

Indian and Tamil sources here who preferred not to be quoted by name said this was a stalling tactic by the rebels, who fear that if they go to New Delhi, they will be subject to further arm-twisting.

In reaction to the refusal to come, Gandhi ordered the expulsion of Chandrahasan; Anton Balashingham, a key leader in the Liberation Tigers of Tamil Eelam, the rebels' biggest guerrilla group, and N. Satyendra of the Tamil Eelam Liberation Organization, who had angered Indian Foreign Secretary Romesh Bhandari in Bhutan by talking back to him.

Although New Delhi relented with Chandrahasan because it was unable to find a haven for him, rebels, Indians close to them and other analysts here said that by removing three of the rebels' brightest people and best strategists, the government had left the rebels afraid, confused and unwilling, perhaps unable, to make the sort of hard decisions Gandhi is pressing for.

"These are often just boys, unsophisticated young men in their twenties, mostly ex-students, who need the sort of strategic thinking that a lawyer like a Chandrahasan or Balashingham can provide," said Anita Pratap, a reporter here for a Calcutta daily, The Telegraph, who specializes in the Tamil rebel movement. "Without them, instead of talking, they will choose to stall, making no decisions."

That much seemed evident today when Gandhi's representatives met with the leaders of three of the four guerrilla groups that went to the talks as a coalition. These leaders assured the central government officials they would go to New Delhi once Velupillai Pirabhakaran, the charismatic leader of the Liberation Tigers, the fourth rebel group, could be found.

Pirabhakaran, according to all reports, has gone underground; he has eluded a massive police manhunt in Tamil Nadu the past five days.

Although the rebels know they cannot continue without Indian sanctuaries and thus ultimately will have to deal with Gandhi, some Indian Tamils here see the issue as one to embarrass Gandhi and the local ruling party, whose leader, the former movie idol M.G. Ramachandran, has yet to recover his speech from a stroke.

Already one veteran of the opposition, former chief minister M. Karunanithi, has sought to appropriate the issue of the Sri Lankan Tamils for his own. As he warned at one public rally for the three deportees last week in the center of town: "If ever the Liberation Tigers are deported from Tamil Nadu, we will set up our own training camps, and the youths who will be trained here will not be from Sri Lanka but the youths of Tamil Nadu."

Although most people here tended to treat such threats as little more than political bombast, the police mobilization for a lightning sweep of preventive arrests in the past 24 hours indicated the government's concern.