An estimated 8 million voters in this once socialist-oriented island in the Indian Ocean will put to a test Wednesday the question of whether a Western-style, free-wheeling, capitalist economic system can work in the Third World.

In Sri Lanka's first presidential election, President Junius R. Jayewardene, who five years ago rewrote the Constitution and elevated himself from the post of prime minister to an executive presidency, is asking voters to continue, amid spiraling inflation and an overheated economy, the Reaganesque, market-oriented, supply-side policies he has resolutely pursued.

He is considered to have a slight edge over Freedom Party candidate Hector Kobbekaduwa, a stand-in for former prime minister Mrs. Sirimavo Bandaranaike, but the balloting could be so close that the results would hinge on a complicated second-preference tallying system crafted by Jayewardene when he first came to power.

There are four other candidates, but the main race is between Jayewardene and Kobbekaduwa.

Anticipating an outbreak of the kind of violence that traditionally accompanies Sri Lankan elections, Jayewardene has deployed Army troops and security forces throughout the island and has made plans to impose emergency restrictions the day after the balloting. Such emergencies are usually declared after elections.

Closely following the election are Sri Lanka's Western benefactors, particularly the United States, which with $100 million a year provides the island with more U.S. assistance per capita than any other country in Asia. Economic aid from the West totals nearly $1 billion annually.

The Reagan administration has lavished praise on the free-wheeling, market-oriented economic system fashioned by Jayewardene -- who once was derisively referred to by socialists as "Yankee Dickie" -- and has regarded it as a model for economic development in the Third World.

But Sri Lankans have turned out every incumbent government that has run for reelection, and there has been a feeling among many voters that Jayewardene's high-caste segment of the country's 15 million people has exacerbated economic disparity and fostered corruption.

Resentment over high inflation has compounded the restiveness and given momentum to the movement to return Sri Lanka to the protectionist, austerely socialist economic system that Bandaranaike fashioned after the Indian model when she was prime minister from 1970 to 1977.

Because she was found guilty of abuse of power during her premiership, Bandaranaike has been barred by the courts from seeking public office for seven years. Although she is prohibited even from making political statements, her presence has been felt in the background of the campaign.

Her photograph appears on Freedom Party billboards, and Kobbekaduwa, a former agriculture minister, has said that if elected, he will restore Bandaranaike's civil rights, dissolve Parliament and revert to the 1972 Constitution, which presumably would put Bandaranaike back on the road to power.

While Bandaranaike herself remains an election issue, the key issue is Jayewardene's bold, unabashedly capitalistic policies, which have made Sri Lanka a haven for free enterprise and an anomaly among developing countries.

When his United National Party swept away the Freedom Party in 1977 and he reformed the Constitution, Jayewardene immediately began implementing his vision of a Singapore-like market economy, dismantling price controls, liberalizing imports and creating a free-trade zone on the outskirts of the capital.

Foreign capital began to flow into Sri Lanka, and the country's 3 percent growth rate doubled, making it overnight the darling of the International Monetary Fund and the World Bank.

Boosted by the $2 billion Mahaweli Hydroelectric and Irrigation Project and an ambitious housing program that aims to build 133,000 new homes, employment soared, with the jobless rate dropping from 20 to 10 percent and per capita income climbing to nearly $300 a year.

Shops where housewives once lined up for hours to buy scarce supplies of rice and other staples suddenly bulged with imported consumer goods, and duty-free stores were built for the 100,000 Sri Lankan workers who took jobs in Persian Gulf states and returned with hard currency.

But, while unleashing frenzied economic development and foreign investment, Jayewardene's policies exacted a price.

Government subsidies on essential foods were sharply reduced to help offset a mounting budget deficit, and deep cuts were made in the social programs that during the Bandaranaike years had made Sri Lanka the envy of the Third World with a nearly 90 percent literacy rate and a life expectancy of 68 years.

Fueled by the overheated economy, the cost of living rate climbed erratically to a 40 percent annual rate before recently taking a downturn to less than half that figure; the budget deficit grew five-fold and had to be financed by printing new money, which in turn weakened the rupee; the balance-of-payments deficit ballooned to $1 billion, an enormous shortfall for a country this size, despite foreign aid.

Jayewardene says he is sensitive to the effects his policies are having on some segments of Sri Lankan society, but he maintains that while the price is being paid now, the payoff is down the road in the form of more jobs and a better standard of living generated by more economic development.

Jayewardene, a 76-year-old patrician whose spryness belies his tall, gaunt appearance, is a pragmatist who likes to deride his leftist opponents as "fuzzy Marxist ideologues" and "Trotskyites."

Addressing a crowd of 8,000 mostly middle-class backers during a late monsoon downpour here this week, Jayewardene recited a litany of communist or socialist Third World countries plagued by political instability and economic chaos, and warned that such a fate lies ahead for Sri Lanka if it reverts to the stagnant, nationalized economy of Bandaranaike.

But he has angrily denied opposition charges that he has turned Sri Lanka into a client of the United States, saying that he is committed to a policy of nonalignment and is vehemently opposed to any foreign naval presence in the Indian Ocean. While ruling out a U.S. naval base at the port of Trincomalee, Jayewardene, however, has encouraged U.S. and Soviet port visits there to generate more foreign exchange for Sri Lanka.

At no time during the campaign, which for the most part has been low-key and lacking the emotionalism that characterizes most Sri Lankan elections, has Jayewardene hinted at modifying his conservative fiscal policies.

"It took him 43 years in politics to climb that greasy pole and reach his vision. He's not going to climb down now," said a Sri Lankan acquaintance of the president. He added, "We've gone too far with our economic development to go back now."

Political analysts here say that Jayewardene advanced the election, which normally would be held in 1984, not only because the opposition parties are in disarray and his chances are enhanced, but because he concluded that Sri Lanka's economy is not likely to continue improving as it has been, and probably will slump before it picks up in the long range.

Although there are no reliable polls here, indications are that the vote will be close, with considerable attention given to the vote of the Tamils of the north, who are Hindus of Indian origin. Communal violence between the Sinhalese majority and the Tamils has been frequent in Sri Lanka, resulting in three national emergencies since Jayewardene took office. Some Tamils have called for a separate state, and guerrillas known as Tamil Tigers have occasionally clashed with security forces.

The mainstrem Tamil Party has called for a boycott of the election, but many Tamils are expected to support Jayewardene because they have benefited from his economic policies.

Under an election system modeled after France's, Sri Lankan voters mark their second and third preferences, and if a candidate fails to attain over 50 percent of the vote, the subsequent preferences are added to his total.

Jayewardene's backers say the president is seeking a popular endorsement in anticipation of parliamentary elections next year, in which the United National Party hopes to retain its five-sixths majority. But the contest appears to be closer than that, and most political observers believe that if Jayewardene is given another term, it will be the result of the second-preference system.