Britain's fledgling Social Democratic Party took a giant first step toward a historic realignment of British politics today with a dramatically strong showing in a closely watched parliamentary by-election in the northern industrial city of Warrington.

The Social Democratic candidate, former Labor Party Cabinet minister Roy Jenkins, greatly exceeded projections by analysts and opinion polls in winning more than 42 percent of the vote to finish a close second in what had been one of the safest Labor constituencies in the country.

The expected winner, Douglas Hoyle, a union leader and left-wing member of the ideologically torn Labor Party's National Executive, held just over 48 percent of the vote, down from the 62 percent won in 1979 by the more moderate Labor member of Parliament who recently had vacated the seat. It was the smallest winning margin for Labor in Warrington since World War II.

Prime Minister Margaret Thatcher suffered a major political embarrassment when the Conservative candidate, Stan Sorrell, a London bus driver and party worker in Thatcher's own constituency, managed to gain only 7 percent of the vote and lost his election deposit. The Conservatives had won 29 percent of the vote in Warrington during their national election victory in 1979.

With Jenkins' strong showing, the four-month-old Social Democratic Party proved it could draw substantial numbers of votes from both the Conservatives, now burdened by high unemployment and Britain's worst recession in a half century, and Labor, from which Jenkins and most of the Social Democrats' 14 members of Parliament broke away earlier this year. Jenkins himself has now made an effective reentry into British politics after serving five years as chief executive of the European Community in Brussels.

Jenkins was supported strongly in Warrington by the Liberal Party in the first trial of an electoral alliance the Social Democratic and the Liberals hope will be a centrist alternative to the present polarization of British politics caused by Labor's turn to the left and Thatcher's right-wing leadership of the Conservatives. Their next test will be an autumn by-election in the suburban Croydon area south of London for a seat vacated by the death of a Conservative member of Parliament.

If the swing of votes from the Conservatives and Labor to the Social Democrats backed by the Liberals in Warrington were projected nationally, Jenkins said, "We have the prospect of a Social Democratic-Liberal government with an overwhelming majority."

Computer projections agreed with him, assuming a Social Democratic-Liberal alliance. But with Thatcher's Conservatives still holding a large parliamentary majority, the next national election is not likely until late 1983 or early 1984.

Two leading left-wing Labor members of Parliament, Eric Heffer and Neil Kinnock, acknowledged that the results were disappointing for their party and would be analyzed carefully.

Some Labor leaders have predicted more defections to the Social Democrats this autumn if Tony Benn, foremost among the left-wingers working to make the party more militantly socialist, becomes deputy Labor leader by defeating Denis Healey, who wants to maintain support for a mixed economy and full membership in the NATO alliance and the European Community.

Conservative candidate Sorrell blamed his poor showing on Conservatives who, he said, switched to the Social Democrats to try to stop a left-wing Labor candidate from representing the traditionally conservative but loyally Labor working-class community in Parliament.

The collapse of the Conservative vote in Warrington confirmed the fears of some Conservative members of Parliament, including Cabinet ministers, who dislike Thatcher's harsh economic policies and believe they make the party very vulnerable to defeat in the next national election.

The Social Democrats and Liberals are undecided about who should contest the Croydon by-election this autumn, although it is the Liberals' turn because Jenkins ran in Warrington. This is still an important impediment to complete electoral cooperation between the two parties. Long-established Liberal hopefuls would have to give way to Social Democrats in many places if the two parties divided up the country's constituencies.

So far, they have formally agreed only on a vague statement of principles. They both support continued British membership in the Common Market and NATO, decentralization of government power to Britain's regions, and a proportional representation voting system that would give third parties larger representation in Parliament.

In Warrington, Jenkins also presented a detailed Social Democratic-Liberal plan to create a million new jobs with the kind of large new investment of government money in the economy opposed by Thatcher but without the further nationalization of industry advocated by Labor.