The opposition Social Democratic Party registered a landslide victory in West Germany's most populous state today in an election that boosted the stature of its moderate state party leader, Johannes Rau, as a leading candidate to challenge Chancellor Helmut Kohl in 1987 national elections.

The Social Democrats, who have controlled North Rhine-Westphalia since 1966, achieved their best result ever in easily winning an absolute majority in the state, where one-third of all West German voters live. The conservative Christian Democrats, behind Kohl's protege Bernhard Worms, fell to their lowest standing in more than three decades.

Kohl is a close political and ideological ally of President Reagan, and the Christian Democrats sought to exploit Reagan's recent visit to the Bitburg military cemetery, which was highly popular among Germans eager to see their country accepted as an equal partner reconciled with former wartime enemies.

But the so-called "Bitburg factor" failed to provide much help for Kohl's party, as Rau effectively transformed the race into a personality contest by contrasting his jaunty, eloquent nature with Worms' colorless, stodgy profile.

An official count completed late today showed the Social Democrats with 52.1 of the vote, an increase of nearly four percentage points over their 1980 performance. The Christian Democrats dropped nearly seven percentage points to 36.5.

The Free Democrats, the junior partner in the coalition with Kohl's Christian Democrats in the federal government, regained their place in the state legislature by clearing, by one percent, the 5 percent hurdle needed for representation. The radical Greens party, which hoped to gain entry by emphasizing their antipollution drive, did more poorly than expected and failed to win any seats.

The Greens, who attracted 4.6 percent of the vote, up from 3 percent five years ago, were hurt badly by a controversial proposal put forward by some party members calling for the abolition of laws that punish adults for having sex with minors.

"The sexuality issue cost us a whole lot of credibility," said Otto Schily, a leading Greens party deputy. "This is politically stormy weather," he said.

In winning another five-year term as state premier, Rau, 54, dodged discussion of the industrial decline and pollution woes that have plagued the Ruhr Valley "rust belt" under Social Democratic control.

Instead, Rau used the campaign to attack Kohl's economic policies, charging that the ruling center-right coalition in Bonn had failed to create new jobs while spawning a class of "new poor" through social welfare cuts.

"I believe that federal politics played a big role," Rau said after the initial vote projections were announced. "Kohl will have to do a lot of thinking about the results."

The chancellor admitted later that "this is without doubt a defeat for the entire party of Christian Democrats . Our policies have meant sacrifices, and many people did not understand this."

With some chagrin, Kohl said "the personality bonus played a big role" in helping the Social Democrats surpass their own expectations at the polls.

Willy Brandt, the Social Democrats' party chairman, described the outcome as proof that "we are a people's party that can win on its own."

Brandt told reporters that the election must be viewed as "a personal defeat for Helmut Kohl" more than 30 months after he took over as chancellor from his Social Democratic predecessor Helmut Schmidt. Brandt later became embroiled in a bitter argument with Kohl during a television program after the chancellor accused him of engaging in "primitive anti-Americanism" in the wake of Reagan's six-day visit here.

"You should be ashamed of yourself, Mr. Chancellor," Brandt retorted. "You have harmed our people with these lies."

Rau's strong triumph has enhanced his position as the party's most likely candidate to oppose Kohl in the next national election. His chief rival is considered to be Oskar Lafontaine, 41, a left-wing party member who also won an absolute majority two months ago when he became premier of the Saarland.

Both men are deemed more charismatic than the current opposition leader, Hans-Jochen Vogel, whose lackluster campaign against Kohl in the March 1983 election contributed to one of the Social Democrats' most sweeping defeats.

Lafontaine, a fiery orator who ardently opposes nuclear weapons and wants to remove West Germany from NATO's military command, is said to enjoy almost paternalistic support from Brandt. The former chancellor and Nobel Peace Prize laureate says Lafontaine could forge "a majority of the left" by wooing many supporters from the Greens.

But many party members fear that Lafontaine's left-wing views would alienate the majority of moderate voters in a national election.

Rau's victory today, coupled with the poor showing of the Greens, is expected to bolster his claim that he alone can attract the broadest spectrum of support against the Christian Democrats on a national level.

The Christian Democrats were troubled throughout the campaign by dissension between Worms and the state party leader, Kurt Biedenkopf, a maverick intellectual whose purportedly arrogant ways led to a serious personal rift with Kohl.