Worried Republicans yesterday began studying the implications of their loss of a House seat in a conservative Mississippi district, while Democrats welcomed their upset winner, Wayne Dowdy, as a harvinger of the honeymoon's end for President Reagan.

A jubilant Speaker Thomas P. (Tip) O'Neil Jr. (D-Mass.), the target of GOP advertising in last Tuesday's special election in Mississippi's 4th District, swore in Dowdy yesterday and then told the 37-year-old freshman and his family, "You're not any more happy than I am."

The reception in O'Neill's office turned into a pageant of party harmony, as Rep. G.V. (Sonny) Montgomery (D-Miss.), a leader of the Conservative Democratic Forum, whose members have given Reagan two major budget victories over O'Neill, joined the handshaking and revelry.

"The prodigal son returns," O'Neil joshed. "Welcome back, Sonny."

Montgomery told a reporter that Dowdy's upset win over Republican Liles Williams in a district the GOP has held since 1972 showed "the Democratic Party is dead in the South -- or certainly in Mississippi.It also proves you can put all the philosophies together in one tent and elect a Democrat. It's an encouraging thing to me as a Democrat."

Republican officials, who brought Reagan heavily into the campaign via television and letter endorsements of Williams, were worried enough about the implications of the loss to order pollster V. Lance Tarrance to take a special follow-up survey in an effort to pinpoint what had gone wrong. Tarrance's interviewers were making their telephone calls last night.

Dowdy repeated to Washington reporters what he had said after his 1,100-vote victory was certified Tuesday night. "My election was not a repudiation of President Reagan," he said. "He is still an extremely popular president in my district. I emphasized that I was running against the Republican candidate not Ronald Reagan, and they failed -- or succeeded just a little bit -- in their effort to make it a referendum on President Reagan."

The view drew confirmation from Nancy Sinnott, executive director of the Republican Congressional Campaign Committee. "Our candidate failed to establish his own identification apart from being Ronald Reagan's candidate," she said. "The voters had a better sense of Wayne Dowdy as a person than they did of Liles Williams."

A White House official agreed, saying that Williams' managers were beguiled by polls showing 74 percent support for Reagan in the district and they "failed to develop any local issues."

Dowdy said the massive television advertising by the GOP in a campaign in which he was outspent by 4 to 1 may have backfired by helping boost the turnout he needed to win. The vote was 25 percent larger in the Tuesday runoff than it had been in the June all-party primary, where Williams came close to a majority and estabilished himself as the favorite in the race.

Dowdy said his estimates were that he had won 95 percent or more of the black vote in the district and about 30 percent of the white vote. Blacks comprise and 45 percent of the voting-age population.

In the previous elections where Republicans had won, their candidates had either gained a significant slice of the black vote or had seen it diverted to black independent candidates.

In the closing phase of the campaign, Dowdy's support of the extension of the Voting Rights Act and William's opposition to it became a major focus of attention. Radio ads for Dowdy, taped by a niece of slain civil rights leader Medgar Evers, urged blacks to "remember that Martin Luther King and Medgar Evers died for your right to vote, and Ronald Regan and Liles Williams want to take it away."

Reagan has delayed taking a position on the Voting Rights Act extension and some GOP officials said yesterday that the Mississippi results may provide ammunition for those in the administration urging him to endorse the measure.

Sinnott said her preliminary reports were that the Voting Rights Act issue had boosted the black vote and that some whites had been lost to Williams or had stayed home because of Democratic claims that Reagan might jeopardize their Social Security benefits.

She also said, "They just did a better job of turning out their vote than we did in turning out ours."

Rep. Tony Coelho (Calif.), chairman of the Democratic Congressional Campaign Committee, said the Social Security issue had also been a prime factor in the near-upset last month in Ohio's 4th District, where Democrats came within 300 votes of victory in a strongly GOP area. Coelho also maintained that the results showed that "even when we are badly outspent, where we put together a technically competent campaign with adequate funds, and have Democratic unity, we can do well despite all the pressures the White House can muster."

He and Dowdy both credited Mississippi Gov. William F. Winter (D) with playing a key role in uniting the Democrats after Dowdy finished ahead of a previously favored Democrat in the June primary.

That unity was still visible yesterday, as veteran Sen. John C. Stennis (D-Miss.); the dean of the House, Rep. Jamie L. Whitten (D-Miss.), and Reps. Montgomery and David R. Bowen (D-Miss.) crowded around Dowdy, introducing him to other Democrats.

House Minority Whip Trent Lott, now the only Republican in the Mississippi House delegation, gamely reached across the aisle to congratulate Dowdy.

Lott said later that he thought local factors had led to the upset. He pointed to the "very difficult set of circumstances that created the vacancy" -- the resignation of Rep. Jon Hinson (R-Miss.) after his arrest on sodomy charges stemming from an incident in a House office building bathroom. Lott also said the Williams campaign "got a little overconfident," turning down offers of campaign help from other Republicans, and failed to match the emotionalism of the Democrats' get-out-the-vote appeal.

Lott said, "I just hope he [Dowdy] joins the boll-weevil caucus," referring to Montgomery and the other conservative southern Democrats. But Dowdy, while attending a reception Montgomery gave in his office, told reporters he had not decided whether to join Montgomery's group. "I'm going to look at every issue as it comes along and just see how it affects my district," he said.