The Republicans appeared assured of continued but possibly diminished control of the Senate yesterday as Jesse Helms (R) won reelection in North Carolina but Roger W. Jepsen (R) lost in Iowa.

In defiance of the Reagan landslide and apparent Republican gains in the House, it appeared likely that the Democrats would shrink the GOP's current 55-45 margin in the Senate by picking up one seat and possibly more.

The Republicans appeared to have gained a seat in Kentucky, where Sen. Walter D. Huddleston (D) was trailing Mitch McConnell (R), but that likely gain was offset by a long-expected Democratic victory in Tennessee.

In Illinois, Republican incumbent Charles H. Percy, chairman of the Foreign Relations Committee, was running several points behind Rep. Paul Simon (D) in Illinois. NBC projected a Simon victory.

If Percy loses, the conservative Helms would be in line to become chairman of Foreign Relations, although he promised North Carolina farmers he would continue as chairman of the Agriculture Committee instead.

Democrats were also winning three of the four seats being vacated by retiring senators.

In Massachusetts, Lt. Gov. John F. Kerry (D) defeated conservative businessman Raymond Shamie (R) for the seat of retiring Sen. Paul E. Tsongas (D).

In Tennessee, Rep. Albert Gore Jr. (D) beat Victor Ashe (R) as expected to fill the seat of Senate Majority Leader Howard H. Baker Jr., who also is stepping down.

In West Virginia, Gov. John D. (Jay) Rockefeller IV (D) defeated John R. Raese (R), a Morgantown businessman and political unknown. But Raese made a strong, late surge-out-of-nowhere that appeared for a time to threaten Rockefeller, who had once seemed an easy successor to retiring Sen. Jennings Randolph (D).

Only in Texas did the Republicans win an open seat. Although the Democrats had once hoped to score another breakthrough in Texas, Rep. Phil Gramm, a Democrat-turned-Republican, snatched an easy, early victory from state Sen. Lloyd Doggett for the seat of retiring Sen. John G. Tower (R).

With the Democrats picking up Republican seats in Tennessee and Iowa and the Republicans possibly unseating a Democrat in Kentucky, the Democrats were ahead by one. They were looking to the Percy seat, still in some doubt late last night, to give them two.

In another race that had been in question, the Democrats held off a Republican challenge in Montana as Sen. Max Baucus defeated Chuck Cozzens (R).

In Kentucky, Jefferson County (Louisville area) Judge McConnell was leading Huddleston by only about 4,000 votes out of nearly 1.3 million cast.

Huddleston had been favored for a third term, although Republicans viewed McConnell as a strong dark-horse possibility and regarded him as a bellwether for testing the strength of President Reagan's coattails in Senate races. Reagan carried Kentucky with ease.

Relatively easy victories were won, as expected, by Sens. John W. Warner (R-Va.), Sam Nunn (D-Ga.), Howell Heflin (D-Ala.), David L. Boren Jr. (D-Okla.), Bill Bradley (D-N.J.), Joseph R. Biden Jr. (D-Del.), J. James Exon (D-Neb.), David H. Pryor (D-Ark.), Claiborne Pell (D-R.I.), Mark O. Hatfield (R-Ore.), Nancy Landon Kassebaum (R-Kan.), Larry Pressler (R-S.D.), Strom Thurmond (R-S.C.), Thad Cochran (R-Miss.), William S. Cohen (R-Maine), Rudy Boschwitz (R-Minn.), Gordon J. Humphrey (R-N.H.), Alan K. Simpson (R-Wyo.), William L. Armstrong (R-Colo.), James A. McClure (R-Idaho), Pete V. Domenici (R-N.M.) and Ted Stevens (R-Alaska).

In Michigan, Sen. Carl Levin (D) appeared to have defeated former astronaut Jack Lousma (R).

Elected to a third term even before the polls opened was Sen. J. Bennett Johnston (D-La.), who defeated two political unknowns six weeks ago in the state's unusual nonpartisan primary, in which any candidate who gets 50 percent of the vote is elected without having to run in November.

Of the group of Southerners who appeared headed for the easiest early victories, only Mississippi's Cochran, a freshman lawmaker who was the first Republican to win a statewide office since Reconstruction, had been considered to be in potentially serious trouble. He was opposed by former governor William F. Winter (D), whom the Democrats had counted on early in the campaign for a possible upset.

New Hampshire's Humphrey, who was elected six years ago as a staunch conservative but veered toward the political center as his reelection battle approached, was an early target for the Democrats. But their hopes quickly faded despite a vigorous campaign by Rep. Norman E. D'Amours (D).

Boschwitz had been a prime early target for the Democrats, in part because Minnesota was the home state of Democratic presidential nominee Walter F. Mondale, but he easily withstood the challenge from Secretary of State Joan A. Growe (D).

Four years ago the Republicans ended a quarter century of Democratic domination of the Senate by riding Reagan's coattails to an unexpected 12-seat victory, the biggest gain since Democrats staged a comeback at the end of the Eisenhower administration.

They went into this election with a margin of 55 to 45, having picked up two more seats in midterm and special elections in 1982 and '83.

Despite their edge from a second ride on Reagan's coattails and a booming economy in most areas of the country, their optimism was tempered by several factors.

Foremost among them was the fact that they were defending 19 of the 33 seats at stake in this year's elections, while the Democrats were defending only 14.

Moreover, most of the Democrats were strongly positioned for reelection. Nunn, for example, was endorsed by his home-state Republican colleague, Sen. Mack Mattingly, who is anxious for any positive ruboff for his own reelection campaign in two years.

Conversely, some of the Republican incumbents who had to face the voters this year were elected originally when Democratic incumbents were feeling the political drag of the Carter administration, and they were not particularly strong contenders on their own.

Some, such as Iowa's Jepsen, had suffered blows, sometimes self-inflicted, during their six years in office. Jepsen had an awesome record of bounding back from adversity but was sorely tested this time around, with the low point coming when it was disclosed earlier this year that this outspoken champion of conservative morality had once visited an X-rated health spa.

But the Republicans also had strong advantages, including Reagan's willingness to campaign for the party's senatorial candidates, which he did repeatedly in several cases, especially for vulnerable incumbents like Helms, Jepsen and Percy. And like Percy, some wrapped themselves from head to toe in the president's mantle, even though they had departed from the Reagan script on some occasions.

Although Democratic incumbents tended to be well-financed on their own and the Democrats were twice as well-financed nationally as they had ever been before, they were still at a clear financial and technological disadvantage.

The Democrats will hold an enormous advantage heading into the 1986 elections, when Republicans will be defending an even higher proportion of seats -- with even shakier incumbents -- than they were this year. In 1986, when their big Class of '80 comes up for reelection, they will be defending 22 seats, compared with 12 for the Democrats. Midterm elections during a president's final term also can be an inauspicious time for the party in power, as the Republicans learned in 1958.

The preelection claims of the two parties going into the election were relatively modest, prompting expectations of relative continuity in the Senate, at least through 1986.

But as Election Day approached, both camps were made nervous by memories of the sharp last-minute swing in the 1980 races and the number of extremely tight races this year that would affect the GOP's somewhat narrow margin of control.

Out on the hustings, that nervousness also contributed to a final spurt of big campaign spending and a cacophony of hard-hitting, sometimes nasty advertising that gave many of the campaigns a distinctly negative and sour tone.

Curiously, tactics that prompted cries of outrage when employed by free-lance campaign groups such as the National Conservative Political Action Committee several years ago were almost routinely used this year by the candidates.

By the accounts of most observers, the tight races in North Carolina and Iowa came close, at least, to setting new records for venom, although a few states like Mississippi managed to get through the campaign season with a semblance of gracefulness.

The campaign also set new records for fund-raising, with the Helms-Hunt race in North Carolina leading the way with a total of nearly $22 million, about double the previous one-state record set by California in 1982. All told, the 33 races were costing one-third again as much as all the 1982 races.

As for preelection expectations, the Democrats, including Senatorial Campaign Committee Chairman Lloyd Bentsen (Tex.), were counting on picking up at least two seats, perhaps as many as four or five. Officially, Bentsen never gave up on the possibility of regaining control of the Senate.

But most Democratic strategists conceded privately that they expected to fall short of the six new seats they would need with Vice President Bush in the presiding officer's chair to break a tie on Senate organization. They would need only five new seats to reclaim control if the Democrats won the presidential election and Geraldine A. Ferraro wound up as the tie-breaker.

Republicans acknowledged that they could slip to as low as 51 seats but said they could also go as high as 56 seats. More likely, said National Republican Senatorial Committee Chairman Richard G. Lugar (Ind.), was a break-even outcome of 55 to 45 again.

As small as the expected gains and losses were, even the smallest shift was viewed as potentially critical for the 99th Congress. Although the Senate marched in virtual lock-step with Reagan in the first months of his administration, working with a bipartisan conservative majority in the House to give the president nearly everything he wanted, it became increasingly balky.