The Senate's historical system of rubber-stamping federal judgeship nominations ran into a new challenge yesterday in a controversial North Carolina case.

An all-day hearing by the Senate Judiciary Committee focused on the fitness and integrity of Charles B. Winberry Jr., of Rocky Mount. He was nominated nearly a year ago to fill a federal district judgeship with the strong support of Sen. Robert Morgan (D-N.C.) whose election campaign he managed in 1974.

The committee has processed 145 new judgeships in the past year and cleared them all. Members and staff aides have said Winberry's nomination is the most controversial, because of allegations about his ethical conduct.

Committee members heard sworn testimony yesterday about an unsubstantiated charge that Winberry helped bribe a judge to fix a case for a client in a 1976 cigarette-smuggling case in North Carolina.

Witnesses also challenged Winberry's denial that he knew -- and hid from the trial judge -- that the same client, David Windham, was a front man for other members of the illegal cigarette business.

Winberry denied all the charges yesterday.

Sens. Patrick J. Leahy (D-Vt.) and Orrin Hatch (R-Utah) have said they will vote against the Winberry nomination. But the Carter administration still supports the candidate and Morgan, who was present throughout the hearing yesterday, is still backing his former aide.

Several witnesses testified on Winberry's behalf yesterday, including North Carolina Gov. James Hunt and the incoming president of the state bar association.

Morgan said during a break in the hearing that he felt most of the criticism against Winberry was hearsay. "You have to take a man's general reputation," the senator said.

A committee vote on the nomination is set for Tuesday.

Though yesterday's hearing concentrated on Winberry's conduct in the cigarette-smuggling case, it is clear that an underlying issue is his general competence.

Hatch said yesterday that he talked privately with 20 North Carolina lawyers, at Morgan's invitation, and found that none of them favored the Winberry nomination. Hatch has referred several times to President Carter's 1976 campaign slogan "Why not the best?" in questioning Winberry's qualifications.

When he took office, Carter promised that his judicial appointments would be made solely on the basis of merit, not politics. Sen. Edward Kennedy (D-Mass.), chairman of the Judiciary Committee and now Carter's challenger for the presidency, promised that he would carefully scrutinize the nominations.

The committee, however, has only a few attorneys and investigators to process hundreds of judicial and other monimations. Until the Winberry case, it had been forced to depend mainly on routine FBI checks.