Attorney General-designate William French Smith, President-elect Ronald Reagan's personal attorney and a man who once said he would prosecute any lawbreaker, including a president, is seething over news reports that his integrity has been tainted by his attending a Frank Sinatra birthday party.

Smith, here for a round of courtesy calls on members of the Senate Judiciary Committee, which is to begin his confirmation hearings next week, acknowledged that he had attended the singer's elaborate birthday party the day after Smith's Cabinet nomination was announced.

But Smith said his relationship with Sinatra, who once had a Nevada gaming license revoked because of alleged ties to organized crime and who long has been rumored to be close to mob figures, was "social and rare -- mostly chance meetings on the golf course and that kind of thing."

Smith said he was "totally unaware of any allegations about Frank Sinatra's background," and that if he ever had read about Sinatra's reputed ties to mob figures the stories had not sunk in. He said he had no access to Justice Department investigatory files and would not see them until after he is confirmed.

But he labeled news reports questioning the propriety of his appearance at the birthday party as "nothing less than scurrilous," repeating the word "scurrilous" three times, until he finally added, "Am I making myself clear?"

Smith, a Los Angeles attorney with mostly corporate and utility company clients in addition to Reagan, clearly was upset, and said the social appearance would "not affect me in the slightest in making any prosecutorial decisions in the future."

The first report appeared in a Maxine Cheshire column in The Washington Post last month. Smith's appearance also was criticized Monday in a New York Times column by William Safire.

The affair unlikely to have any serious impact on Smith's confirmation hearings, scheduled to begin next Thursday. But the reports once again brought questions to the surface about Reagan's ties to the famous singer, whose colorful and somewhat murky past includes occasionally overlapping friendships with both presidents and crime figures.

Sinatra was a major fund-rasier for Reagan, pulling in large campaign contributions with his Hollywood glitter and mystique. Reagan also was invited to the birthday party, although he did not attend. But he and his wife, Nancy, interrupted their 1976 presidential campaign to attend Sinatra's fourth wedding.

After the election, Sinatra was named as the producer of a million-dollar, star-studded pre-inaugural "gala" for the Reagans. The party will be televised nationally Jan. 19.

Sinatra long has fancied himself as a friend of presidents and presidential contenders. In 1960 he raised money for John F. Kennedy, and later helped organize inaugural festivities for Kennedy just as he is for Reagan. Kennedy's relationship with Sinatra later backfired, however, and proved to be a significant embarrassment.

In 1963 Sinatra, then a part owner of two Nevada casinos, had his gambling license revoked by the State Gaming Commission because of his "continued association with Sam Giancana, well knowing his unsavory and notorious reputation." Giancana was an alleged mob boss banned from gambling activity in Nevada. He later was murdered.

Sinatra reportedly introduced President Kennedy to Judith Campbell Exner, who testified long after Kennedy's death that she had had relationships with both Kennedy and Giancana. Kennedy's friendship with Sinatra was ended abruptly, according to reports, when FBI wiretaps picked up Exner's calls to the White House.

At about the same time the CIA is said to have enlisted Giancana's help, according to reports that emerged much later, in its bizarre and abortive efforts to use organized-crime figures in assassination plots against Cuban premier Fidel Castro.

Because of his show-business past, Reagan's friendship with Sinatra dates back many years. However, the friendship began receiving prominent attention only after the singer used Reagan as a personal reference in his attempt to regain the Nevada gambling license he lost 17 years ago.

Sinatra is seeking the license to "participate in management" of Caesars Palace in Las Vegas. New Jersey gambling authorities recently rejected an application by Caesars Palace management to open a casino in Atlantic City. The New Jersey authorities said the application was rejected because Clifford S. Perlaman, the casino's president, associated with organized-crime figures.

A hearing on Sinatra's Nevada application was scheduled for last month, but was delayed until Feb. 11, at the singer's request, because he said he will be busy with the inaugural.

Meanwhile, Smith said questions about his relationship with Sinatra had not been raised by any members of the Senate Judiciary Committee with whom he has been talking this week. That was confirmed, in general, by staff sources on Capitol Hill, although an aide to Sen. Edward M. Kennedy (D-Mass.), the committee's ranking minority member, said Kennedy had told Smith that he may be questioned during the hearing about his "friendships and relationships."

"I hope we talk about more substantive matters than this," the Cabinet nominee said.

Smith said he went to the party -- which he described as attended by civic and entertainment leaders "typical of people who would attend any southern California function of this kind" -- as part of a group of friends, and was not sure whether he had received a personal invitation.

Smith said he "had not had any occasion -- or the exposure, or the reason -- to know of any allegations" against Sinatra. But he said that if he "had any information to conclude that whoever was involved was involved in some criminal activity, then, no, I would not have gone."

Asked if he now wished he had not gone to the party, Smith said he tries "not to look back, because you sometimes find yourself wrestling with nonexistent problems."

Sinatra would not discuss the party, according to his New York public-relations aide, Lee Solters, who said the singer "won't talk to members of the press about anything." Asked if any other members of Reagan's prospective Cabinet were at the party, Solters replied, "I wouldn't know one if I saw one."