The nomination of President Reagan's choice for ambassador to Ireland, New Jersey insurance executive William E. McCann, is in serious trouble because of allegations of improper business practices and his close ties with a convicted stock fraud and insurance swindler.

McCann, the second-largest fund-raiser in New Jersey for the Reagan campaign, was sponsored for the post by William J. Casey, now the CIA director, and Reagan national finance chairman Daniel Terra, now an assistant secretary of state. Reagan introduced McCann as his designated ambassador at a lunch at the Irish Embassy on St. Patrick's Day.

But the nomination has not made it back to the White House; it has been held up for four months by the State Department, longer than any other Reagan appointment.

The apparent reason for the delay in McCann's case is that the State Department security officials have been unable to complete the background report. Normally, the security check takes four to six weeks.

"I think the reason it has taken so long is that they're trying to verify or run down the allegations," said White House personnel director E. Pendleton James. "I would hope they're doing that."

Other high White House officials confirmed that the investigation into McCann's past was the reason for the delay. At the Senate Foreign Relations Committee, one staffer said that the long delay in receiving McCann's nomination "has become a touchy subject around here."

It is also a touchy subject at the White House, which under its procedures cannot even submit a formal statement to the Senate that it intends to nominate McCann until it receives a security approval from the State Department. But McCann has been behaving very much like an ambassador, attending functions at the Irish Embassy, including one this week where Loyal Davis, father of First Lady Nancy Reagan, received an honorary fellowship from Ireland's Royal College of Surgeons.

McCann served as national chairman of the Reagan-Bush Pioneers, a fund-raising group.With the backing of Casey and Terra, he won out as nominee for the ambassadorship over two other formidable candidates, one sponsored by premier New Jersey fund-raiser Raymond J. Donavan, who was so highly regarded by Reagan he was appointed secretary of labor. The other candidate was backed by Sen. Mark O. Hatfield (R-Ore.) plus a number of longtime western Reagan supporters.

McCann's problems are twofold. First, his firm, Foundation Life Insurance Co. of Chatham, N.J., is under investigation by New York state insurance officials for allegedly selling insurance without a license to a New York local of the Teamsters union.

Second, questions have been raised about McCann's close business relationship over a decade with Louis C. Ostret, a former insurance entrepreneur who has been convicted of stock fraud, embezzlement, grand larceny and income tax evasion. He is alleged to have high labor-organized crime contacts, including New York reputed labor racketeer Johnny (Johnny Dio) Duoguardi.

A New York insurance official said yesterday that McCann has refused to testify on allegations that his company sold insurance through a front company to Teamsters Local 918 in Brooklyn.

McCann was unavailable for comment yesterday, but he has steadfastly denied any wrongdoing.

Until recently, the White House hasn't paid too much attention to the delay, partly because of the backlog of ambassadorial appointments.

But concern rose in some quarters after the embarrassing resignation of CIA clandestine service chief Max Hugel earlier this week. Hugel, who was accused of improper of illegal stock practices, was the personal choice of CIA Director Casey, who had strongly defended him in a letter to the president.

Casey's role in the McCann appointment is less clear, but Reagan officials who worked with McCann in raising funds for the campaign said yesterday that the original suggestion for his appointment came from Casey. One of these officials said that Casey "strongly pushed" the appointment; another said that Casey only made the suggestion, as he did with many fund-raisers, and that the push had come from Terra.

In interviews in recent months, McCann has characterized Casey as a close friend and his chief benefactor.

McCann was described by someone who knew him well in the campaign as "a person also willing to help and a terrific fund-raiser." In any administration it is not unusual for such fund-raisers to be rewarded with ambassadorships.

What is unusual in this case is that McCann prevailed over the choice of Donovan, who is credited in the Reagan camp with carrying the state.

New York insurance investigators are preparing to subpoena McCann and other representatives of his company to force them to produce records and testimony. If eventually confirmed by the Senate, this could put McCann in the position of having to avoid diplomatic functions in New York to avoid a subpoena.

McCann has acknowledged that 30 to 40 percent of his company's insurance business was delivered under a special arrangement with Ostrer, to whom he referred in court testimony as "a bird dog who was out scaring up business" for Foundation Life. In a confidential interview with Senate investigators in 1972, McCann reportedly put Ostrer's involvement in providing his company's business at 40 to 50 percent.

In early June, Ostrer, who was already imprisoned, was indicted along with organized crime leaders Anthony (Joe Batters) Accardo of Chicago and Santo Trafficante of Miami in an alleged insurance swindle involving the welfare and benefit plans of the Laborers International Union.

Last year, Ostret was found guilty of embezzling $1.2 million from the Brooklyn Teamsters' local to which he and McCann's company were selling life insurance policies.

Ostrer is serving a term at the federal penitentiary in Danbury, Conn. The details of the Ostrer-McCann dealings were first made public by Newhouse News Service reporter Robert Cohen.

Since his relationship with Ostret first surfaced in the criminal proceedings against Ostrer last year, McCann has attempted to put distance between his own dealings, the operations of his company and Ostrer.

"Mr. Ostrer was not during the period of my affiliation with Foundation an agent of Foundation nor was he authorized to act on behalf of Foundation in any way . . .," McCann testified last year.

But a review of hundreds of pages of testimony given by McCan at Ostrer's 1980 trial in New York and investigative reports prepared by the Senate Permanent Subcommittee on Investigations shows that:

Ostrer was a close associate of McCann. They held regular business meetings from 1968 to 1976 and Ostrer owned 100,000 shares of Foundation's stock. Ostrer and McCann made joint presentations to union officials to obtain new business, and used each other's offices to hold business meetings.

Since Ostrer's license to sell insurance had been revoked in most jurisdictions, he worked through a firm called Modern Agency Inc., nominally owned by Ostrer's sister, Dina Gelman, but controlled by Ostrer. McCann testified in the 1980 trial that his company paid "in excess of $1 million" in commissions to Modern Agency for insurance policies with a face value totaling $150 million to $200 million and placed with Foundation Life.

During their nearly decade-long association, Ostrer engaged in a pattern of fraudulent practices that reaped millions of dollars in illegal profits from the pension and welfare funds of various unions. McCann, unwittingly, he maintains, acted as the underwriter of much of this business.

In a presentation to the directors of one Miami-based union local, Ostrer assured the union officials of the strength of his relationship to McCann's company by saying: "When we tell them to s---, they squat. We tell them what claims we want them to pay," according to the minutes of the meeting obtained by Senate investigators.

The subcommittee conducted an extensive investigation of Ostrer's insurance schemes beginning in 1972 and the probe led up to highly critical reports published in 1976 and 1977. McCann was interviewed for these studies.

Starting about 1970 with Teamsters Local 295 in New York, Ostrer was the architect of "severance life insurance" plans that parlayed the growth of severance pay benefits in union contracts into multimillion-dollar insurance business.

Basically, the plans called for using the new severance pay funds won by the unions in labor negotiations with management to buy life insurance policies for union members. But the Senate study found that, under Ostrer's plan, commissions and administrative fees were structured "more to benefit the creator of the system -- Louis C. Ostrer -- and his associates than the workers."

One GAO study of the Local 295 severance pay plan showed that commissions charged at individual rates as opposed to group rates led to the payment of $800,000 in commissions for policies written on 1,400 workers. The GAO study concluded that normal commission costs should have been about $10,000.

The Senate study called these commissions "unconscionable." The subcommittee investigated Ostrer's role in the sale of 12 union insurance plans before it published its highly critical report.Four of the plans were underwritten by McCann's Foundation Life.

Ostrer lost his license to sell insurance in New York in 1967 after he was convicted of stealing as much as $1.2 million from Canada Life Insurance Co. But Ostrer personally set up the insurance plans with Foundation Life and garnered most of the commissions earned in the name of two agents of record, Cy Reeves Snyder and Robert Kincel, according to their testimony in federal court last year.

Those agents worked through modern Agency, controlled by Ostrer through his sister. "Modern Agency was a substantial producer for the company [Foundation]," McCann told the Newhouse reporter.

McCann has said that he considered Ostrer a "consultant" to Modern Agency and was not aware that the agency was owned in the name of Ostrer's sister. McCann has also maintained that he did not read the Senate reports that spelled out the ownership of Modern Agency in 1976, and he disputes the findings of those reports.

As for Ostrer's reputation, McCann told Newhouse that he had no reason to question Ostrer's background because "he had no official capacity with Foundation Life."

"I was under the impression he had some disciplinary action taken by the Insurance Department of the state of New York, but I can't recall when I knew he had his license revokes," he told the New Jersey reporter, who also inquired whether McCann had known of Ostrer's 1973 stock fraud conviction: "I heard something about that, but I never knew too much about it."

In 1976, Ostrer helped set up the deal to sell insurance with a face value of about $30 million to Teamsters Local 918 in New York. None of the insurance agents affiliated with Modern Agency or Foundation had a license to do business in New York. According to the trial testimony last year, Ostrer, in consultation with McCann, arranged to set up a bogus branch office of the union next to the Modern Agency office in suburban New Jersey.

McCann reportedly denies this allegation, and was quoted in a New Jersey newspaper as saying, "I was not aware of Local 918 being located in Brooklyn until the case broke."