A leading South Africa newspaper reported yesterday that the South African government had secretly funneled about $11.5 million to an American publisher in recent years in an unsuccessful attempt to purchase The Washington Star.

The report - the latest development in a widening South African political scandal - drew an immediate denial from John P. McGoff, the Michigan publisher who was thwarted in his repeated efforts to buy The Star from its former owners. In a statement issued by his office, McGoff termed the report by The Rand Daily Mail "utterly false" and "wantonly crude."

Although there was no independent confirmation yesterday of the Rand Daily Mail account, knowledgeable officials and observers here and in South Africa said the existence of stringent South African libel laws appeared to lend credence to the report. Unless the newspaper had substantial documentation for its allegations, they said, it would not likely have dared to print them.

McGoff, president of the Panax Corp. newspaper publishig firm in East Lansing, Mich., is described by his assoicates as an "ultraconservative" and a supporter of South Africa, although one official noted that McGoff has criticized South Africa's policy of apartheid, or strict racial segregation. McGofff is know to have some financial interests in South Africa and personal ties with key South African officials.

In Washington, several businessmen familiar with McGoff's attempts, mainly in 1974 and 1975, to buy The Star said yesterday that considerably mystery had surrounded the financial underpinnings of his offers. McGoff had bid about $25 million for The Star, which was then gripped by severe financial troubles, but he never disclosed how he would obtain the funds, the businessmen noted.

At the time, speculation in financial circles here had centered on McGoff's reported ties to an heir to a major U.S chemical company fortune but, one former Star official said yesterday, there was later a rumor that South African funds were available to McGoff to help buy The Star.

The Star was eventually taken over by millionaire Texas banker Joe L. Allbritton, who sold the newspaper to the Time Inc. publishing empire earlier this year. McGoff efforts in 1974 and 1975 to buy the paper and renewed hints in 1977 that he might still be interested in purchasing it never appear to have led to serious negotiations.

The Rand Daily Mail's report yesterday was an outgrowth of the biggest political scandal in more than 30 years of National Party rule in South Africa. The deepening controversy centers on alleged abuses stemming from a secret government fund intended to finance a public relations campaign to win support for the South African government among other nations.

The scandal has prompted the 5-week-old government of Prime Minister P. W. Botha to step up a previously launched government investigation of the secret fund. It apparently has caused political damage to former information minister Connie Mulder, once viewed as a likely prime minister. Mulder is now minister of black affairs. The controversy-ridden Department of Information has been dissolved and Mulder's deputy, information secretary Eschel Rhoodie, retired.

McGoff, Mulder and Rhoodie together own a ranch in South Africa. McGoff's associates describe the ranch as a private venture for the Michigan publisher, whose Panax firm has one corporate holding in South Africa, an investment in a printing plant.

The reported $11.5 million payment to McGoff appears to be the largest of a series of expenditures allegedly made from the secret fund. In May, when the scandal first surfaced, a government auditor reported that Information Mininstry offcials had misused hundreds of thousands of dollars to finance "unnecessary, probably wasteful and extravagant trips."

There was also been indications that the secret fund may have been used to finance South Africa promotional advertisements in The Washington Post and other major newspapers in Western countries. In a stunning disclosure last month, including the Rand Daily Mail, reported that the secret fund was used to set up and finance a pro-government newspaper, The Citizen.

In its account yesterday, The Rand Daily Mail reported that McGoff had received 10 million rand - about $11.5 million - from former information secretary Rhoodie to help purchase The Star. After failing in his efforts to buy The Star, McGoff eventually completed repayment of the funds last January, the newspaper said. The Daily Mail did not report how much interest McGoff paid on what apparently amounted to a loan.

The South African newspaper reported that the money was returned by way of Switzerland, where, it said, a portion of the funds was used to pay off a separate loan that had been obtained to help finance The Citizen newspaper. The Mail said McGoff could not return the funds until he had completed a European business deal, about which no further details were disclosed.

McGoff, in his statements yesterday, said:

"The allegations made by the Rand Daily Mail - an implacable and frequently rabidly irresponsible enemy of its own government - are not only utterly false but wantonly crude. They are so lacking in any kind of substantiation or documentation as to cast severe doubt on the seriousness and integrity of publications which reproduce them. And these allegations, so recklessly repeated, are so crude as to err by a full two years in the timetable of my actual involvement in negotiations to acquire The Washington Post.

"Responsible journalism, it seems to me, would direct its investigative energies and curiosity to a newspaper which would publish such flimsy charges, rather than to the victims of such charges," the Michigan publisher's statement added.

In a telephone interview from Johannesburg, a spokesman for the Rand Daily Mail dismissed McGoff's denial, saying that the South African newspaper stood by accuracy of its account. The spokesman said that McGoff's reference to a two-year discrepancy in th Mail's account stemmed from a misinterpretation of the newspaper's information. The Mail will publish a clarification of this point in an editorial today, the newspaper.

The dispute over the dates stems from a passage in the Rand Daily Mail report which appears to imply that McGoff received the $11.5 million in 1976 - some time after his main efforts to buy The Star had already failed. The Mail spokesman said, however, that the newspaper's information shows that McGoff had already received the money by 1975 - a time when he was actively seeking to purchase The Star.

Several key former and present Star officials said yesterday thay never had any indication that McGoff might have had South African aid in his attempts to buy The Star.

John H. Kauffmann, who was Star president during the negotiations in 1974 over the newspaper's sale, said in a telephone interview. "There was never any inkling in my mind or on the part of anybody on this side of the table that there was any South African money involved." Kauffman added that McGoff's finances were largely unknown at the time.

Murray J. Gart, The Star's current editor, said that neither he nor any other Time Inc. official was aware of the details of McGoff's efforts to buy The Star. Former Star publisher Allbritton could not be reached for comment yesterday.

Some of McGoff's assoicates expressed surprise at the Rand Daily Mail report yesterday. "I would be shocked if John was getting money from the South African government," said Jason L. Shrinsky, a Washington lawyer who has represented McGoff in various communications proceedings, including McGoff's proposals to buy the Star.

Neither the South African government nor the U.S. State Department had any comment on the Rand Daily Mail report. The Citizen, in an editorial yesterday, deplored what it described as a "campaign of gossip, insinuation innuendo and outright denigration" stemming from the investigation of the secret government fund.

"Newspapers involved in this campaign should not be carried away by their zeal for senation," the editorial added. "They should be aware of the fact that it is one thing to make wild allegations; it is another to substantial them. Unless they can prove their allegations they run the danger of landing themselves in trouble of the worst kind.