The Smithsonian Institution is struggling with little success to renegotiate the terms of a $5 million gift from Saudi Arabia because influential critics of the gift say it came with too many strings attached and would give Saudi Arabia too high a profile on the Mall.

The gift, which was solicited and negotiated by former Smithsonian Secretary S. Dillon Ripley, was promised in December 1983 to help build the Smithsonian's $75 million Quadrangle project. Its critics include Ripley's successor, Smithsonian Secretary Robert McCormick Adams; members of the Smithsonian's Board of Regents, and Rep. Sidney Yates, chairman of the House subcommittee that appropriates the Smithsonian's federal budget each year.

A lawyer for the Saudi government said the Smithsonian's objections have "unilaterally terminated" the contract and that the money, $1 million of which has been received, should be returned.

"We're saying return the money, and let's start from scratch," said Frederick G. Dutton, Washington counsel for the Saudi government. Dutton added that the Smithsonian's turnabout has left the Saudis feeling "sideswiped."

He believes critics of the contract's provision for a Center for Islamic Arts and Culture in the Quadrangle's so-called International Center are injecting Middle East politics where they don't belong.

"It's one secretary pulling the rug out from what Ripley before had been pushing," Dutton said. "The bottom line is he [Adams] is being told by Yates, 'You're getting too interested in Islamic things' . . . What annoys me is they've got the Holocaust thing [museum] here, and fine, they should have it, but they don't want a quality Islamic thing that would be under the Smithsonian's control."

Yates and Adams rejected that characterization yesterday. Yates said that his objections were only to the terms of the Saudi contract:

"I did not see the contract when it was signed and when I subsequently learned about it I thought it was a contract that relinquished the Smithsonian's control," Yates said.

Yates and Adams say details of the contract were never shown to the Smithsonian Regents or Congress, and that when those details became known last year, there was immediate concern. "I knew as soon as I saw it I had a problem," Adams said.

The nature of that problem has been laid out in a series of letters between Dutton and Adams. ". . . it is overbalanced to the point of being inappropriate," Adams wrote Dutton late last month, "to establish and endow a single center of this kind within an International Center of such modest size as we now anticipate."

According to Adams the primary problem with the Saudi contract is that it has been overtaken by events. Adams said that original plans for the International Center have been drastically cut back because of federal budgetary restraints as well as limited interest from the foreign governments Ripley was counting on.

Dutton said he heard of the funding problem for the first time last month, and believes that it developed only after Yates decided he didn't want an Islamic study center on the Mall.

Yates counters that Ripley never presented sufficient justification for the International Center and that the subcommittee cut it from the Quadrangle appropriation before it knew of the Saudi gift. It remains in the Quadrangle plan, however.

Dutton has since proposed using the Saudi $5 million to pay for an Islamic study center outright. When Adams rejected that idea, Dutton says, he proposed using the $5 million to establish a program of Islamic studies and scholarship grants.

The Smithsonian's most recent counterproposal has been to use the Saudi money to help pay for an impending acquisition of Islamic and Persian art. Dutton rejects that. "Why should we pay for something that is already being acquired?" he said.

Adams wrote to members of Congress this week informing them that the "long-term prognosis" for the negotiations "remains doubtful."

Some confusion and controversy over the Saudi deal can be attributed to the Smithsonian's unique, quasi-federal status, as well as the ruling style of its former secretary and his cadre of Yale-educated lieutenants. Ripley, who retired last year, is famous both for his autocratic style and for the institution's renaissance. Congressional critics, however, said that under Ripley the Smithsonian became Kremlinesque, notorious for its reluctance to tell Capitol Hill too much about what it was doing.

Also known as the Center for African, Near Eastern and Asian Cultures, the Quadrangle has been described as Ripley's swan song, the last of his grand ventures during an expansionist reign.

The design, much of it underground below the Smithsonian Castle's Victorian garden, includes a new space for the National Museum of African Art (now on Capitol Hill); a second museum, to be called the Arthur M. Sackler Gallery, that will house the collection of 1,000 objects of Mideastern and Asian art donated by New York research psychiatrist Arthur M. Sackler; and a third area known as the International Center.

Although its definition and purpose have been vague, Ripley most often described the International Center as a place for research, internships, symposiums and performances of traditional music and readings from African and Asian literature.

"My whole thesis was that we are innocents abroad, and this was an effort to increase understanding about that vast arc from the western shore of Africa to the eastern shore of Asia," Ripley said recently.

Faced with congressional skepticism for the International Center, Ripley volunteered to raise half the Quadrangle construction money, or more than $35 million, from private sources and the Smithsonian's own trust fund. He received $1 million each from Japan and South Korea, $50,000 from Bahrain and a smaller amount from Cameroon. The Saudi gift was the largest and the only one accompanied by contractual obligations, according to a Smithsonian spokesman.

Those obligations included the establishment of an Islamic Study Center in the International Center; Saudi representation on an informal Islamic Study Center committee that would "assure historical and content accuracy" of Islamic materials; and the understanding that Islamic art and artifacts would be exhibited in the Quadrangle's new Museum of African Art, which, the contract said, "will be named the Asian and African Gallery."

That last provision appears to rename the National Museum for African Art and permits the installation of Islamic art and artifacts there instead of in the Sackler Gallery, which the Smithsonian had designated as the place for Islamic, Persian and other art from the Arab world.

Yates said both changes came as a surprise to him and were unacceptable. "I haven't any idea why Ripley wanted to change it but it seems to me on a major change of this kind, the committee ought to have been advised. It was directly contrary to everything the Smithsonian had told the committee."

Dutton says he is not sure how the clause got into the contract or precisely what it means, but that it was Ripley's idea, as was the entire gift.

"I mean, the Saudis didn't suggest the gift," Dutton said. "The Smithsonian kept asking, they specified $5 million, they specified an Islamic Center and at this point we're in the position that the contract is terminated."

Reached yesterday at his home in Connecticut, Ripley professed ignorance of the controversy surrounding the contract he negotiated:

"I'm afraid I don't know what this hubbub is about," he said. "I'm afraid I don't know anything about this . . . I would never dream of 'giving away the store.' "

Ripley said he always believed money would be found to complete his plan for the International Center and added that he saw no problem in putting Islamic art in the National Museum of African Art because Islam has a dominant place in the culture of North Africa and sub-Saharan Africa.

"If you want to make something political out of this, you can, but we're simply interested in culture and the history of culture," Ripley said. "I mentioned the International Center to all governments, whether Israel or Arab . . . We had hoped that by developing the Islamic Center we would make people aware of that part of the world and the great contribution of Islamic culture to the culture of Western Europe.

"We are all sons of Arabia," he said. "This was my dream. The dream is tarnished."