Interior Secretary James G. Watt had repeatedly refused to meet with Audubon Society leaders, but he managed to find time less than 24 hours after the White House apparently leaned on him to restrain his confrontational approach to environmental groups, society spokesmen said yesterday.

The incident, which appeared to be the first White House move to restrain Watt on this issue, was "pure coincidence and nothing more," according to Watt spokesman Douglas Baldwin. He acknowledged that "the timing makes it look funny," but said there had been no White House intervention.

Watt met in Washington May 1 with Audubon Society President Russell Peterson and Senior Vice President Rupert Cutler. Yesterday Cutler described the meeting as "relaxed, with a fair amount of humor."

Audubon had been asking since January to meet with Watt, Cutler said, and Washington representative William Butler finally received a refusal letter, dated April 21, from G. Ray Arnett, assistant secretary for fish, wildlife and parks. Cutler called the letter "offensive" and said that Arnett apparently wanted a "constructive change of attitude."

It said that a meeting between Watt and environmental leaders chosen by Peterson "is not in the cards at this time."

"Until the administration goals in objectives are clearly defined [and] in operation and a sense of understanding is exhibited among your followers, little will be gained from a meeting structured on opposition, distress, resentment and/or frustration," the letter said.

Noting that he had seen no evidence of an effort to "cull the bad apples" from environmental organizations, Arnett wrote: "Actions do speak louder than words, Bill. I hope you will agree that a positive, constructive change of attitude toward the Reagan administration exhibited by Audubon, the National Wildlife Federation, the Sierra Club and other prominent conservation organizations will go a long way toward lessening tensions and enabling a meaningful dialogue to begin."

Butler described the letter April 30 to James P. Miller, associate director of the Office of Management and Budget, during a 90-minute session Miller held with environmental groups to ask them for better communication. "There probably was a connection," Cutler said, between that conversation and the fact that Arnett called Peterson the next day to set up the meeting.

Watt told reporters yesterday before the Arnett letter came up that he had been meeting with environmentalists and found out-of-towners "very good, very helpful and committed." But "the hired guns of the movement haven't been as cooperative," he said. "Their agenda is different: membership, dollars and headlines."

He mentioned Wilderness Society officials as among those with whom he had not yet managed to coordinate schedules. Responded former Wisconsin senator Gaylord Nelson, who heads the Wilderness Society, "I'd rather be a hired gun for the public interest than a hired gun for the special interests as Watts is now and has been for years."