William A. Wilson, the U.S. ambassador to the Vatican, today denied any wrongdoing during his five years as President Reagan's representative to the Holy See and said that his resignation, announced yesterday, resulted from his desire to return to his previous life as a California real estate and investments magnate.

Wilson, 71, a longtime personal friend and financial adviser to Reagan, refused to discuss his controversial relations with Libya during his tenure as U.S. envoy, but he emphatically denied using his office to conduct private commercial dealings with Libya at a time when U.S. policy was trying to isolate Libyan leader Col. Muammar Qaddafi.

In answer to repeated questions at his farewell press conference, Wilson said he could "honestly deny" that he had conducted any business dealings with Libya. He said he was leaving not because of pressure but because he and his wife had decided to return to their private life in California.

"We are leaving because we feel what had to be done has been done," he said.

Asked about a secret, and apparently unauthorized, trip he made to Libya to meet with Qaddafi in January following terrorist attacks a month earlier against the Rome and Vienna airports, Wilson said, "I don't think this is the proper time to discuss that issue." The Reagan administration has blamed the attacks on Libyan-supported terrorists.

According to State Department sources here and in Washington, that visit, at a time when the Reagan administration was trying to limit U.S. contacts with Qaddafi, enraged Secretary of State George P. Shultz and U.S. Ambassador to Italy Maxwell Rabb.

Shultz, according to State Department sources, reprimanded Wilson at the time. Wilson, however, denied today that he had ever had his wrists slapped by Washington for such activities.

Vatican spokesman Joaquin Navarro Valls said today that the Holy See holds Wilson in "great personal esteem," United Press International reported.

At his press conference, Wilson responded to every question on his trip to Libya by stating that all that was to be said on it had already been said by the State Department.

Wilson came to Rome in 1981 as Reagan's personal envoy to the Holy See. When diplomatic relations were reestablished in January 1984, he became ambassador.

While ambassador, he has been accused of continuing to conduct his personal business, especially with Libya, a country he allegedly visited several times and with whose business and government agents he allegedly has met here.

Wilson also has been accused of trying to intervene in two criminal investigations of business friends -- for which he was admonished by the State Department -- and attempting to influence U.S. policy in favor of Chilean President Augusto Pinochet, with whom his family has done business.

He said he hoped to return to the board of the Pennzoil Co., a post he resigned "early this year." State Department officials had approved his board membership, he said, but later "they had had a change of mind so I . . . presented my resignation." He said Pennzoil had never had any oil dealings with Libya.