Georgia attorney W. Spencer Lee IV said under oath yesterday that he never discussed fugitive financier Robert Vesco's legal problems with his longtime friend Hamilton Jordan - despite a promise that he would get $1 million from Vesco associates just for approaching President Carter's top aide.

Lee told Securities and Exchange Commission attorneys in U.S. District Court in Manhattan that he backed out of the deal at the last minute during a trip to Washington in February 1977 because he didn't want to hurt the Carter administration.

He also insisted that he never drafted or signed letters addressed to Jordan and Charles Kirbo, another confidant of President Carter, which appear to implicate them in the deal being pushed by R. L. Herring, a Vesco associate from Georgia.

"I didn't have any difficulty [with the deal] until they went to the White House," Lee said of this change of heart in approaching Jordan. "But it's not so easy to knock on the White House door and go in as a friend and then pop this on him [Jordan]."

Lee said he then dropped out of the scheme. But a year later, he testified, he agreed to a Herring proposal to represent Vesco in plea-bargaining on federal charges the exiled businessman faced - in return for a $500,000 fee.

In denying any knowledge of the alleged letters to Jordan and Kirbo, Lee's sworn testimony conflicts with that given Thursday in the SEC proceeding.

Gerolyn Hobbs, Herring's former secretary, said she had typed and mailed the letters addressed to Jordan and Kirbo. And she placed Lee in Herring's office at the time.

The alleged letters were the basis for a controversial series of articles by syndicated columnist Jack Anderson, which originally "linked" the two presidential advisers to a $10 million political fix."

Anderson revised the columns after indignant rebuttals by the White House last weekend and revised his initial story to soften the allegation. Jordan and Kirbo deny receiving the letters or doing anything to aid Vesco.

Vesco has been living in Costa Rica and, more recently, in the Bahamas since fleeing the United States following criminal indictments in 1972. He is charged with looting millions of dollars from an SEC-monitored company and then trying to buy his way out with a $200,000 gift to the Nixon campaign in 1972.

Vesdo is reported to have approached Herring in late 1976 with plans for approaching the new Carter administration to settle his legal problems. In return for their influence, he allegedly transferred several million dollars in stock to the Georgia group.

The SEC is interested in the alleged scheme because Vesco has been under court orders barring him from access to his former company's funds. In the New York hearings this week, SEC attorney Gregory Glynn asked federal Judge Charles Stewart to hold Vesco in contempt and extend the earlier order to include the Georgians.

Lee depicted himself yesterday as having had little knowledge of the complicated scheme he joined, though he accepted $10,000 from Herring at the outset, took trips to see Vesco in Costa Rica and set up a dummy corporation in Nassau.

Of one meeting with Vesco, in Costa Rica he said: "Everybody in that room knew why they were there but me."

The Vesco plan, he said, was to convince the Carter administration that "the United States had to sit down and listen to Robert Vesco or the Panama Canal treaties wouldn't pass." This was so, Lee said, because Vesco calimed to control the presidents of six Latin American countries.

He delivered a letter from one president, Daniel Oduber of Costa Rica, to Carter people in December 1976, Lee said.

He said he also attended a meeting with Kirbo in Atlanta in January 1977 to discuss Vesco's plan. But he quoted Kirbo as saying, "I'm sorry, boys. I can't help you all. Nor my law firm. I'm too close to the president of the United States." Kirbo has said he doesn't remeber such an encounteor.

Lee has remained close to the administration. In May 1977, a few months after he said he dropped out of the Vesco scheme, he was named by President Carter as a member of a panel to nominate U.S. Circuit Court of Appeals judges.

Last fall he sponsored a "roast" of Jordan in their hometown, Albany, Ga., with Kirbo among the guests, he said.

Lee joked with reporters during a break in the hearing yesterday and offered a theory on the sometimes bizarre dealings: "I think R.L. Herring was conning Robert Vesco."