Sen. Lindsey Graham (R-S.C.) is so far finding little support on either side of the aisle for a boycott of the 2014 Winter Olympics in Sochi if Russia grants NSA leaker Edward Snowden asylum.

"I love Senator Graham. We've been close friends for 20 years. But I think he's dead wrong," House Speaker John Boehner (R-Ohio) told reporters Wednesday. “Why would we want to punish U.S. athletes who've been training for three years to compete in the Olympics over a traitor who can't find a place to call home?"

White House Communications Director Jay Carney wasn't quite as blunt, saying only that he would not "engage in speculation" and that "the Olympics are a long way off."

The administration is "focused on trying to resolve this matter in a way that is in the interest of the United States and in the interest of U.S.- Russian relations," he added.

"I don't know whether we should go that far," Sen. Chuck Schumer (D-N.Y.) said on MSNBC.

Graham first suggested the boycott in an interview with The Hill newspaper published Tuesday evening. By Wednesday, he was expressing his own doubts:

In an interview with CNN, he explained that his concern was not just Snowden but Russia's support for the Syrian government and Iranian nuclear development.

"I don't want to boycott the Olympics, but I want a policy that will get the Russians' attention," Graham said.

The U.S., along with several other countries, boycotted the 1980 Summer Olympics in Moscow over the Soviet invasion of Afghanistan. In response, the U.S.S.R. and allied Eastern Bloc countries boycotted the 1984 Summer Olympics in Los Angeles.