In an interview with ABC News, President Obama told Barbara Walters that recreational pot smoking in states that have legalized the drug is not a major concern for his administration. 

"We've got bigger fish to fry," Obama said of marijuana smokers in Colorado and Washington, the two states where recreational use is now legal.

"It would not make sense for us to see a top priority as going after recreational users in states that have determined that it's legal," he said. 

Going after individual users has never been part of federal policy. But under Obama, the Drug Enforcement Administration has aggressively gone after medical marijuana dispensaries in California, where they are legal. In September, federal officials raided several Los Angeles shops and sent warnings to many more. 

"This is a tough problem, because Congress has not yet changed the law," Obama told Walters of the legalization in Colorado and Washington. "I head up the executive branch; we're supposed to be carrying out laws. And so what we're going to need to have is a conversation about, how do you reconcile a federal law that still says marijuana is a federal offense and state laws that say that it's legal?"

Attorney General Eric Holder said in a speech Wednesday that he would announce a policy on the new state laws "relatively soon." Notably, Holder declined to weigh in against the measures in Colorado and Washington before they passed, as he did with a 2010 attempt to decriminalize recreational use in California.

Senate Judiciary Chairman Patrick Leahy (D-Vt.) is holding a hearing to examine how federal laws and enforcement square with new state laws legalizing pot. In a letter to drug czar Gil Kerlikowske, Leahy suggested that one option would be amending the Federal Controlled Substances Act "to allow possession of up to one ounce of marijuana, at least in jurisdictions where it is legal under state law."

The president, who smoked pot often in high school, told Walters that he does not support general legalization "at this point." It's the same position he's taken throughout his political career, despite his own history. 

"There are a bunch of things I did that I regret when I was a kid," Obama told Walters. "My attitude is, substance abuse generally is not good for our kids, not good for our society."

But Ethan Nadelmann, executive director of the pro-legalization Drug Policy Alliance, saw the interview as a "tentative step in the right direction." Obama's response, he said, "reminded me so much of gay marriage," which the president opposed, had "evolving" feelings about and eventually supported in May 2012.

Gallup polling has shown support for pot legalization, like gay marriage, steadily increasing. Last October, it hit 50 percent.  

 At the same time, Nadelmann said, the Obama administration's record on marijuana policy is "not reassuring." 

Tom Angell of Marijuana Majority, another pro-legalization group, criticized the president in a statement for trying "to unjustifiably pass the buck to Congress."

"The president should lead on this issue instead of deferring to Congress, a branch of government that he probably knows better than most isn't exactly prone to getting a whole lot done these days," he said. 

Video: Wonkblog's Dylan Matthews explains the complex legal and economic structures around possessing marijuana