Sen. Rand Paul (R-Ky.) said Sunday that while he doesn't want to legalize drugs, he doesn't think people should go to jail for non-violent drug crimes.

Paul and Sen. Patrick Leahy (D-Vt.) last week introduced a bill that would give judges greater flexibility by allowing them to work around mandatory minimum sentences for drug crimes, which civil rights groups say often don't fit the crime.

"I don't want to encourage people to do [drugs]. I think even marijuana's a bad thing to do. I think it takes away your incentive to work and show up and do the things that you should be doing," Paul said on "Fox News Sunday." "But I also don't want to put people in jail who make a mistake. There are a lot of young people who do this and then later on in their twenties they grow up and get married, they quit doing things like this."

Paul noted that recent presidents have admitted to drug use in their past, and that strict drug laws could have put them behind bars and zapped their potential.

"Look, the last two presidents could conceivably have been put in jail for their drug use. Look what would have happened. It would have ruined their lives," he said. "They got lucky, but a lot of poor kids, particularly in the inner city, don't get lucky. They don't have good attorneys. They go to jail for these things, and I think it's a big mistake."

Paul's idea is also gaining some support from fiscal conservatives, including Grover Norquist, who noted the cost of incarcerating so many non-violent criminals.