It was easy in President Obama’s first term for libertarians and conservatives to stick together. Whether one believes in small government or limited government (there is a difference), one could agree that the stimulus bill and Obamacare were ineffective and counterproductive pieces of legislation that unwisely grew the federal government. Likewise, on opposition to tax increases, cap-and-trade and excessive Environmental Protection Agency regulations, there was no daylight between libertarians and conservatives.

In the second Obama term, however, and with candidates running on an affirmative message in 2014 and readying themselves for 2016, the tension between the groups is straining to the breaking point. In part this is because of the re-emergence of social issues and even more so because of foreign policy.

Longtime conservative and evangelical leader Gary Bauer questions the political viability of  libertarianism, noting that a recent Pew study showed that most self-described libertarians are not very libertarian. “An even bigger problem is that once people find out what libertarianism means, most reject it. Pew found that just 11 percent of Americans both know what libertarianism means — defined by Pew as ‘someone whose political views emphasize individual freedom by limiting the role of government’ — and embrace the label.” He wonders, “Would libertarians abolish Social Security, Medicaid and Medicare, an idea that many libertarians endorse but which is unlikely to win over middle America?”

It is on social issues and foreign policy, however, that push really comes to shove. Classic conservatives recoil at what Bauer describes as the libertarians’ ” ‘anything goes’ view of life — the legalization of prostitution and all drugs, for example — and reject the idea that virtue should be encouraged and rewarded.” And on foreign policy Bauer is unsparing:

One of the main tenets of libertarianism is a less active foreign policy. But at the moment most Americans want exactly the opposite. A late August Pew Research poll concluded “As new dangers loom, more think the U.S. does ‘too little’ to solve world problems.” . . . Perhaps this widespread support for a strong American presence in the world is why Rand Paul surprised many people recently by invoking Reagan and claiming that if he were president he would “seek congressional authorization to destroy ISIS militarily.

Some libertarians, including Richard Epstein, argue that an isolationist foreign policy reflects a misunderstanding of true libertarian principles, but in practice the overwhelming number of libertarians vehemently oppose U.S. interventionism and want to eliminate foreign aid and slash defense spending. This, after all, was part and parcel of Paul’s pitch to non-Republican voters — with him they can have a foreign policy more like liberal Democrats’ than conservative Republicans’.

The difference between conservatives and libertarians also arises in the context of fighting Islamist terror. Libertarians have gotten the notion that the Bill of Rights supplants the laws of war and protects, for example, American jihadists from being droned and data gathering to detect terror plots. In that they often seem to be mimicking the Obama administration’s fetish with applying criminal justice concepts to anti-terrorism policy. Most Republicans resist that leap of logic and constitutional misinterpretation.

Bauer is also right that there is a tonal difference between the two groups. (“Republicans won’t be able to close the ‘empathy gap’ by embracing a political philosophy based on extreme self-interest and a cranky ‘leave-us-alone’ mentality.”) Libertarians’ single-minded focus on liberty and rooting out government’s presence wherever possible puts them at odds with many conservative reformers, who, for example, think fighting poverty, increasing opportunity and public investment in human capital (e.g. education, scientific research) are legitimate aims of government. The difference becomes clear in the tax reform debate where libertarians want to root out all social policy from the tax code while conservatives are interested in, for example, expanding the child tax credit and the Earned Income Tax Credit, in order to promote broad social goals. Libertarians don’t much care if a flat tax is regressive; conservatives for the most part do.

We saw the difference when Paul took on civil rights laws (before he changed 180 degrees) on TV with Rachel Maddow, arguing that the portion of those laws regulating private property were problematic. Invoking property rights to defend the “right” to discriminate is an argument that the overwhelming number of conservatives would reject.

In sum, fighting against big-government liberals draws libertarians and conservatives together. But when it comes to articulating their own vision, we see how different the philosophies are. And judging from Paul’s contortions on everything from civil rights laws to destroying the Islamic State, we see it is hard to be both libertarian and conservative on some critical issues. Trying to be both conveys a lack of principle and candor.

One might conclude then, as Bauer does, that “libertarian conservatives certainly have a role to play in the future of the Republican Party,” but still agree that the GOP cannot be subsumed to a libertarian philosophy without losing its appeal – and many of its current voters. Republican voters will decide which way to go in 2016.