I think part of the problem is that they don't bounce their principles off reality. I think many of their principles are actually very sound and could be appreciated by the general population, particularly the libertarian ones, but they don't apply them to reality (this is the reason why the LP has never taken off and why Ron Paul is never taken seriously).
Take abortion for example. If research could show that abortion rates actually increase in countries who outlawed abortion (I don't know if this is true; I'm just using this as a demonstration of how principle can be applied to reality), wouldn't the principle of keeping abortion as low as possible be better applied by keeping it legal? From there, the principle could be further applied to disallowing federal funding for abortion, which is something far less extreme then outright outlawing abortion and failing to recognize the reality of how your principle is applied (again, this is just a principle/reality example and not a legal abortion = less abortion claim). Federal monies previously spent for abortion could then be spent on increased educational awareness of what abortion entails and how ya might not want to do it, or just saved and not spent at all...both actions that support conservative principles! Then, boom, ya might pull in that female vote afterall.
The same can be applied to immigration and a resulting hispanic vote. And on and on.