You could say it is involuntary servitude, yes. So is being forced to serve on a jury, amongst other things.
It is a reasonable infringement upon liberty, with a net benefit to both society and the individual. The very worst part of such servitude is making more money, whereas the very worst part of denying services based upon bigotry is self-evident, and a great deal worse.
I can deal with making an exception in this case.
As for the Constitutionality of it, it is obviously legal, as the Civil Rights Act still stands and has the same implications.