We have an ideal standard of living now as a result of the cheap labor of hordes of poor people. Otherwise things like palm oil, coffee, and kitchen appliances would be far, far, more expensive. Our enviable position on the supply chain can only be balanced by ten times more people on the other side of it.
With a small population that enjoys higher wages, the costs of everything goes up, and the resulting standard of living gets diminished.
This is not to say we couldn't still be healthy and happy, but consumerism would have to end. No more smartphones and mango salsa and cheap shoes for the masses.
Which really wouldn't be a bad thing, in the final analysis. I just wonder about what effect diminishing returns would have on the development of truly useful technologies.