The last time someone asked when we were having our second, I said: "we got it right the first time, so don't need to go back and try again." It might come off as arrogant, but it makes people stop for a second. The next response if they push is "I can't have any more" and it serves them right if they feel awkward. People will always find fault whether you have none or half a dozen, so as long as you can provide a happy and healthy environment for any children you do have, that's the only thing that truly matters.
In my very humble opinion: