That's awful! My kids are similar. My youngest has a speech issue due to being born with a cyst in his brain and has a "low avg" IQ. My oldest is bordering on genius IQ but has a communication disorder (not as in speech but as in being able to express himself, his feelings, etc. adequately or understand others feelings... he's a sweet kid but he literally has zero empathy. In his world if it doesn't affect him then he doesn't understand what your issue is and if it does affect him it does so to the nth degree), behavior issues and obsessive tendencies. BOTH my kids potty trained at age 4, not because we weren't trying sooner but because they weren't ready. Literally, when they did potty train they did so over night. Diapers one day, undies the next and never looked back. So, I "get" having atypical kids, but that has nothing to do with parenting. In fact, forcing your kids to be who they aren't or do what they aren't ready to do is a recipe for a maladjusted kid in the long term. Some ppl can be so dense and think that just because they popped out a couple "neuro-typical" kids who at least seem to have grown up okay that they are the expert in parenting.
When we told my Dad and Grandma (Dad's Mom) that we were going to start trying for one more they weren't the most supportive, they were concerned that next time we'd "get one even worse off" than the 2 we have, and that we have our hands full already. TBH with all the doc appt's and needing to be SOOOO involved with their education (making sure my oldest is challenged when his knowledge base exceeds most of his peers... not braggng in any way it's just a fact. For example, in math his peers are learning long division while DS1 is sinking his teeth into pre-algebra as I tutor him at home to keep him challenged and growing. DS2 I have to be involved to make sure he isn't falling too far behind his peers) we really do have our hands full. BUT as I told them we will gladly accept whoever God chooses to send us, we are praying for a healthy baby but that may not be God's plan and we accept that and will accept the challenge if given to us. They have since come around and were a great support through our losses. My grandma doesn't have a clue about this one yet as we plan to surprise her when we visit her in May. My Dad is excited and loves getting ultrasound pics and finding out how things are going.
I am guessing it's a long shot that your nay-sayers will come around but in the end it's yours and your DH's life and nobody has to deal with your decisions but you. (((HUGS))) I am sorry you have someone bringing you down during a time that is so special!