That's pretty sad. One should always be open to learn about other cultures and people. There's not such thing as "acting your color", the color of your skin does not define you any more than the size of our ears or the shape of your teeth. I see that you have a better head on your shoulders than those busybodies.
Anyways, keep at it. Spanish is a pretty language, and if your husband decides to speak to your child in Spanish and you don't learn some you'll be left out. I talk from experience, we have a similar problem at home.