"""Math anxiety""" exists because of this retarded association people make between being smart and math ability. You're supposed to be automagically good at math or you're a retard, or so the meme goes. Or people conflate math with mental calculations, which to some degree is an innate skill, and think they can't do math because they lack that skill.
So people get scared of pursuing math. They don't want to prove to themselves that they are idiots.
Sure, to some degree it's true that you need some IQ to do math, but it really isn't any more true for math than for any other subject, say, chemistry. You need to be able to read and think logically at least a little bit.
In reality, getting good math is just like getting good at any other academic subject. You have to study a lot, do things in the right order, and be patient. There's no magic or special gene that will let you get out of doing the work. I guess the biggest difference with other subjects is that you can't cram or bullshit your way through it.
The question is: did our brains evolve to learn megabytes of high-level math or is it counterproductive to train our neural networks into an autistic equation solvers? I tend to hold the second opinion.