Depression has its own stigma, people who are depressed are thought of as weak, they should "just get over it and be happy," or are told its "just a case of the blues." I don't want to suggest that depression isn't a real disease or minimize it in any way, but it's not the kind of mental illness that people are referring to when they use the word "crazy."
Crazy is something altogether different. Crazy is delusion, psychosis, mania, schizophrenia. Insanity, in the depths of society's psyche, is jabbering in tongues rocking back and forth in a padded room. It can't be trusted. It is the serial killer, the mother who kills her children, the man who laughs while committing the most vile crimes - this is what "crazy" conjures up in the minds of the general public.
This terror, this nightmare looming in the dark places of our collective consciousness is harmful. Incredibly so. It means that people who are not neurotypical are stuck with the paradoxical choice of lying or being mistrusted. Perhaps more importantly, it makes us less likely to seek help when it is needed. It took me years to admit, even to myself, that my brain was fundamentally different than most. Because I didn't want to be crazy.
I still don't. I'm terrified of the psychotic break that I am 30 - 50% more likely than others to have, even though I know that it would probably be manageable if it were to happen. I don't generally share the fact of my mental illness with others, even with those I am close to. Admitting who I am is risking ever being trusted or taken seriously again. The reason for this is "crazy."