Casual in the US used to mean athletic and athletic socks, tube socks especially, were invented in white in the 70s and considered part of youth and sport culture. Before fast fashion options like tube socks most clothes were more formal and expensive reserved for school or work and (I guess this part is related to trends) a black sock would be far less jarring peeking out from a dark suit pant than white. Dark suits go back to the 1800s France I believe, maybe not quite that late. So it’s a combination of established fashion norms for men, the rise of fast fashion and tube socks and of women’s adoption of pants/women entering the work place in jobs that don’t require suits. Prior to that women tended to wear stockings or wool socks which don’t tend to be bright white.

I can’t speak for athletic or casual socks outside the US but in the US this is why. From my experience much of Europe has had very strict fashion rules compared to the US, like my French ex who literally will not wear shorts unless he’s at the beach or pool because it’s a faux pas and only wears black socks on a daily basis.