As I grew up and lived with Japanese culture, I accepted the description "yellow voice" without question. So now I've looked up why the high voices of women or children are called yellow (黄色い声). If it was described as black voice, blue voice or brown voice I would be confused. Maybe red voice would be OK, but to me yellow voice is better. Anyway, I looked it up.
In ancient China, Buddhist chanting had melody. They described the tones of sounds with colours. The highest tone was yellow.
『「言葉のルーツ」おもしろ雑学(エンサイクロネット著)PHP文庫』の「黄色い声」の項目163頁 (Roots of Words, interesting knowledge by Encyclonet PHP inc. p.163) http://homepage2.nifty.com/osiete/s661.htm
Does this enlighten you on why their voices are yellow?
When the first women won seats in a parliament after WWII, Japanese newspapers wrote, ”With the yellow voice, the female parliament member ----" (黄色い声を張り上げて、女性議員は、。。。).
When I learned this, I was very proud as a woman.
Now no one would say "A female member spoke with a yellow voice."
When I hear "A woman with a yellow voice", I imagine a woman who works hard. (頑張る女性。)
Some say that the yellow voices of children are cute.
However, others say that the colour yellow symbolizes unease and stress; therefore, a yellow voice is uneasy or stressed.
If you do not like women and children, I guess that you get stress from a yellow voice. How you feel about words depends on your experience with the words.
Note:
I could not find the English spelling for "エンサイクロネット", so I made something up from the sound.
「言葉のルーツ」おもしろ雑学 -> I could not find the proper English name; therefore, I translated it into English.
I read or heard on T.V. news about a woman with a yellow voice, though I do not remember exactly which news program and date. I think that Fusae Ichikawa was interviewed. I don't quite recall.
There are also many articles about yellow voices. Among those articles, I thought that this was the most suitable. じゃあね。