I recently sent this email to someone who asked me about my experience of this phenomenon/condition, along with sub-queries on a particular (though theoretical) key and 'sadness' in music.
...
It's definitely the case that people with synasthesia experience colours differently - I read about Scriabin's very pronounced condition and many of the colours were different to mine.
"G sharp major" does not really exist for me, since I would hear it as A flat major. That is a dull, but rich red with hints of purple. (I'm not sure what I would see in the almost inconceivable case that someone was in B major and then modulated through sharp keys all the way to G sharp major!)
"Sad keys" are partly determined by musical context. All the minor keys are potentially sad. Perhaps I could say that E flat minor and B minor are particularly "sad", while G major, A major and E major are particularly "happy" (I omit D major from that list because it seems richer than "happy", but this is leaving the question of colours somewhat). Here's a non-systematic and incomplete list of impressions...
Greens come from E, E flat, G and G flat
Yellows come from B and D
Reds come from A, A flat and F sharp minor (F sharp major is like arainbow, with orange emphasis)
F minor is ivory and purple, sometimes pink
C minor is black, dark brown, also gun metal, and other things hard toput into words (C major is like a shiny version of that, or sometimes can be matt, like peat)
B flat is almost white; the minor is grey
D flat is almost colourless, but also sometimes off-white
C sharp minor is impossible to put into words! But I love it.
