The Color Theory

Before I begin, let me preface this with the disclaimer that this theory is complete and utter bull. I don't believe it, nor is there any actual evidence to support it. It's simply something I thought up years ago and like to play with from time to time.

How do we know that the colors we see actually correspond to anything in reality? Once the wavelengths of light hit our eye, the trigger a nervous response. The light doesn't actually go into our brains. So perhaps colors aren't actually what they look like to us. Maybe they don't actually exist at all. But regardless of that, if the light doesn't actually reach our brains, how do we know it is interpreted in the same way?

For example, I look at a banana and I see what I call yellow. But maybe to someone else, that's not the same color. In their mind, it could be what I call green. We both call it yellow, because that's what we've been taught it is. That wavelength of light, regardless of the object, has the same name. We would never know the difference, because we can't send the image in our head into someone else's head. Describing it, we use the same term for the same wavelength, and have no capability to convey our mental image beyond that same term. That could explain favorite colors. Maybe we all like the same color, but it corresponds to different wavelengths in reality.

Like I said, the theory is complete bull, and there is plenty of evidence to discount it. But it's an interesting concept, at least to me. Sounds like the start for a sci-fi book or something.