Type integer or string: does it matter when querying?

[eluser]codex[/eluser]
Let's say you have a 'color' field in your db, where 'color' represents the color of a (for instance) shoe. You could have brown, black, white. But you could also have the colors be represented by an integer: 1=brown, 2=black, 3=white.

Would it make a difference on performance when you query something like "where color = 'brown'" instead of "where color = 1" '?