Nothing is switching left to right or up-down. Instead, it's being inverted front to back.
Imagine you had an arrow and pointed it straight at the mirror. The arrow in the reflection would be pointing in the opposite direction — toward you. That reflection represents the photons of light, bouncing back in the same direction from which they came. (Warning: You might consider doing this in your home bathroom. I took a handmade paper arrow into the ladies room at The Washington Post and got a lot of weird looks.)
Now imagine you have six friends, each holding an arrow. The three friends on your right hold blue arrows, the three friends on your left hold green ones, and you stand in the middle with a black arrow. All of you point your arrows at the mirror. In the reflection, all of the arrows point back the opposite direction — blue reflected arrows pointing back at their blue real-life counterparts, green pointing at green, black pointing at black.
Clearly, the arrows haven't been altered in any way. But when you look at the reflected image, it will appear that the friends holding blue arrows are on your left, not your right. So what gives?
Think about it from the mirror's perspective. To a person standing where the mirror is, looking at the row of you and your friends, it would appear that the blue arrows are on the left. That's the image that the mirror is sending back — exactly what you present to it.
Still confused? Let's go through this with text. Imagine you printed out this article and held it in front of your face. You'd be able to read it clearly. But if you wanted to see the article's reflection, you'd have to hold it up to a mirror. And to do that, you'd have to turn it around.
Theoretically, you could do this by flipping the paper vertically — and if everyone looked at stuff in the mirror that way, we'd be talking about why mirrors reverse things up-down.
But if you're like most people, you turn the paper horizontally. That's what creates the impression that mirrors reverse things left to right. The mirror doesn't change the lateral direction of the text on the paper — you do.
One more example: Imagine putting a glove on your right hand and extending your arm toward a mirror. Your reflection will appear to be wearing a glove on its left hand.
How do you turn a right-handed glove into a left-handed one? By flipping it front to back, just as a mirror does. If you could slowly peel the glove from your right hand — pulling it from the bottom, the way nurses do, so it is inverted as you take it off — and then pass it through a mirror, it would fit on the left hand of your reflection perfectly.
For a seemingly simple question, the answer can really mess with your head. It's certainly something that is more easily explained visually. Try watching this great video from the Physics Girl, or go stand in front of a mirror and experiment yourself. If someone happens upon you waving your hands there and asks what you're doing, you can just tell them “science.”