In a perfect world, that would be true. In fact, skin color is just another phenotype, like hair color, eye color, etc. However, skin color has been inextricably linked with the notion of "race" and everything that race connotes. We live in a racist society and it is impossible not to be affected by it.
If people were actually able to say that they weren't into a certain skin tone in the same way that they aren't into a certain eye color, there would be no problem, and perhaps in a few generations we will get there.
You say you don't seek out people who look like you as potential partners. But I bet you haven't made a categorical decision that all blue-eyed people, forever and all, are unsuitable as potential partners. And I don't imagine you have a lot of stories and preconceived notions about how blue-eyed people are and why you don't choose them as partners. You're just not into them.
Because skin color is so solidly linked with "race," it is nearly impossible to make that choice.