Is my Goldfish a White Supremacist?

I bought a white goldfish. And I also have black and white pebbles in the tank he swims in. And for some reason, he tends to stay away from the black pebbles. Should I be concerned with my goldfish's bigotry?