Whiter skin correlates with neanderthal admixture and cold climate adaptation, which correlates with robust immune systems. Therefore, all else being equal a man who sires offspring with a light-skinned woman is going to pass his genes on in children who are less likely to get wiped out by novel diseases and parasites. Because, all else equal, men pursue the R-selection strategy of sowing their wild oats, they are inclined to “sex upward” (like women are inclined to “marry upward”).
It has been reported that men prefer women whose skin is “slightly paler than average in their society” but experience shows this to be false. All races of men prefer white blondes sexually, when they can get them. This is confused somewhat by confounding factors like racial solidarity and the primacy of female sexual preferences (sexual psychology is very weird to begin with).
Women, on the other hand, actually do prefer men who are slightly darker than average in their society (but not by too much!). Why is this? I suspect it is due to the “sexy son” phenomenon, but regardless of the cause it is a self-reinforcing system.