I was using V_GetColor in some places I shouldn't have, making the game look wrong with a non-default color profile. Though, I left R_RainbowColormap alone.
V_GetColor
R_RainbowColormap