Skip to content

Fix float comparison for colormap maskamt

mazmazz requested to merge float-compare-colormap-fix into master

Welp, looks like we got our first regression from !383 (merged)...

This code block compares a proposed colormap's maskamt, a double, to the list of existing colormaps' maskamt. If there's a match, then don't create a new colormap.

The old check failed in 32-bit release builds because the returned delta was greater than the compare value 1.0-E36. This resulted in more colormaps being created than necessary -- in one case, crashing DSZ2 for exceeding MAXCOLORMAPS.

I thought about changing the comparison value, but then I figured that we don't need to cast maskamt - extracolormaps[i].maskamt to float when it's already a double.

So I just changed the comparison to a double and it now works. Tested in 32-bit release and debug builds; and 64-bit release and debug builds.

On another note, it may be worth considering a different compare value... DBL_EPSILON maybe?

Merge request reports