Skip to content
Snippets Groups Projects

Fix float comparison for colormap maskamt

Merged mazmazz requested to merge float-compare-colormap-fix into master

Welp, looks like we got our first regression from !383 (merged)...

This code block compares a proposed colormap's maskamt, a double, to the list of existing colormaps' maskamt. If there's a match, then don't create a new colormap.

The old check failed in 32-bit release builds because the returned delta was greater than the compare value 1.0-E36. This resulted in more colormaps being created than necessary -- in one case, crashing DSZ2 for exceeding MAXCOLORMAPS.

I thought about changing the comparison value, but then I figured that we don't need to cast maskamt - extracolormaps[i].maskamt to float when it's already a double.

So I just changed the comparison to a double and it now works. Tested in 32-bit release and debug builds; and 64-bit release and debug builds.

On another note, it may be worth considering a different compare value... DBL_EPSILON maybe?

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading