Skip to content
Snippets Groups Projects

Fix crash in R_PointToDist2 when passing -2147483648

Merged Hanicef requested to merge Hanicef/SRB2Classic:fix-pointtodist-negative-abs into next

So, here's a fun one: if any of the arguments to R_PointToDist2 is exactly -2147483648, the game crashes with a segfault. That's because 2147483648 cannot be represented as a 32-bit integer, and thus abs(-2147483648) == -2147483648. What then causes the crash is a check that picks the largest value of the two arguments will then pick the wrong one, causing it to fetch a value beyond the size of tantoangle and thus crash.

This can easily be reproduced by creating a Lua script containing this single line:

FixedHypot(-2147483648, 1572864)  -- latter value doesn't matter, I'm just using what originally triggered the bug.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
Please register or sign in to reply
Loading