I looked at the TEXTMAP and the parser code in p_setup.c. Some observations:
The editor writes out high precision doubles for floorplane_a/b/c/d. I think this is fine though; fixed_t shouldn't reach a precision where the differences in atof parsers should matter. UDMF should probably never have used decimal representation in this fashion in the first place but that's baked into the spec and there's nothing we can do about that now.
When interpreting the value token on a property assignment, the parsing code always assumes the token is of the expected type i.e. calling atof on the value based only on the key name and not the type of the token itself. This is concerning and may be problematic for malformed TEXTMAPs, but I don't think the inconsistency is rooted in this.
Z_Calloc is used to allocate the arrays of map data, so it's not an issue of reading undefined values.
On RR, we are seeing a related issue for apple clang aarch64 on mac exhibited as an I_Assert on dynamic slope configuration, but we also see that with clang on linux.
Inspected extended bsp loading. while there are some issues there they are not undefined behavior. so it's probably UDMF parsing. but it would be nice to have a version of this map in binary format with both vanilla nodes and XGL3 nodes so we can confirm that
I am able to reproduce this only with Debug builds for CMAKE_OSX_ARCHITECTURES=arm64. RelWithDebInfo works fine. This matches the issue we see on Ring Racers.