• Dave Gosselin's avatar
    MDEV-34969: test fail main.spatial_utility_function_simplify · 613b4aa1
    Dave Gosselin authored
    On aarch64 and when computing the perpendicular distance, we need to
    avoid the fmsub (Fused Multiply-Subtract) because it can return
    slightly different precision results when evaluating expressions
    like
      x = a - (b * c)
    such as we did (before this patch) in perpendicular-distance.  Instead,
    we now store the result of the multiplication (b * c) and then subtract
    it from a which avoids (in all build types) fmsub.
    
    This error occurs because the C++ standard allows implementations to
    use higher precision throughout calculations before storing to the destination
    type.  This can allow for differences in the final result if using higher
    precision meaningfully changes intermediate values which is what happened in
    our case.  By breaking the operation into two, we prevent the use case where
    fmsub applies, even for optimized builds under our current build flags
    configuration.
    
    See also
      https://github.com/dotnet/runtime/issues/64591
      https://stackoverflow.com/questions/51124436/strange-issue-with-floating-point-accuracy-on-arm64
    613b4aa1
spatial.cc 108 KB