1. 22 Dec, 2006 2 commits
    • cmiller@zippy.cornsilk.net's avatar
      Merge zippy.cornsilk.net:/home/cmiller/work/mysql/bug22555/my50-bug22555 · 83355bfb
      cmiller@zippy.cornsilk.net authored
      into  zippy.cornsilk.net:/home/cmiller/work/mysql/bug22555/my51-bug22555
      83355bfb
    • cmiller@zippy.cornsilk.net's avatar
      Bug#22555: STDDEV yields positive result for groups with only one row · 50726b23
      cmiller@zippy.cornsilk.net authored
      When only one row was present, the subtraction of nearly the same number 
      resulted in catastropic cancellation, introducing an error in the 
      VARIANCE calculation near 1e-15.  That was sqrt()ed to get STDDEV, the 
      error was escallated to near 1e-8.  
      
      The simple fix of testing for a row count of 1 and forcing that to yield 
      0.0 is insufficient, as two rows of the same value should also have a
      variance of 0.0, yet the error would be about the same.
      
      So, this patch changes the formula that computes the VARIANCE to be one
      that is not subject to catastrophic cancellation.
      
      In addition, it now uses only (faster-than-decimal) floating point numbers
      to calculate, and renders that to other types on demand.
      50726b23
  2. 11 Nov, 2006 3 commits
  3. 10 Nov, 2006 9 commits
  4. 09 Nov, 2006 7 commits
  5. 08 Nov, 2006 19 commits