floating point numbers

Some math functions did not check if the result is a valid number
(i.e. neither of +-inf or nan).

Fixed by validating the result where necessary and returning NULL in
case of invalid result.