This reverts commit 24cf7c96.

According to the numpy documentation npy_isinf merely provides an
equivalent of C99 isinf, which in turn is only required to return a
non-zero value for a given infinite argument.

The original commit breaks the test when compiled with clang & libc++.