The case where test runner is not enabled got broken in 2a457867

This time, we had test to prevent regression.