This will prevent tests from dirtying the source/build directories,
or worse overwriting/removing files we care about.

This commit expands on some existing hacks, which I don't like, but
they should hopefully be going away soon.