Clean up the O(n^2) stuff. The best way I found to do this was using sets; even the old _dirty flag could call things multiple times, which was not good.