Refactor use of raw tuples/frozensets into custom classes.

This permits some more optimizations and speeds up the build another 20%
or so, in addition to being hopefully easier to follow.