Sure, both listener and conn need them, but for different things
(listener uses them simply to set up conn).  Putting them in the
common union was a mistake.
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>