Now that we don't have tty->termios tied to drivers->tty we can untangle
the logic here. In addition we can push the removal logic out of the
destructor path.

At that point we can think about sorting out tty_port and console and all
the other ugly hangovers.
Signed-off-by: Alan Cox <alan@linux.intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>