After binding to a l3mdev, it should use the route table from the
corresponding VRF to verify the addr when binding to an address.

Note ipv6 doesn't need it, as binding to ipv6 address does not
verify the addr with route lookup.
Signed-off-by: Xin Long <lucien.xin@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>