1. 22 Jan, 2021 10 commits
    • Parav Pandit's avatar
      net/mlx5: SF, Add port add delete functionality · 8f010541
      Parav Pandit authored
      To handle SF port management outside of the eswitch as independent
      software layer, introduce eswitch notifier APIs so that mlx5 upper
      layer who wish to support sf port management in switchdev mode can
      perform its task whenever eswitch mode is set to switchdev or before
      eswitch is disabled.
      
      Initialize sf port table on such eswitch event.
      
      Add SF port add and delete functionality in switchdev mode.
      Destroy all SF ports when eswitch is disabled.
      Expose SF port add and delete to user via devlink commands.
      
      $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
      
      $ devlink port show
      pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
      
      $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
      pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:00:00 state inactive opstate detached
      
      $ devlink port show ens2f0npf0sf88
      pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:00:00 state inactive opstate detached
      
      or by its unique port index:
      $ devlink port show pci/0000:06:00.0/32768
      pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:00:00 state inactive opstate detached
      
      $ devlink port show ens2f0npf0sf88 -jp
      {
          "port": {
              "pci/0000:06:00.0/32768": {
                  "type": "eth",
                  "netdev": "ens2f0npf0sf88",
                  "flavour": "pcisf",
                  "controller": 0,
                  "pfnum": 0,
                  "sfnum": 88,
                  "external": false,
                  "splittable": false,
                  "function": {
                      "hw_addr": "00:00:00:00:00:00",
                      "state": "inactive",
                      "opstate": "detached"
                  }
              }
          }
      }
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      8f010541
    • Parav Pandit's avatar
      net/mlx5: E-switch, Add eswitch helpers for SF vport · d970812b
      Parav Pandit authored
      Add helpers to enable/disable eswitch port, register its devlink port and
      load its representor.
      Signed-off-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarRoi Dayan <roid@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      d970812b
    • Vu Pham's avatar
      net/mlx5: E-switch, Prepare eswitch to handle SF vport · d7f33a45
      Vu Pham authored
      Prepare eswitch to handle SF vport during
      (a) querying eswitch functions
      (b) egress ACL creation
      (c) account for SF vports in total vports calculation
      
      Assign a dedicated placeholder for SFs vports and their representors.
      They are placed after VFs vports and before ECPF vports as below:
      [PF,VF0,...,VFn,SF0,...SFm,ECPF,UPLINK].
      
      Change functions to map SF's vport numbers to indices when
      accessing the vports or representors arrays, and vice versa.
      Signed-off-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarRoi Dayan <roid@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      d7f33a45
    • Parav Pandit's avatar
      net/mlx5: SF, Add auxiliary device driver · 1958fc2f
      Parav Pandit authored
      Add auxiliary device driver for mlx5 subfunction auxiliary device.
      
      A mlx5 subfunction is similar to PCI PF and VF. For a subfunction
      an auxiliary device is created.
      
      As a result, when mlx5 SF auxiliary device binds to the driver,
      its netdev and rdma device are created, they appear as
      
      $ ls -l /sys/bus/auxiliary/devices/
      mlx5_core.sf.4 -> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/mlx5_core.sf.4
      
      $ ls -l /sys/class/net/eth1/device
      /sys/class/net/eth1/device -> ../../../mlx5_core.sf.4
      
      $ cat /sys/bus/auxiliary/devices/mlx5_core.sf.4/sfnum
      88
      
      $ devlink dev show
      pci/0000:06:00.0
      auxiliary/mlx5_core.sf.4
      
      $ devlink port show auxiliary/mlx5_core.sf.4/1
      auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
      
      $ rdma link show mlx5_0/1
      link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
      
      $ rdma dev show
      8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
      13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
      
      In future, devlink device instance name will adapt to have sfnum
      annotation using either an alias or as devlink instance name described
      in RFC [1].
      
      [1] https://lore.kernel.org/netdev/20200519092258.GF4655@nanopsycho/Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      1958fc2f
    • Parav Pandit's avatar
      net/mlx5: SF, Add auxiliary device support · 90d010b8
      Parav Pandit authored
      Introduce API to add and delete an auxiliary device for an SF.
      Each SF has its own dedicated window in the PCI BAR 2.
      
      SF device is similar to PCI PF and VF that supports multiple class of
      devices such as net, rdma and vdpa.
      
      SF device will be added or removed in subsequent patch during SF
      devlink port function state change command.
      
      A subfunction device exposes user supplied subfunction number which will
      be further used by systemd/udev to have deterministic name for its
      netdevice and rdma device.
      
      An mlx5 subfunction auxiliary device example:
      
      $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
      
      $ devlink port show
      pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
      
      $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
      pci/0000:08:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:00:00 state inactive opstate detached
      
      $ devlink port show ens2f0npf0sf88
      pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:88:88 state inactive opstate detached
      
      $ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88 state active
      
      On activation,
      
      $ ls -l /sys/bus/auxiliary/devices/
      mlx5_core.sf.4 -> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/mlx5_core.sf.4
      
      $ cat /sys/bus/auxiliary/devices/mlx5_core.sf.4/sfnum
      88
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      90d010b8
    • Parav Pandit's avatar
      net/mlx5: Introduce vhca state event notifier · f3196bb0
      Parav Pandit authored
      vhca state events indicates change in the state of the vhca that may
      occur due to a SF allocation, deallocation or enabling/disabling the
      SF HCA.
      
      Introduce vhca state event handler which will be used by SF devlink
      port manager and SF hardware id allocator in subsequent patches
      to act on the event.
      
      This enables single entity to subscribe, query and rearm the event
      for a function.
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      f3196bb0
    • Parav Pandit's avatar
      devlink: Support get and set state of port function · a556dded
      Parav Pandit authored
      devlink port function can be in active or inactive state.
      Allow users to get and set port function's state.
      
      When the port function it activated, its operational state may change
      after a while when the device is created and driver binds to it.
      Similarly on deactivation flow.
      
      To clearly describe the state of the port function and its device's
      operational state in the host system, define state and opstate
      attributes.
      
      Example of a PCI SF port which supports a port function:
      
      $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
      
      $ devlink port show
      pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
      
      $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
      pci/0000:08:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:00:00 state inactive opstate detached
      
      $ devlink port show pci/0000:06:00.0/32768
      pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:88:88 state inactive opstate detached
      
      $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 state active
      
      $ devlink port show pci/0000:06:00.0/32768 -jp
      {
          "port": {
              "pci/0000:06:00.0/32768": {
                  "type": "eth",
                  "netdev": "ens2f0npf0sf88",
                  "flavour": "pcisf",
                  "controller": 0,
                  "pfnum": 0,
                  "sfnum": 88,
                  "external": false,
                  "splittable": false,
                  "function": {
                      "hw_addr": "00:00:00:00:88:88",
                      "state": "active",
                      "opstate": "attached"
                  }
              }
          }
      }
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarJiri Pirko <jiri@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      a556dded
    • Parav Pandit's avatar
      devlink: Support add and delete devlink port · cd76dcd6
      Parav Pandit authored
      Extended devlink interface for the user to add and delete a port.
      Extend devlink to connect user requests to driver to add/delete
      a port in the device.
      
      Driver routines are invoked without holding devlink instance lock.
      This enables driver to perform several devlink objects registration,
      unregistration such as (port, health reporter, resource etc) by using
      existing devlink APIs.
      This also helps to uniformly use the code for port unregistration
      during driver unload and during port deletion initiated by user.
      
      Examples of add, show and delete commands:
      $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
      
      $ devlink port show
      pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false
      
      $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
      pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:00:00 state inactive opstate detached
      
      $ devlink port show pci/0000:06:00.0/32768
      pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:00:00 state inactive opstate detached
      
      $ udevadm test-builtin net_id /sys/class/net/eth6
      Load module index
      Parsed configuration file /usr/lib/systemd/network/99-default.link
      Created link configuration context.
      Using default interface naming scheme 'v245'.
      ID_NET_NAMING_SCHEME=v245
      ID_NET_NAME_PATH=enp6s0f0npf0sf88
      ID_NET_NAME_SLOT=ens2f0npf0sf88
      Unload module index
      Unloaded link configuration context.
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      cd76dcd6
    • Parav Pandit's avatar
      devlink: Introduce PCI SF port flavour and port attribute · b8288837
      Parav Pandit authored
      A PCI sub-function (SF) represents a portion of the device similar
      to PCI VF.
      
      In an eswitch, PCI SF may have port which is normally represented
      using a representor netdevice.
      To have better visibility of eswitch port, its association with SF,
      and its representor netdevice, introduce a PCI SF port flavour.
      
      When devlink port flavour is PCI SF, fill up PCI SF attributes of the
      port.
      
      Extend port name creation using PCI PF and SF number scheme on best
      effort basis, so that vendor drivers can skip defining their own
      scheme.
      This is done as cApfNSfM, where A, N and M are controller, PCI PF and
      PCI SF number respectively.
      This is similar to existing naming for PCI PF and PCI VF ports.
      
      An example view of a PCI SF port:
      
      $ devlink port show pci/0000:06:00.0/32768
      pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
        function:
          hw_addr 00:00:00:00:88:88 state active opstate attached
      
      $ devlink port show pci/0000:06:00.0/32768 -jp
      {
          "port": {
              "pci/0000:06:00.0/32768": {
                  "type": "eth",
                  "netdev": "ens2f0npf0sf88",
                  "flavour": "pcisf",
                  "controller": 0,
                  "pfnum": 0,
                  "sfnum": 88,
                  "splittable": false,
                  "function": {
                      "hw_addr": "00:00:00:00:88:88",
                      "state": "active",
                      "opstate": "attached"
                  }
              }
          }
      }
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarJiri Pirko <jiri@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      b8288837
    • Parav Pandit's avatar
      devlink: Prepare code to fill multiple port function attributes · 1230d948
      Parav Pandit authored
      Prepare code to fill zero or more port function optional attributes.
      Subsequent patch makes use of this to fill more port function
      attributes.
      Signed-off-by: default avatarParav Pandit <parav@nvidia.com>
      Reviewed-by: default avatarJiri Pirko <jiri@nvidia.com>
      Reviewed-by: default avatarVu Pham <vuhuong@nvidia.com>
      Signed-off-by: default avatarSaeed Mahameed <saeedm@nvidia.com>
      1230d948
  2. 20 Jan, 2021 14 commits
  3. 19 Jan, 2021 16 commits