Commit f8ab3047 authored by David S. Miller's avatar David S. Miller

Merge branch 'Introduce-connection-tracking-offload'

Paul Blakey says:

====================
Introduce connection tracking offload

Background
----------

The connection tracking action provides the ability to associate connection state to a packet.
The connection state may be used for stateful packet processing such as stateful firewalls
and NAT operations.

Connection tracking in TC SW
----------------------------

The CT state may be matched only after the CT action is performed.
As such, CT use cases are commonly implemented using multiple chains.
Consider the following TC filters, as an example:
1. tc filter add dev ens1f0_0 ingress prio 1 chain 0 proto ip flower \
    src_mac 24:8a:07:a5:28:01 ct_state -trk \
    action ct \
    pipe action goto chain 2

2. tc filter add dev ens1f0_0 ingress prio 1 chain 2 proto ip flower \
    ct_state +trk+new \
    action ct commit \
    pipe action tunnel_key set \
        src_ip 0.0.0.0 \
        dst_ip 7.7.7.8 \
        id 98 \
        dst_port 4789 \
    action mirred egress redirect dev vxlan0

3. tc filter add dev ens1f0_0 ingress prio 1 chain 2 proto ip flower \
    ct_state +trk+est \
    action tunnel_key set \
        src_ip 0.0.0.0 \
        dst_ip 7.7.7.8 \
        id 98 \
        dst_port 4789 \
    action mirred egress redirect dev vxlan0

Filter #1 (chain 0) decides, after initial packet classification, to send the packet to the
connection tracking module (ct action).
Once the ct_state is initialized by the CT action the packet processing continues on chain 2.

Chain 2 classifies the packet based on the ct_state.
Filter #2 matches on the +trk+new CT state while filter #3 matches on the +trk+est ct_state.

MLX5 Connection tracking HW offload - MLX5 driver patches
------------------------------

The MLX5 hardware model aligns with the software model by realizing a multi-table
architecture. In SW the TC CT action sets the CT state on the skb. Similarly,
HW sets the CT state on a HW register. Driver gets this CT state while offloading
a tuple with a new ct_metadata action that provides it.

Matches on ct_state are translated to HW register matches.

TC filter with CT action broken to two rules, a pre_ct rule, and a post_ct rule.
pre_ct rule:
   Inserted on the corrosponding tc chain table, matches on original tc match, with
   actions: any pre ct actions, set fte_id, set zone, and goto the ct table.
   The fte_id is a register mapping uniquely identifying this filter.
post_ct_rule:
   Inserted in a post_ct table, matches on the fte_id register mapping, with
   actions: counter + any post ct actions (this is usally 'goto chain X')

post_ct table is a table that all the tuples inserted to the ct table goto, so
if there is a tuple hit, packet will continue from ct table to post_ct table,
after being marked with the CT state (mark/label..)

This design ensures that the rule's actions and counters will be executed only after a CT hit.
HW misses will continue processing in SW from the last chain ID that was processed in hardware.

The following illustrates the HW model:

+-------------------+      +--------------------+    +--------------+
+ pre_ct (tc chain) +----->+ CT (nat or no nat) +--->+ post_ct      +----->
+ original match    +   |  + tuple + zone match + |  + fte_id match +  |
+-------------------+   |  +--------------------+ |  +--------------+  |
                        v                         v                    v
                     set chain miss mapping    set mark             original
                     set fte_id                set label            filter
                     set zone                  set established      actions
                     set tunnel_id             do nat (if needed)
                     do decap

To fill CT table, driver registers a CB for flow offload events, for each new
flow table that is passed to it from offloading ct actions. Once a flow offload
event is triggered on this CB, offload this flow to the hardware CT table.

Established events offload
--------------------------

Currently, act_ct maintains an FT instance per ct zone. Flow table entries
are created, per ct connection, when connections enter an established
state and deleted otherwise. Once an entry is created, the FT assumes
ownership of the entries, and manages their aging. FT is used for software
offload of conntrack. FT entries associate 5-tuples with an action list.

The act_ct changes in this patchset:
Populate the action list with a (new) ct_metadata action, providing the
connection's ct state (zone,mark and label), and mangle actions if NAT
is configured.

Pass the action's flow table instance as ct action entry parameter,
so  when the action is offloaded, the driver may register a callback on
it's block to receive FT flow offload add/del/stats events.

Netilter changes
--------------------------
The netfilter changes export the relevant bits, and add the relevant CBs
to support the above.

Applying this patchset
--------------------------

On top of current net-next ("r8169: simplify getting stats by using netdev_stats_to_stats64"),
pull Saeed's ct-offload branch, from git git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git
and fix the following non trivial conflict in fs_core.c as follows:

Then apply this patchset.

Changelog:
  v2->v3:
    Added the first two patches needed after rebasing on net-next:
     "net/mlx5: E-Switch, Enable reg c1 loopback when possible"
     "net/mlx5e: en_rep: Create uplink rep root table after eswitch offloads table"
====================
Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
parents bf3347c4 1ef3018f
......@@ -78,6 +78,16 @@ config MLX5_ESWITCH
Legacy SRIOV mode (L2 mac vlan steering based).
Switchdev mode (eswitch offloads).
config MLX5_TC_CT
bool "MLX5 TC connection tracking offload support"
depends on MLX5_CORE_EN && NET_SWITCHDEV && NF_FLOW_TABLE && NET_ACT_CT && NET_TC_SKB_EXT
default y
help
Say Y here if you want to support offloading connection tracking rules
via tc ct action.
If unsure, set to Y
config MLX5_CORE_EN_DCB
bool "Data Center Bridging (DCB) Support"
default y
......
......@@ -37,6 +37,7 @@ mlx5_core-$(CONFIG_MLX5_ESWITCH) += en_rep.o en_tc.o en/tc_tun.o lib/port_tu
lib/geneve.o en/mapping.o en/tc_tun_vxlan.o en/tc_tun_gre.o \
en/tc_tun_geneve.o diag/en_tc_tracepoint.o
mlx5_core-$(CONFIG_PCI_HYPERV_INTERFACE) += en/hv_vhca_stats.o
mlx5_core-$(CONFIG_MLX5_TC_CT) += en/tc_ct.o
#
# Core extra
......
This diff is collapsed.
/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
/* Copyright (c) 2018 Mellanox Technologies. */
#ifndef __MLX5_EN_TC_CT_H__
#define __MLX5_EN_TC_CT_H__
#include <net/pkt_cls.h>
#include <linux/mlx5/fs.h>
#include <net/tc_act/tc_ct.h>
struct mlx5_esw_flow_attr;
struct mlx5e_tc_mod_hdr_acts;
struct mlx5_rep_uplink_priv;
struct mlx5e_tc_flow;
struct mlx5e_priv;
struct mlx5_ct_flow;
struct nf_flowtable;
struct mlx5_ct_attr {
u16 zone;
u16 ct_action;
struct mlx5_ct_flow *ct_flow;
struct nf_flowtable *nf_ft;
};
#define zone_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_2,\
.moffset = 0,\
.mlen = 2,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_2) + 2,\
}
#define ctstate_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_2,\
.moffset = 2,\
.mlen = 2,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_2),\
}
#define mark_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_3,\
.moffset = 0,\
.mlen = 4,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_3),\
}
#define labels_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_4,\
.moffset = 0,\
.mlen = 4,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_4),\
}
#define fteid_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_5,\
.moffset = 0,\
.mlen = 4,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_5),\
}
#define tupleid_to_reg_ct {\
.mfield = MLX5_ACTION_IN_FIELD_METADATA_REG_C_1,\
.moffset = 0,\
.mlen = 3,\
.soffset = MLX5_BYTE_OFF(fte_match_param,\
misc_parameters_2.metadata_reg_c_1),\
}
#define TUPLE_ID_BITS (mlx5e_tc_attr_to_reg_mappings[TUPLEID_TO_REG].mlen * 8)
#define TUPLE_ID_MAX GENMASK(TUPLE_ID_BITS - 1, 0)
#if IS_ENABLED(CONFIG_MLX5_TC_CT)
int
mlx5_tc_ct_init(struct mlx5_rep_uplink_priv *uplink_priv);
void
mlx5_tc_ct_clean(struct mlx5_rep_uplink_priv *uplink_priv);
int
mlx5_tc_ct_parse_match(struct mlx5e_priv *priv,
struct mlx5_flow_spec *spec,
struct flow_cls_offload *f,
struct netlink_ext_ack *extack);
int
mlx5_tc_ct_parse_action(struct mlx5e_priv *priv,
struct mlx5_esw_flow_attr *attr,
const struct flow_action_entry *act,
struct netlink_ext_ack *extack);
struct mlx5_flow_handle *
mlx5_tc_ct_flow_offload(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow,
struct mlx5_flow_spec *spec,
struct mlx5_esw_flow_attr *attr,
struct mlx5e_tc_mod_hdr_acts *mod_hdr_acts);
void
mlx5_tc_ct_delete_flow(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow,
struct mlx5_esw_flow_attr *attr);
bool
mlx5e_tc_ct_restore_flow(struct mlx5_rep_uplink_priv *uplink_priv,
struct sk_buff *skb, u32 tupleid);
#else /* CONFIG_MLX5_TC_CT */
static inline int
mlx5_tc_ct_init(struct mlx5_rep_uplink_priv *uplink_priv)
{
return 0;
}
static inline void
mlx5_tc_ct_clean(struct mlx5_rep_uplink_priv *uplink_priv)
{
}
static inline int
mlx5_tc_ct_parse_match(struct mlx5e_priv *priv,
struct mlx5_flow_spec *spec,
struct flow_cls_offload *f,
struct netlink_ext_ack *extack)
{
return -EOPNOTSUPP;
}
static inline int
mlx5_tc_ct_parse_action(struct mlx5e_priv *priv,
struct mlx5_esw_flow_attr *attr,
const struct flow_action_entry *act,
struct netlink_ext_ack *extack)
{
return -EOPNOTSUPP;
}
static inline struct mlx5_flow_handle *
mlx5_tc_ct_flow_offload(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow,
struct mlx5_flow_spec *spec,
struct mlx5_esw_flow_attr *attr,
struct mlx5e_tc_mod_hdr_acts *mod_hdr_acts)
{
return ERR_PTR(-EOPNOTSUPP);
}
static inline void
mlx5_tc_ct_delete_flow(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow,
struct mlx5_esw_flow_attr *attr)
{
}
static inline bool
mlx5e_tc_ct_restore_flow(struct mlx5_rep_uplink_priv *uplink_priv,
struct sk_buff *skb, u32 tupleid)
{
if (!tupleid)
return true;
return false;
}
#endif /* !IS_ENABLED(CONFIG_MLX5_TC_CT) */
#endif /* __MLX5_EN_TC_CT_H__ */
......@@ -1607,6 +1607,7 @@ static int mlx5e_create_rep_root_ft(struct mlx5e_priv *priv)
}
ft_attr.max_fte = 0; /* Empty table, miss rule will always point to next table */
ft_attr.prio = 1;
ft_attr.level = 1;
rpriv->root_ft = mlx5_create_flow_table(ns, &ft_attr);
......
......@@ -55,6 +55,7 @@ struct mlx5e_neigh_update_table {
unsigned long min_interval; /* jiffies */
};
struct mlx5_tc_ct_priv;
struct mlx5_rep_uplink_priv {
/* Filters DB - instantiated by the uplink representor and shared by
* the uplink's VFs
......@@ -86,6 +87,8 @@ struct mlx5_rep_uplink_priv {
struct mapping_ctx *tunnel_mapping;
/* maps tun_enc_opts to a unique id*/
struct mapping_ctx *tunnel_enc_opts_mapping;
struct mlx5_tc_ct_priv *ct_priv;
};
struct mlx5e_rep_priv {
......
......@@ -56,6 +56,7 @@
#include "en/port.h"
#include "en/tc_tun.h"
#include "en/mapping.h"
#include "en/tc_ct.h"
#include "lib/devcom.h"
#include "lib/geneve.h"
#include "diag/en_tc_tracepoint.h"
......@@ -87,6 +88,7 @@ enum {
MLX5E_TC_FLOW_FLAG_DUP = MLX5E_TC_FLOW_BASE + 4,
MLX5E_TC_FLOW_FLAG_NOT_READY = MLX5E_TC_FLOW_BASE + 5,
MLX5E_TC_FLOW_FLAG_DELETED = MLX5E_TC_FLOW_BASE + 6,
MLX5E_TC_FLOW_FLAG_CT = MLX5E_TC_FLOW_BASE + 7,
};
#define MLX5E_TC_MAX_SPLITS 1
......@@ -193,6 +195,12 @@ struct mlx5e_tc_attr_to_reg_mapping mlx5e_tc_attr_to_reg_mappings[] = {
.soffset = MLX5_BYTE_OFF(fte_match_param,
misc_parameters_2.metadata_reg_c_1),
},
[ZONE_TO_REG] = zone_to_reg_ct,
[CTSTATE_TO_REG] = ctstate_to_reg_ct,
[MARK_TO_REG] = mark_to_reg_ct,
[LABELS_TO_REG] = labels_to_reg_ct,
[FTEID_TO_REG] = fteid_to_reg_ct,
[TUPLEID_TO_REG] = tupleid_to_reg_ct,
};
static void mlx5e_put_flow_tunnel_id(struct mlx5e_tc_flow *flow);
......@@ -1143,8 +1151,16 @@ mlx5e_tc_offload_fdb_rules(struct mlx5_eswitch *esw,
struct mlx5_flow_spec *spec,
struct mlx5_esw_flow_attr *attr)
{
struct mlx5e_tc_mod_hdr_acts *mod_hdr_acts;
struct mlx5_flow_handle *rule;
if (flow_flag_test(flow, CT)) {
mod_hdr_acts = &attr->parse_attr->mod_hdr_acts;
return mlx5_tc_ct_flow_offload(flow->priv, flow, spec, attr,
mod_hdr_acts);
}
rule = mlx5_eswitch_add_offloaded_rule(esw, spec, attr);
if (IS_ERR(rule))
return rule;
......@@ -1163,10 +1179,15 @@ mlx5e_tc_offload_fdb_rules(struct mlx5_eswitch *esw,
static void
mlx5e_tc_unoffload_fdb_rules(struct mlx5_eswitch *esw,
struct mlx5e_tc_flow *flow,
struct mlx5_esw_flow_attr *attr)
struct mlx5_esw_flow_attr *attr)
{
flow_flag_clear(flow, OFFLOADED);
if (flow_flag_test(flow, CT)) {
mlx5_tc_ct_delete_flow(flow->priv, flow, attr);
return;
}
if (attr->split_count)
mlx5_eswitch_del_fwd_rule(esw, flow->rule[1], attr);
......@@ -1938,6 +1959,11 @@ static void mlx5e_put_flow_tunnel_id(struct mlx5e_tc_flow *flow)
enc_opts_id);
}
u32 mlx5e_tc_get_flow_tun_id(struct mlx5e_tc_flow *flow)
{
return flow->tunnel_id;
}
static int parse_tunnel_attr(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow,
struct mlx5_flow_spec *spec,
......@@ -2103,6 +2129,7 @@ static int __parse_cls_flower(struct mlx5e_priv *priv,
BIT(FLOW_DISSECTOR_KEY_ENC_CONTROL) |
BIT(FLOW_DISSECTOR_KEY_TCP) |
BIT(FLOW_DISSECTOR_KEY_IP) |
BIT(FLOW_DISSECTOR_KEY_CT) |
BIT(FLOW_DISSECTOR_KEY_ENC_IP) |
BIT(FLOW_DISSECTOR_KEY_ENC_OPTS))) {
NL_SET_ERR_MSG_MOD(extack, "Unsupported key");
......@@ -2913,7 +2940,9 @@ struct ipv6_hoplimit_word {
__u8 hop_limit;
};
static bool is_action_keys_supported(const struct flow_action_entry *act)
static int is_action_keys_supported(const struct flow_action_entry *act,
bool ct_flow, bool *modify_ip_header,
struct netlink_ext_ack *extack)
{
u32 mask, offset;
u8 htype;
......@@ -2932,7 +2961,13 @@ static bool is_action_keys_supported(const struct flow_action_entry *act)
if (offset != offsetof(struct iphdr, ttl) ||
ttl_word->protocol ||
ttl_word->check) {
return true;
*modify_ip_header = true;
}
if (ct_flow && offset >= offsetof(struct iphdr, saddr)) {
NL_SET_ERR_MSG_MOD(extack,
"can't offload re-write of ipv4 address with action ct");
return -EOPNOTSUPP;
}
} else if (htype == FLOW_ACT_MANGLE_HDR_TYPE_IP6) {
struct ipv6_hoplimit_word *hoplimit_word =
......@@ -2941,15 +2976,27 @@ static bool is_action_keys_supported(const struct flow_action_entry *act)
if (offset != offsetof(struct ipv6hdr, payload_len) ||
hoplimit_word->payload_len ||
hoplimit_word->nexthdr) {
return true;
*modify_ip_header = true;
}
if (ct_flow && offset >= offsetof(struct ipv6hdr, saddr)) {
NL_SET_ERR_MSG_MOD(extack,
"can't offload re-write of ipv6 address with action ct");
return -EOPNOTSUPP;
}
} else if (ct_flow && (htype == FLOW_ACT_MANGLE_HDR_TYPE_TCP ||
htype == FLOW_ACT_MANGLE_HDR_TYPE_UDP)) {
NL_SET_ERR_MSG_MOD(extack,
"can't offload re-write of transport header ports with action ct");
return -EOPNOTSUPP;
}
return false;
return 0;
}
static bool modify_header_match_supported(struct mlx5_flow_spec *spec,
struct flow_action *flow_action,
u32 actions,
u32 actions, bool ct_flow,
struct netlink_ext_ack *extack)
{
const struct flow_action_entry *act;
......@@ -2957,7 +3004,7 @@ static bool modify_header_match_supported(struct mlx5_flow_spec *spec,
void *headers_v;
u16 ethertype;
u8 ip_proto;
int i;
int i, err;
headers_v = get_match_headers_value(actions, spec);
ethertype = MLX5_GET(fte_match_set_lyr_2_4, headers_v, ethertype);
......@@ -2972,10 +3019,10 @@ static bool modify_header_match_supported(struct mlx5_flow_spec *spec,
act->id != FLOW_ACTION_ADD)
continue;
if (is_action_keys_supported(act)) {
modify_ip_header = true;
break;
}
err = is_action_keys_supported(act, ct_flow,
&modify_ip_header, extack);
if (err)
return err;
}
ip_proto = MLX5_GET(fte_match_set_lyr_2_4, headers_v, ip_protocol);
......@@ -2998,13 +3045,24 @@ static bool actions_match_supported(struct mlx5e_priv *priv,
struct netlink_ext_ack *extack)
{
struct net_device *filter_dev = parse_attr->filter_dev;
bool drop_action, pop_action;
bool drop_action, pop_action, ct_flow;
u32 actions;
if (mlx5e_is_eswitch_flow(flow))
ct_flow = flow_flag_test(flow, CT);
if (mlx5e_is_eswitch_flow(flow)) {
actions = flow->esw_attr->action;
else
if (flow->esw_attr->split_count && ct_flow) {
/* All registers used by ct are cleared when using
* split rules.
*/
NL_SET_ERR_MSG_MOD(extack,
"Can't offload mirroring with action ct");
return -EOPNOTSUPP;
}
} else {
actions = flow->nic_attr->action;
}
drop_action = actions & MLX5_FLOW_CONTEXT_ACTION_DROP;
pop_action = actions & MLX5_FLOW_CONTEXT_ACTION_VLAN_POP;
......@@ -3021,7 +3079,7 @@ static bool actions_match_supported(struct mlx5e_priv *priv,
if (actions & MLX5_FLOW_CONTEXT_ACTION_MOD_HDR)
return modify_header_match_supported(&parse_attr->spec,
flow_action, actions,
extack);
ct_flow, extack);
return true;
}
......@@ -3826,6 +3884,13 @@ static int parse_tc_fdb_actions(struct mlx5e_priv *priv,
action |= MLX5_FLOW_CONTEXT_ACTION_COUNT;
attr->dest_chain = act->chain_index;
break;
case FLOW_ACTION_CT:
err = mlx5_tc_ct_parse_action(priv, attr, act, extack);
if (err)
return err;
flow_flag_set(flow, CT);
break;
default:
NL_SET_ERR_MSG_MOD(extack, "The offload action is not supported");
return -EOPNOTSUPP;
......@@ -4066,6 +4131,10 @@ __mlx5e_add_fdb_flow(struct mlx5e_priv *priv,
if (err)
goto err_free;
err = mlx5_tc_ct_parse_match(priv, &parse_attr->spec, f, extack);
if (err)
goto err_free;
err = mlx5e_tc_add_fdb_flow(priv, flow, extack);
complete_all(&flow->init_done);
if (err) {
......@@ -4350,7 +4419,7 @@ int mlx5e_stats_flower(struct net_device *dev, struct mlx5e_priv *priv,
goto errout;
}
if (mlx5e_is_offloaded_flow(flow)) {
if (mlx5e_is_offloaded_flow(flow) || flow_flag_test(flow, CT)) {
counter = mlx5e_tc_get_counter(flow);
if (!counter)
goto errout;
......@@ -4622,6 +4691,10 @@ int mlx5e_tc_esw_init(struct rhashtable *tc_ht)
uplink_priv = container_of(tc_ht, struct mlx5_rep_uplink_priv, tc_ht);
priv = container_of(uplink_priv, struct mlx5e_rep_priv, uplink_priv);
err = mlx5_tc_ct_init(uplink_priv);
if (err)
goto err_ct;
mapping = mapping_create(sizeof(struct tunnel_match_key),
TUNNEL_INFO_BITS_MASK, true);
if (IS_ERR(mapping)) {
......@@ -4648,6 +4721,8 @@ int mlx5e_tc_esw_init(struct rhashtable *tc_ht)
err_enc_opts_mapping:
mapping_destroy(uplink_priv->tunnel_mapping);
err_tun_mapping:
mlx5_tc_ct_clean(uplink_priv);
err_ct:
netdev_warn(priv->netdev,
"Failed to initialize tc (eswitch), err: %d", err);
return err;
......@@ -4662,6 +4737,8 @@ void mlx5e_tc_esw_cleanup(struct rhashtable *tc_ht)
uplink_priv = container_of(tc_ht, struct mlx5_rep_uplink_priv, tc_ht);
mapping_destroy(uplink_priv->tunnel_enc_opts_mapping);
mapping_destroy(uplink_priv->tunnel_mapping);
mlx5_tc_ct_clean(uplink_priv);
}
int mlx5e_tc_num_filters(struct mlx5e_priv *priv, unsigned long flags)
......@@ -4779,7 +4856,9 @@ bool mlx5e_tc_rep_update_skb(struct mlx5_cqe64 *cqe,
struct mlx5e_tc_update_priv *tc_priv)
{
#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
u32 chain = 0, reg_c0, reg_c1, tunnel_id;
u32 chain = 0, reg_c0, reg_c1, tunnel_id, tuple_id;
struct mlx5_rep_uplink_priv *uplink_priv;
struct mlx5e_rep_priv *uplink_rpriv;
struct tc_skb_ext *tc_skb_ext;
struct mlx5_eswitch *esw;
struct mlx5e_priv *priv;
......@@ -4813,6 +4892,13 @@ bool mlx5e_tc_rep_update_skb(struct mlx5_cqe64 *cqe,
}
tc_skb_ext->chain = chain;
tuple_id = reg_c1 & TUPLE_ID_MAX;
uplink_rpriv = mlx5_eswitch_get_uplink_priv(esw, REP_ETH);
uplink_priv = &uplink_rpriv->uplink_priv;
if (!mlx5e_tc_ct_restore_flow(uplink_priv, skb, tuple_id))
return false;
}
tunnel_moffset = mlx5e_tc_attr_to_reg_mappings[TUNNEL_TO_REG].moffset;
......
......@@ -94,6 +94,12 @@ void mlx5e_tc_reoffload_flows_work(struct work_struct *work);
enum mlx5e_tc_attr_to_reg {
CHAIN_TO_REG,
TUNNEL_TO_REG,
CTSTATE_TO_REG,
ZONE_TO_REG,
MARK_TO_REG,
LABELS_TO_REG,
FTEID_TO_REG,
TUPLEID_TO_REG,
};
struct mlx5e_tc_attr_to_reg_mapping {
......@@ -139,6 +145,9 @@ int alloc_mod_hdr_actions(struct mlx5_core_dev *mdev,
struct mlx5e_tc_mod_hdr_acts *mod_hdr_acts);
void dealloc_mod_hdr_actions(struct mlx5e_tc_mod_hdr_acts *mod_hdr_acts);
struct mlx5e_tc_flow;
u32 mlx5e_tc_get_flow_tun_id(struct mlx5e_tc_flow *flow);
#else /* CONFIG_MLX5_ESWITCH */
static inline int mlx5e_tc_nic_init(struct mlx5e_priv *priv) { return 0; }
static inline void mlx5e_tc_nic_cleanup(struct mlx5e_priv *priv) {}
......
......@@ -42,6 +42,7 @@
#include <linux/mlx5/vport.h>
#include <linux/mlx5/fs.h>
#include "lib/mpfs.h"
#include "en/tc_ct.h"
#define FDB_TC_MAX_CHAIN 3
#define FDB_FT_CHAIN (FDB_TC_MAX_CHAIN + 1)
......@@ -236,6 +237,7 @@ struct mlx5_esw_functions {
enum {
MLX5_ESWITCH_VPORT_MATCH_METADATA = BIT(0),
MLX5_ESWITCH_REG_C1_LOOPBACK_ENABLED = BIT(1),
};
struct mlx5_eswitch {
......@@ -390,6 +392,7 @@ enum {
enum {
MLX5_ESW_ATTR_FLAG_VLAN_HANDLED = BIT(0),
MLX5_ESW_ATTR_FLAG_SLOW_PATH = BIT(1),
MLX5_ESW_ATTR_FLAG_NO_IN_PORT = BIT(2),
};
struct mlx5_esw_flow_attr {
......@@ -420,6 +423,9 @@ struct mlx5_esw_flow_attr {
u16 prio;
u32 dest_chain;
u32 flags;
struct mlx5_flow_table *fdb;
struct mlx5_flow_table *dest_ft;
struct mlx5_ct_attr ct_attr;
struct mlx5e_tc_flow_parse_attr *parse_attr;
};
......
......@@ -324,7 +324,12 @@ mlx5_eswitch_add_offloaded_rule(struct mlx5_eswitch *esw,
if (flow_act.action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST) {
struct mlx5_flow_table *ft;
if (attr->flags & MLX5_ESW_ATTR_FLAG_SLOW_PATH) {
if (attr->dest_ft) {
flow_act.flags |= FLOW_ACT_IGNORE_FLOW_LEVEL;
dest[i].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
dest[i].ft = attr->dest_ft;
i++;
} else if (attr->flags & MLX5_ESW_ATTR_FLAG_SLOW_PATH) {
flow_act.flags |= FLOW_ACT_IGNORE_FLOW_LEVEL;
dest[i].type = MLX5_FLOW_DESTINATION_TYPE_FLOW_TABLE;
dest[i].ft = mlx5_esw_chains_get_tc_end_ft(esw);
......@@ -378,9 +383,14 @@ mlx5_eswitch_add_offloaded_rule(struct mlx5_eswitch *esw,
if (split) {
fdb = esw_vport_tbl_get(esw, attr);
} else {
fdb = mlx5_esw_chains_get_table(esw, attr->chain, attr->prio,
0);
mlx5_eswitch_set_rule_source_port(esw, spec, attr);
if (attr->chain || attr->prio)
fdb = mlx5_esw_chains_get_table(esw, attr->chain,
attr->prio, 0);
else
fdb = attr->fdb;
if (!(attr->flags & MLX5_ESW_ATTR_FLAG_NO_IN_PORT))
mlx5_eswitch_set_rule_source_port(esw, spec, attr);
}
if (IS_ERR(fdb)) {
rule = ERR_CAST(fdb);
......@@ -402,7 +412,7 @@ mlx5_eswitch_add_offloaded_rule(struct mlx5_eswitch *esw,
err_add_rule:
if (split)
esw_vport_tbl_put(esw, attr);
else
else if (attr->chain || attr->prio)
mlx5_esw_chains_put_table(esw, attr->chain, attr->prio, 0);
err_esw_get:
if (!(attr->flags & MLX5_ESW_ATTR_FLAG_SLOW_PATH) && attr->dest_chain)
......@@ -499,7 +509,7 @@ __mlx5_eswitch_del_rule(struct mlx5_eswitch *esw,
} else {
if (split)
esw_vport_tbl_put(esw, attr);
else
else if (attr->chain || attr->prio)
mlx5_esw_chains_put_table(esw, attr->chain, attr->prio,
0);
if (attr->dest_chain)
......@@ -763,14 +773,21 @@ void mlx5_eswitch_del_send_to_vport_rule(struct mlx5_flow_handle *rule)
mlx5_del_flow_rules(rule);
}
static bool mlx5_eswitch_reg_c1_loopback_supported(struct mlx5_eswitch *esw)
{
return MLX5_CAP_ESW_FLOWTABLE(esw->dev, fdb_to_vport_reg_c_id) &
MLX5_FDB_TO_VPORT_REG_C_1;
}
static int esw_set_passing_vport_metadata(struct mlx5_eswitch *esw, bool enable)
{
u32 out[MLX5_ST_SZ_DW(query_esw_vport_context_out)] = {};
u32 in[MLX5_ST_SZ_DW(modify_esw_vport_context_in)] = {};
u8 fdb_to_vport_reg_c_id;
u8 curr, wanted;
int err;
if (!mlx5_eswitch_vport_match_metadata_enabled(esw))
if (!mlx5_eswitch_reg_c1_loopback_supported(esw) &&
!mlx5_eswitch_vport_match_metadata_enabled(esw))
return 0;
err = mlx5_eswitch_query_esw_vport_context(esw->dev, 0, false,
......@@ -778,24 +795,33 @@ static int esw_set_passing_vport_metadata(struct mlx5_eswitch *esw, bool enable)
if (err)
return err;
fdb_to_vport_reg_c_id = MLX5_GET(query_esw_vport_context_out, out,
esw_vport_context.fdb_to_vport_reg_c_id);
curr = MLX5_GET(query_esw_vport_context_out, out,
esw_vport_context.fdb_to_vport_reg_c_id);
wanted = MLX5_FDB_TO_VPORT_REG_C_0;
if (mlx5_eswitch_reg_c1_loopback_supported(esw))
wanted |= MLX5_FDB_TO_VPORT_REG_C_1;
if (enable)
fdb_to_vport_reg_c_id |= MLX5_FDB_TO_VPORT_REG_C_0 |
MLX5_FDB_TO_VPORT_REG_C_1;
curr |= wanted;
else
fdb_to_vport_reg_c_id &= ~(MLX5_FDB_TO_VPORT_REG_C_0 |
MLX5_FDB_TO_VPORT_REG_C_1);
curr &= ~wanted;
MLX5_SET(modify_esw_vport_context_in, in,
esw_vport_context.fdb_to_vport_reg_c_id, fdb_to_vport_reg_c_id);
esw_vport_context.fdb_to_vport_reg_c_id, curr);
MLX5_SET(modify_esw_vport_context_in, in,
field_select.fdb_to_vport_reg_c_id, 1);
return mlx5_eswitch_modify_esw_vport_context(esw->dev, 0, false,
in, sizeof(in));
err = mlx5_eswitch_modify_esw_vport_context(esw->dev, 0, false, in,
sizeof(in));
if (!err) {
if (enable && (curr & MLX5_FDB_TO_VPORT_REG_C_1))
esw->flags |= MLX5_ESWITCH_REG_C1_LOOPBACK_ENABLED;
else
esw->flags &= ~MLX5_ESWITCH_REG_C1_LOOPBACK_ENABLED;
}
return err;
}
static void peer_miss_rules_setup(struct mlx5_eswitch *esw,
......@@ -2831,6 +2857,12 @@ bool mlx5_eswitch_is_vf_vport(const struct mlx5_eswitch *esw, u16 vport_num)
vport_num <= esw->dev->priv.sriov.max_vfs;
}
bool mlx5_eswitch_reg_c1_loopback_enabled(const struct mlx5_eswitch *esw)
{
return !!(esw->flags & MLX5_ESWITCH_REG_C1_LOOPBACK_ENABLED);
}
EXPORT_SYMBOL(mlx5_eswitch_reg_c1_loopback_enabled);
bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw)
{
return !!(esw->flags & MLX5_ESWITCH_VPORT_MATCH_METADATA);
......
......@@ -722,6 +722,36 @@ mlx5_esw_chains_get_tc_end_ft(struct mlx5_eswitch *esw)
return tc_end_fdb(esw);
}
struct mlx5_flow_table *
mlx5_esw_chains_create_global_table(struct mlx5_eswitch *esw)
{
int chain, prio, level, err;
if (!fdb_ignore_flow_level_supported(esw)) {
err = -EOPNOTSUPP;
esw_warn(esw->dev,
"Couldn't create global flow table, ignore_flow_level not supported.");
goto err_ignore;
}
chain = mlx5_esw_chains_get_chain_range(esw),
prio = mlx5_esw_chains_get_prio_range(esw);
level = mlx5_esw_chains_get_level_range(esw);
return mlx5_esw_chains_create_fdb_table(esw, chain, prio, level);
err_ignore:
return ERR_PTR(err);
}
void
mlx5_esw_chains_destroy_global_table(struct mlx5_eswitch *esw,
struct mlx5_flow_table *ft)
{
mlx5_esw_chains_destroy_fdb_table(esw, ft);
}
static int
mlx5_esw_chains_init(struct mlx5_eswitch *esw)
{
......@@ -870,6 +900,19 @@ mlx5_esw_chains_destroy(struct mlx5_eswitch *esw)
mlx5_esw_chains_cleanup(esw);
}
int
mlx5_esw_chains_get_chain_mapping(struct mlx5_eswitch *esw, u32 chain,
u32 *chain_mapping)
{
return mapping_add(esw_chains_mapping(esw), &chain, chain_mapping);
}
int
mlx5_esw_chains_put_chain_mapping(struct mlx5_eswitch *esw, u32 chain_mapping)
{
return mapping_remove(esw_chains_mapping(esw), chain_mapping);
}
int mlx5_eswitch_get_chain_for_tag(struct mlx5_eswitch *esw, u32 tag,
u32 *chain)
{
......
......@@ -25,6 +25,19 @@ mlx5_esw_chains_put_table(struct mlx5_eswitch *esw, u32 chain, u32 prio,
struct mlx5_flow_table *
mlx5_esw_chains_get_tc_end_ft(struct mlx5_eswitch *esw);
struct mlx5_flow_table *
mlx5_esw_chains_create_global_table(struct mlx5_eswitch *esw);
void
mlx5_esw_chains_destroy_global_table(struct mlx5_eswitch *esw,
struct mlx5_flow_table *ft);
int
mlx5_esw_chains_get_chain_mapping(struct mlx5_eswitch *esw, u32 chain,
u32 *chain_mapping);
int
mlx5_esw_chains_put_chain_mapping(struct mlx5_eswitch *esw,
u32 chain_mapping);
int mlx5_esw_chains_create(struct mlx5_eswitch *esw);
void mlx5_esw_chains_destroy(struct mlx5_eswitch *esw);
......
......@@ -70,6 +70,7 @@ u16 mlx5_eswitch_get_total_vports(const struct mlx5_core_dev *dev);
enum devlink_eswitch_encap_mode
mlx5_eswitch_get_encap_mode(const struct mlx5_core_dev *dev);
bool mlx5_eswitch_reg_c1_loopback_enabled(const struct mlx5_eswitch *esw);
bool mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw);
/* Reg C0 usage:
......@@ -108,6 +109,12 @@ mlx5_eswitch_get_encap_mode(const struct mlx5_core_dev *dev)
return DEVLINK_ESWITCH_ENCAP_MODE_NONE;
}
static inline bool
mlx5_eswitch_reg_c1_loopback_enabled(const struct mlx5_eswitch *esw)
{
return false;
};
static inline bool
mlx5_eswitch_vport_match_metadata_enabled(const struct mlx5_eswitch *esw)
{
......
......@@ -69,6 +69,10 @@ struct flow_match_enc_opts {
struct flow_dissector_key_enc_opts *key, *mask;
};
struct flow_match_ct {
struct flow_dissector_key_ct *key, *mask;
};
struct flow_rule;
void flow_rule_match_meta(const struct flow_rule *rule,
......@@ -111,6 +115,8 @@ void flow_rule_match_enc_keyid(const struct flow_rule *rule,
struct flow_match_enc_keyid *out);
void flow_rule_match_enc_opts(const struct flow_rule *rule,
struct flow_match_enc_opts *out);
void flow_rule_match_ct(const struct flow_rule *rule,
struct flow_match_ct *out);
enum flow_action_id {
FLOW_ACTION_ACCEPT = 0,
......@@ -136,6 +142,7 @@ enum flow_action_id {
FLOW_ACTION_SAMPLE,
FLOW_ACTION_POLICE,
FLOW_ACTION_CT,
FLOW_ACTION_CT_METADATA,
FLOW_ACTION_MPLS_PUSH,
FLOW_ACTION_MPLS_POP,
FLOW_ACTION_MPLS_MANGLE,
......@@ -224,7 +231,13 @@ struct flow_action_entry {
struct { /* FLOW_ACTION_CT */
int action;
u16 zone;
struct nf_flowtable *flow_table;
} ct;
struct {
unsigned long cookie;
u32 mark;
u32 labels[4];
} ct_metadata;
struct { /* FLOW_ACTION_MPLS_PUSH */
u32 label;
__be16 proto;
......
......@@ -16,6 +16,29 @@ struct nf_flow_rule;
struct flow_offload;
enum flow_offload_tuple_dir;
struct nf_flow_key {
struct flow_dissector_key_meta meta;
struct flow_dissector_key_control control;
struct flow_dissector_key_basic basic;
union {
struct flow_dissector_key_ipv4_addrs ipv4;
struct flow_dissector_key_ipv6_addrs ipv6;
};
struct flow_dissector_key_tcp tcp;
struct flow_dissector_key_ports tp;
} __aligned(BITS_PER_LONG / 8); /* Ensure that we can do comparisons as longs. */
struct nf_flow_match {
struct flow_dissector dissector;
struct nf_flow_key key;
struct nf_flow_key mask;
};
struct nf_flow_rule {
struct nf_flow_match match;
struct flow_rule *rule;
};
struct nf_flowtable_type {
struct list_head list;
int family;
......@@ -44,6 +67,7 @@ struct nf_flowtable {
struct delayed_work gc_work;
unsigned int flags;
struct flow_block flow_block;
struct mutex flow_block_lock; /* Guards flow_block */
possible_net_t net;
};
......@@ -129,10 +153,18 @@ struct nf_flow_route {
struct flow_offload *flow_offload_alloc(struct nf_conn *ct);
void flow_offload_free(struct flow_offload *flow);
int nf_flow_table_offload_add_cb(struct nf_flowtable *flow_table,
flow_setup_cb_t *cb, void *cb_priv);
void nf_flow_table_offload_del_cb(struct nf_flowtable *flow_table,
flow_setup_cb_t *cb, void *cb_priv);
int flow_offload_route_init(struct flow_offload *flow,
const struct nf_flow_route *route);
int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow);
void flow_offload_refresh(struct nf_flowtable *flow_table,
struct flow_offload *flow);
struct flow_offload_tuple_rhash *flow_offload_lookup(struct nf_flowtable *flow_table,
struct flow_offload_tuple *tuple);
void nf_flow_table_cleanup(struct net_device *dev);
......
......@@ -27,6 +27,7 @@ struct tcf_ct_params {
struct rcu_head rcu;
struct tcf_ct_flow_table *ct_ft;
struct nf_flowtable *nf_ft;
};
struct tcf_ct {
......@@ -50,11 +51,27 @@ static inline int tcf_ct_action(const struct tc_action *a)
return to_ct_params(a)->ct_action;
}
static inline struct nf_flowtable *tcf_ct_ft(const struct tc_action *a)
{
return to_ct_params(a)->nf_ft;
}
#else
static inline uint16_t tcf_ct_zone(const struct tc_action *a) { return 0; }
static inline int tcf_ct_action(const struct tc_action *a) { return 0; }
static inline struct nf_flowtable *tcf_ct_ft(const struct tc_action *a)
{
return NULL;
}
#endif /* CONFIG_NF_CONNTRACK */
#if IS_ENABLED(CONFIG_NET_ACT_CT)
void tcf_ct_flow_table_restore_skb(struct sk_buff *skb, unsigned long cookie);
#else
static inline void
tcf_ct_flow_table_restore_skb(struct sk_buff *skb, unsigned long cookie) { }
#endif
static inline bool is_tcf_ct(const struct tc_action *a)
{
#if defined(CONFIG_NET_CLS_ACT) && IS_ENABLED(CONFIG_NF_CONNTRACK)
......
......@@ -188,6 +188,13 @@ void flow_action_cookie_destroy(struct flow_action_cookie *cookie)
}
EXPORT_SYMBOL(flow_action_cookie_destroy);
void flow_rule_match_ct(const struct flow_rule *rule,
struct flow_match_ct *out)
{
FLOW_DISSECTOR_MATCH(rule, FLOW_DISSECTOR_KEY_CT, out);
}
EXPORT_SYMBOL(flow_rule_match_ct);
struct flow_block_cb *flow_block_cb_alloc(flow_setup_cb_t *cb,
void *cb_ident, void *cb_priv,
void (*release)(void *cb_priv))
......
......@@ -252,6 +252,19 @@ int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow)
}
EXPORT_SYMBOL_GPL(flow_offload_add);
void flow_offload_refresh(struct nf_flowtable *flow_table,
struct flow_offload *flow)
{
flow->timeout = nf_flowtable_time_stamp + NF_FLOW_TIMEOUT;
if (likely(!nf_flowtable_hw_offload(flow_table) ||
!test_and_clear_bit(NF_FLOW_HW_REFRESH, &flow->flags)))
return;
nf_flow_offload_add(flow_table, flow);
}
EXPORT_SYMBOL_GPL(flow_offload_refresh);
static inline bool nf_flow_has_expired(const struct flow_offload *flow)
{
return nf_flow_timeout_delta(flow->timeout) <= 0;
......@@ -372,6 +385,50 @@ static void nf_flow_offload_work_gc(struct work_struct *work)
queue_delayed_work(system_power_efficient_wq, &flow_table->gc_work, HZ);
}
int nf_flow_table_offload_add_cb(struct nf_flowtable *flow_table,
flow_setup_cb_t *cb, void *cb_priv)
{
struct flow_block *block = &flow_table->flow_block;
struct flow_block_cb *block_cb;
int err = 0;
mutex_lock(&flow_table->flow_block_lock);
block_cb = flow_block_cb_lookup(block, cb, cb_priv);
if (block_cb) {
err = -EEXIST;
goto unlock;
}
block_cb = flow_block_cb_alloc(cb, cb_priv, cb_priv, NULL);
if (IS_ERR(block_cb)) {
err = PTR_ERR(block_cb);
goto unlock;
}
list_add_tail(&block_cb->list, &block->cb_list);
unlock:
mutex_unlock(&flow_table->flow_block_lock);
return err;
}
EXPORT_SYMBOL_GPL(nf_flow_table_offload_add_cb);
void nf_flow_table_offload_del_cb(struct nf_flowtable *flow_table,
flow_setup_cb_t *cb, void *cb_priv)
{
struct flow_block *block = &flow_table->flow_block;
struct flow_block_cb *block_cb;
mutex_lock(&flow_table->flow_block_lock);
block_cb = flow_block_cb_lookup(block, cb, cb_priv);
if (block_cb)
list_del(&block_cb->list);
else
WARN_ON(true);
mutex_unlock(&flow_table->flow_block_lock);
}
EXPORT_SYMBOL_GPL(nf_flow_table_offload_del_cb);
static int nf_flow_nat_port_tcp(struct sk_buff *skb, unsigned int thoff,
__be16 port, __be16 new_port)
{
......@@ -494,6 +551,7 @@ int nf_flow_table_init(struct nf_flowtable *flowtable)
INIT_DEFERRABLE_WORK(&flowtable->gc_work, nf_flow_offload_work_gc);
flow_block_init(&flowtable->flow_block);
mutex_init(&flowtable->flow_block_lock);
err = rhashtable_init(&flowtable->rhashtable,
&nf_flow_offload_rhash_params);
......@@ -550,11 +608,13 @@ void nf_flow_table_free(struct nf_flowtable *flow_table)
mutex_lock(&flowtable_lock);
list_del(&flow_table->list);
mutex_unlock(&flowtable_lock);
cancel_delayed_work_sync(&flow_table->gc_work);
nf_flow_table_iterate(flow_table, nf_flow_table_do_cleanup, NULL);
nf_flow_table_iterate(flow_table, nf_flow_offload_gc_step, flow_table);
nf_flow_table_offload_flush(flow_table);
rhashtable_destroy(&flow_table->rhashtable);
mutex_destroy(&flow_table->flow_block_lock);
}
EXPORT_SYMBOL_GPL(nf_flow_table_free);
......
......@@ -232,13 +232,6 @@ static unsigned int nf_flow_xmit_xfrm(struct sk_buff *skb,
return NF_STOLEN;
}
static bool nf_flow_offload_refresh(struct nf_flowtable *flow_table,
struct flow_offload *flow)
{
return nf_flowtable_hw_offload(flow_table) &&
test_and_clear_bit(NF_FLOW_HW_REFRESH, &flow->flags);
}
unsigned int
nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb,
const struct nf_hook_state *state)
......@@ -279,8 +272,7 @@ nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb,
if (nf_flow_state_check(flow, ip_hdr(skb)->protocol, skb, thoff))
return NF_ACCEPT;
if (unlikely(nf_flow_offload_refresh(flow_table, flow)))
nf_flow_offload_add(flow_table, flow);
flow_offload_refresh(flow_table, flow);
if (nf_flow_offload_dst_check(&rt->dst)) {
flow_offload_teardown(flow);
......@@ -290,7 +282,6 @@ nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb,
if (nf_flow_nat_ip(flow, skb, thoff, dir) < 0)
return NF_DROP;
flow->timeout = nf_flowtable_time_stamp + NF_FLOW_TIMEOUT;
iph = ip_hdr(skb);
ip_decrease_ttl(iph);
skb->tstamp = 0;
......@@ -508,8 +499,7 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
sizeof(*ip6h)))
return NF_ACCEPT;
if (unlikely(nf_flow_offload_refresh(flow_table, flow)))
nf_flow_offload_add(flow_table, flow);
flow_offload_refresh(flow_table, flow);
if (nf_flow_offload_dst_check(&rt->dst)) {
flow_offload_teardown(flow);
......@@ -522,7 +512,6 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
if (nf_flow_nat_ipv6(flow, skb, dir) < 0)
return NF_DROP;
flow->timeout = nf_flowtable_time_stamp + NF_FLOW_TIMEOUT;
ip6h = ipv6_hdr(skb);
ip6h->hop_limit--;
skb->tstamp = 0;
......
......@@ -23,29 +23,6 @@ struct flow_offload_work {
struct flow_offload *flow;
};
struct nf_flow_key {
struct flow_dissector_key_meta meta;
struct flow_dissector_key_control control;
struct flow_dissector_key_basic basic;
union {
struct flow_dissector_key_ipv4_addrs ipv4;
struct flow_dissector_key_ipv6_addrs ipv6;
};
struct flow_dissector_key_tcp tcp;
struct flow_dissector_key_ports tp;
} __aligned(BITS_PER_LONG / 8); /* Ensure that we can do comparisons as longs. */
struct nf_flow_match {
struct flow_dissector dissector;
struct nf_flow_key key;
struct nf_flow_key mask;
};
struct nf_flow_rule {
struct nf_flow_match match;
struct flow_rule *rule;
};
#define NF_FLOW_DISSECTOR(__match, __type, __field) \
(__match)->dissector.offset[__type] = \
offsetof(struct nf_flow_key, __field)
......@@ -610,6 +587,7 @@ static int nf_flow_offload_tuple(struct nf_flowtable *flowtable,
if (cmd == FLOW_CLS_REPLACE)
cls_flow.rule = flow_rule->rule;
mutex_lock(&flowtable->flow_block_lock);
list_for_each_entry(block_cb, block_cb_list, list) {
err = block_cb->cb(TC_SETUP_CLSFLOWER, &cls_flow,
block_cb->cb_priv);
......@@ -618,6 +596,7 @@ static int nf_flow_offload_tuple(struct nf_flowtable *flowtable,
i++;
}
mutex_unlock(&flowtable->flow_block_lock);
return i;
}
......@@ -692,8 +671,10 @@ static void flow_offload_tuple_stats(struct flow_offload_work *offload,
FLOW_CLS_STATS,
&offload->flow->tuplehash[dir].tuple, &extack);
mutex_lock(&flowtable->flow_block_lock);
list_for_each_entry(block_cb, &flowtable->flow_block.cb_list, list)
block_cb->cb(TC_SETUP_CLSFLOWER, &cls_flow, block_cb->cb_priv);
mutex_unlock(&flowtable->flow_block_lock);
memcpy(stats, &cls_flow.stats, sizeof(*stats));
}
......
......@@ -55,7 +55,219 @@ static const struct rhashtable_params zones_params = {
.automatic_shrinking = true,
};
static struct flow_action_entry *
tcf_ct_flow_table_flow_action_get_next(struct flow_action *flow_action)
{
int i = flow_action->num_entries++;
return &flow_action->entries[i];
}
static void tcf_ct_add_mangle_action(struct flow_action *action,
enum flow_action_mangle_base htype,
u32 offset,
u32 mask,
u32 val)
{
struct flow_action_entry *entry;
entry = tcf_ct_flow_table_flow_action_get_next(action);
entry->id = FLOW_ACTION_MANGLE;
entry->mangle.htype = htype;
entry->mangle.mask = ~mask;
entry->mangle.offset = offset;
entry->mangle.val = val;
}
/* The following nat helper functions check if the inverted reverse tuple
* (target) is different then the current dir tuple - meaning nat for ports
* and/or ip is needed, and add the relevant mangle actions.
*/
static void
tcf_ct_flow_table_add_action_nat_ipv4(const struct nf_conntrack_tuple *tuple,
struct nf_conntrack_tuple target,
struct flow_action *action)
{
if (memcmp(&target.src.u3, &tuple->src.u3, sizeof(target.src.u3)))
tcf_ct_add_mangle_action(action, FLOW_ACT_MANGLE_HDR_TYPE_IP4,
offsetof(struct iphdr, saddr),
0xFFFFFFFF,
be32_to_cpu(target.src.u3.ip));
if (memcmp(&target.dst.u3, &tuple->dst.u3, sizeof(target.dst.u3)))
tcf_ct_add_mangle_action(action, FLOW_ACT_MANGLE_HDR_TYPE_IP4,
offsetof(struct iphdr, daddr),
0xFFFFFFFF,
be32_to_cpu(target.dst.u3.ip));
}
static void
tcf_ct_add_ipv6_addr_mangle_action(struct flow_action *action,
union nf_inet_addr *addr,
u32 offset)
{
int i;
for (i = 0; i < sizeof(struct in6_addr) / sizeof(u32); i++)
tcf_ct_add_mangle_action(action, FLOW_ACT_MANGLE_HDR_TYPE_IP6,
i * sizeof(u32) + offset,
0xFFFFFFFF, be32_to_cpu(addr->ip6[i]));
}
static void
tcf_ct_flow_table_add_action_nat_ipv6(const struct nf_conntrack_tuple *tuple,
struct nf_conntrack_tuple target,
struct flow_action *action)
{
if (memcmp(&target.src.u3, &tuple->src.u3, sizeof(target.src.u3)))
tcf_ct_add_ipv6_addr_mangle_action(action, &target.src.u3,
offsetof(struct ipv6hdr,
saddr));
if (memcmp(&target.dst.u3, &tuple->dst.u3, sizeof(target.dst.u3)))
tcf_ct_add_ipv6_addr_mangle_action(action, &target.dst.u3,
offsetof(struct ipv6hdr,
daddr));
}
static void
tcf_ct_flow_table_add_action_nat_tcp(const struct nf_conntrack_tuple *tuple,
struct nf_conntrack_tuple target,
struct flow_action *action)
{
__be16 target_src = target.src.u.tcp.port;
__be16 target_dst = target.dst.u.tcp.port;
if (target_src != tuple->src.u.tcp.port)
tcf_ct_add_mangle_action(action, FLOW_ACT_MANGLE_HDR_TYPE_TCP,
offsetof(struct tcphdr, source),
0xFFFF, be16_to_cpu(target_src));
if (target_dst != tuple->dst.u.tcp.port)
tcf_ct_add_mangle_action(action, FLOW_ACT_MANGLE_HDR_TYPE_TCP,
offsetof(struct tcphdr, dest),
0xFFFF, be16_to_cpu(target_dst));
}
static void
tcf_ct_flow_table_add_action_nat_udp(const struct nf_conntrack_tuple *tuple,
struct nf_conntrack_tuple target,
struct flow_action *action)
{
__be16 target_src = target.src.u.udp.port;
__be16 target_dst = target.dst.u.udp.port;
if (target_src != tuple->src.u.udp.port)
tcf_ct_add_mangle_action(action, FLOW_ACT_MANGLE_HDR_TYPE_TCP,
offsetof(struct udphdr, source),
0xFFFF, be16_to_cpu(target_src));
if (target_dst != tuple->dst.u.udp.port)
tcf_ct_add_mangle_action(action, FLOW_ACT_MANGLE_HDR_TYPE_TCP,
offsetof(struct udphdr, dest),
0xFFFF, be16_to_cpu(target_dst));
}
static void tcf_ct_flow_table_add_action_meta(struct nf_conn *ct,
enum ip_conntrack_dir dir,
struct flow_action *action)
{
struct nf_conn_labels *ct_labels;
struct flow_action_entry *entry;
enum ip_conntrack_info ctinfo;
u32 *act_ct_labels;
entry = tcf_ct_flow_table_flow_action_get_next(action);
entry->id = FLOW_ACTION_CT_METADATA;
#if IS_ENABLED(CONFIG_NF_CONNTRACK_MARK)
entry->ct_metadata.mark = ct->mark;
#endif
ctinfo = dir == IP_CT_DIR_ORIGINAL ? IP_CT_ESTABLISHED :
IP_CT_ESTABLISHED_REPLY;
/* aligns with the CT reference on the SKB nf_ct_set */
entry->ct_metadata.cookie = (unsigned long)ct | ctinfo;
act_ct_labels = entry->ct_metadata.labels;
ct_labels = nf_ct_labels_find(ct);
if (ct_labels)
memcpy(act_ct_labels, ct_labels->bits, NF_CT_LABELS_MAX_SIZE);
else
memset(act_ct_labels, 0, NF_CT_LABELS_MAX_SIZE);
}
static int tcf_ct_flow_table_add_action_nat(struct net *net,
struct nf_conn *ct,
enum ip_conntrack_dir dir,
struct flow_action *action)
{
const struct nf_conntrack_tuple *tuple = &ct->tuplehash[dir].tuple;
struct nf_conntrack_tuple target;
nf_ct_invert_tuple(&target, &ct->tuplehash[!dir].tuple);
switch (tuple->src.l3num) {
case NFPROTO_IPV4:
tcf_ct_flow_table_add_action_nat_ipv4(tuple, target,
action);
break;
case NFPROTO_IPV6:
tcf_ct_flow_table_add_action_nat_ipv6(tuple, target,
action);
break;
default:
return -EOPNOTSUPP;
}
switch (nf_ct_protonum(ct)) {
case IPPROTO_TCP:
tcf_ct_flow_table_add_action_nat_tcp(tuple, target, action);
break;
case IPPROTO_UDP:
tcf_ct_flow_table_add_action_nat_udp(tuple, target, action);
break;
default:
return -EOPNOTSUPP;
}
return 0;
}
static int tcf_ct_flow_table_fill_actions(struct net *net,
const struct flow_offload *flow,
enum flow_offload_tuple_dir tdir,
struct nf_flow_rule *flow_rule)
{
struct flow_action *action = &flow_rule->rule->action;
int num_entries = action->num_entries;
struct nf_conn *ct = flow->ct;
enum ip_conntrack_dir dir;
int i, err;
switch (tdir) {
case FLOW_OFFLOAD_DIR_ORIGINAL:
dir = IP_CT_DIR_ORIGINAL;
break;
case FLOW_OFFLOAD_DIR_REPLY:
dir = IP_CT_DIR_REPLY;
break;
default:
return -EOPNOTSUPP;
}
err = tcf_ct_flow_table_add_action_nat(net, ct, dir, action);
if (err)
goto err_nat;
tcf_ct_flow_table_add_action_meta(ct, dir, action);
return 0;
err_nat:
/* Clear filled actions */
for (i = num_entries; i < action->num_entries; i++)
memset(&action->entries[i], 0, sizeof(action->entries[i]));
action->num_entries = num_entries;
return err;
}
static struct nf_flowtable_type flowtable_ct = {
.action = tcf_ct_flow_table_fill_actions,
.owner = THIS_MODULE,
};
......@@ -80,6 +292,7 @@ static int tcf_ct_flow_table_get(struct tcf_ct_params *params)
goto err_insert;
ct_ft->nf_ft.type = &flowtable_ct;
ct_ft->nf_ft.flags |= NF_FLOWTABLE_HW_OFFLOAD;
err = nf_flow_table_init(&ct_ft->nf_ft);
if (err)
goto err_init;
......@@ -87,6 +300,7 @@ static int tcf_ct_flow_table_get(struct tcf_ct_params *params)
__module_get(THIS_MODULE);
out_unlock:
params->ct_ft = ct_ft;
params->nf_ft = &ct_ft->nf_ft;
mutex_unlock(&zones_mutex);
return 0;
......@@ -319,6 +533,7 @@ static bool tcf_ct_flow_table_lookup(struct tcf_ct_params *p,
ctinfo = dir == FLOW_OFFLOAD_DIR_ORIGINAL ? IP_CT_ESTABLISHED :
IP_CT_ESTABLISHED_REPLY;
flow_offload_refresh(nf_ft, flow);
nf_conntrack_get(&ct->ct_general);
nf_ct_set(skb, ct, ctinfo);
......@@ -1323,6 +1538,17 @@ static void __exit ct_cleanup_module(void)
destroy_workqueue(act_ct_wq);
}
void tcf_ct_flow_table_restore_skb(struct sk_buff *skb, unsigned long cookie)
{
enum ip_conntrack_info ctinfo = cookie & NFCT_INFOMASK;
struct nf_conn *ct;
ct = (struct nf_conn *)(cookie & NFCT_PTRMASK);
nf_conntrack_get(&ct->ct_general);
nf_ct_set(skb, ct, ctinfo);
}
EXPORT_SYMBOL_GPL(tcf_ct_flow_table_restore_skb);
module_init(ct_init_module);
module_exit(ct_cleanup_module);
MODULE_AUTHOR("Paul Blakey <paulb@mellanox.com>");
......
......@@ -3636,6 +3636,7 @@ int tc_setup_flow_action(struct flow_action *flow_action,
entry->id = FLOW_ACTION_CT;
entry->ct.action = tcf_ct_action(act);
entry->ct.zone = tcf_ct_zone(act);
entry->ct.flow_table = tcf_ct_ft(act);
} else if (is_tcf_mpls(act)) {
switch (tcf_mpls_action(act)) {
case TCA_MPLS_ACT_PUSH:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment