net/mlx5e: Change inline mode correctly when changing trust state

net/mlx5e: Change inline mode correctly when changing trust state

The current steps that are performed when the trust state changes, if
the channels are active:

1. The trust state is changed in hardware.

2. The new inline mode is calculated.

3. If the new inline mode is different, the channels are recreated using
the new inline mode.

This approach has some issues:

1. There is a time gap between changing trust state in hardware and
starting sending enough inline headers (the latter happens after
recreation of channels). It leads to failed transmissions and error
CQEs.

2. If the new channels fail to open, we'll be left with the old ones,
but the hardware will be configured for the new trust state, so the
interval when we can see TX errors never ends.

This patch fixes the issues above by moving the trust state change into
the preactivate hook that runs during the recreation of the channels
when no channels are active, so it eliminates the gap of partially
applied configuration. If the inline mode doesn't change with the change
of the trust state, the channels won't be recreated, just like before
this patch.

Signed-off-by: Maxim Mikityanskiy <maximmi@mellanox.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
Signed-off-by: Saeed Mahameed <saeedm@mellanox.com>