Revert "RDMA/rxe: Remove VLAN code leftovers from RXE"

Revert "RDMA/rxe: Remove VLAN code leftovers from RXE"

This reverts commit ce88c194200adbcae97194b6ae2f387bceb74b5e.

It's true that creating rxe on top of 802.1q interfaces doesn't work.
Thus, commit af037127e330 ("RDMA/rxe: prevent rxe creation on top of vlan
interface") was absolutely correct.

But ce88c194200a was incorrect assuming that with this change, RDMA and
VLAN don't work togehter at all. It just has to be set up
differently. Rather than creating rxe on top of the VLAN interface, rxe
must be created on top of the physical interface.  RDMA then works just
fine through VLAN interfaces on top of that physical interface, via the
"upper device" logic.

This is hard to see in the rxe logic because it never talks about vlan,
but instead rxe carefully selects upper vlan netdevices when working with
packets which in turn imply certain vlan tagging. This is all done
correctly and interacts with the gid table with VLAN support the same as
real HW does.

ce88c194200a broke this setup deliberately and should thus be
reverted. Also, ce88c194200a removed rxe_dma_device(), so adapt the revert
to discard that hunk.

Fixes: ce88c194200a ("RDMA/rxe: Remove VLAN code leftovers from RXE")
Link: https://lore.kernel.org/r/20210120161913.7347-1-mwilck@suse.com
Signed-off-by: Martin Wilck <mwilck@suse.com>
Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>