mlx4_core: Fix integer overflows so 8TBs of memory registration works

mlx4_core: Fix integer overflows so 8TBs of memory registration works

This patch adds on the fixes done in commits 6761f9e8708b ("mlx4_core:
Allow large mlx4_buddy bitmaps") and c8556eaff3dc ("mlx4_core: Fix
integer overflow issues around MTT table") so that memory registration
of up to 8TB (log_num_mtt=31) finally works.

It fixes integer overflows in a few mlx4_table_yyy routines in icm.c
by using a u64 intermediate variable, and int/uint issues that caused
table indexes to become nagive by setting some variables to be u32
instead of int.  These problems cause crashes when a user attempted to
register > 512GB of RAM.

Signed-off-by: Yishai Hadas <yishaih@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: Roland Dreier <roland@purestorage.com>