powerpc/powernv: Fix kexec races going back to OPAL

powerpc/powernv: Fix kexec races going back to OPAL

We have a subtle race when sending CPUs back to OPAL on kexec.

We mark them as "in real mode" right before we send them down. Once
we've booted the new kernel, it might try to call opal_reinit_cpus()
to change endianness, and that requires all CPUs to be spinning inside
OPAL.

However there is no synchronization here and we've observed cases
where the returning CPUs hadn't established their new state inside
OPAL before opal_reinit_cpus() is called, causing it to fail.

The proper fix is to actually wait for them to go down all the way
from the kexec'ing kernel.

Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>