48V at 50A, thats not very much current, did you mean Ah? Make sure you're not confusing units. Ah is capacity.
Your controller will not do 650A continuous, thats peak.
But even more important, Battery current is never equal to motor current.
Power in is basically equal to power out + controller losses, but the voltage changes on the output, as does the current. You get high current and low voltage on the output for the same power, but on the input, its full voltage.
Also, that high current is at low RPM when you're pulling a large load and accelerating, but once you aren't pulling as much of a load (you've gotten to speed), your current drops. Motors also have a Torque curve, and Torque drops as RPM increases, and torque is directly proportional to input current, which drops as well. Volts is RPM, Current is Torque.
So consider it like this (ignoring controller loss for a second in order to simplify the explanation):
the controller is putting out low voltage (and low RPM), but high current (and high torque) and is giving essentially 25% of the full voltage of the pack, but 100% of the current from the controller.... lets say the pack is 72V. That means the motor is seeing about 18V @ 650A, which is 11.7kW.
On the input, at 72V, that 11.7kW is only 162.5A, nowhere near 650A.
I could go explain WHY but its been covered online quite a bit if you want to learn more about controllers, but essentially it involves the PWM of the FET's, the Capacitors and the freewheel diodes. The controller more of a power converter than a switch.
You should be fine, even if you do spike above 200A, it'l be very short. Its the continuous current rating that is important. The main thing with the contactor is continuous current, which will overheat if it is given continuous current above its limits.