This is totally ok and completely normal behavior of cold batteries. It will accelerate quick at slow speeds where power is low and torque is high but limit acceleration at higher speeds and limit the top speed you can go. It is amplified the lower the SOC is. If the bike sits for weeks and you know you want to ride it in a few hours, plug it in and top it off. That easy. But you are doing the right thing keeping it not fully charged while sitting for long periods weeks at a time.
When the batteries are kept cold and at a low state of charge, they lack the ability to deliver large amounts of power. But this is also exactly what helps extend their calendar life. When you want to ride, top it off and even bring it inside overnight to warm it up if you want. But remember, the conditions that give you the most power (full charge, above 100F) are also the same conditions that degrade the battery the fastest.
Actual charge cycles and fast charging rates have very little to do with battery life compared to sitting at high SOC and/or hot temps.