I don't think the batteries particularly care if they are too cold (like a ICE engine might if not warmed up first), they will just not give you the sort of performance as you expect when they are warm. Or, to put it another way, I don't think you are doing any damage to the batteries by not warming them up before riding.
A few years ago Zero gave the Aerostitch motorcycle accessory company a new Zero FX to keep for the winter. It was ridden every day during the winter in Duluth, Minnesota, (that has one of the coldest winters in the U.S.) by the company staff. The bike was left out all night in the snow, with just an electric blanket keeping the batteries warmed up while the bike was hooked up to a charger all night. Then it was ridden home and back to work by whomever had the guts to ride it in below zero F, temperatures. The FX never failed to run and came through the ordeal (apparently) with flying colors. This event was documented in the 135-page book published by Aerostitch, titled "Zero Below Zero". I believe they are selling it in their internet catalog. I have a copy that Zero was handing out for free at their tenth anniversary party. I seem to recall that there is a thread on this forum also detailing this adventure. It would be dated sometime during the winter of 2015.