Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[BrianTRice@gmail.com]

In the course of working on a project, I realized I needed to finish an idea I've been hacking on for a while, to turn Zero logs into a tabular format that can be processed by commodity software into data visualizations and analytics.


Zero logs are just event sequences, so right now this is a bit rough: battery data only shows up on events from the BMS, and ride data are on separate events. So it doesn't plot easily yet, but the CSV/TSV formats are working well and I pull out all of the columns that Zero puts into "key: value key: value" sequences, and categorize the events and their attributes.



https://github.com/zero-motorcycle-community/zero-log-data-extractor


Here's some example output:

entry,timestamp,component,event_type,event,PackTemp (h),PackTemp (l),Vpack,BattAmps,MotTemp,AmbTemp,vmod,minsys,vcap,MinCell,PackSOC,MbbChgEn,serial,Error Code,Bmvolts,Amps
1,2018-05-21 21:12:20,MBB,,Disarmed,21,20,113.044,2,26,20,,,,,,,,,,
2,2018-05-13 10:06:37,Controller,,Sevcon CAN Link Down,,,,,,,,,,,,,,,,
3,2018-05-13 10:06:38,Controller,,Sevcon CAN Link Up,,,,,,,,,,,,,,,,
4,2018-05-13 10:06:40,MBB,,Contactor Welded,,,,,,,,,,,,,,,,
5,2018-05-13 10:06:40,Controller,OFF,Sevcon Turned Off,,,,,,,,,,,,,,,,
6,2018-05-13 10:06:42,Controller,ON,Sevcon Turned On,,,,,,,,,,,,,,,,
7,2018-05-13 10:06:42,Controller,,Sevcon CAN Link Down,,,,,,,,,,,,,,,,
8,2018-05-13 10:06:43,Controller,,Sevcon CAN Link Up,,,,,,,,,,,,,,,,
9,2018-05-13 10:06:43,Controller,DEBUG,Sevcon Contactor Drive ON.,,,,,,,,,,,,,,,,
10,2018-05-13 10:06:43,BMS,,Module 00,,,,,,,93.175,93.197,86.75,,,,,,,
11,2018-05-13 10:06:43,BMS,DEBUG,Module 00 Contactor is now Closed,,,,,,,,,,,,,,,,
12,2018-05-13 10:06:43,MBB,INFO, Enabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,
13,2018-05-13 10:10:25,MBB,DISCONNECTED,External Chg 0 Charger 2 Disconnected,,,,,,,,,,,,,,,,
14,2018-05-13 10:10:25,BMS,DEBUG,Module scheme changed from Charging mode to Stopped mode,,,,,,,,,,,,,,,,
15,2018-05-13 10:10:25,BMS,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,
16,2018-05-13 10:10:32,BMS,DEBUG,Module scheme changed from Stopped mode to Running mode,,,,,,,,,,,,,,,,
17,2018-05-13 10:10:32,BMS,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,
18,2018-05-13 10:10:35,MBB,RIDING,Riding,37,36,93.271,1,43,18,,,,,,,,,,
19,2018-05-13 10:10:35,MBB,,Batt Dischg Cur Limited,,,,,,,,,,3.28,,,,,,
20,2018-05-13 10:10:42,BMS,DEBUG,Module scheme changed from Running mode to Stopped mode,,,,,,,,,,,,,,,,
21,2018-05-13 10:10:42,BMS,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,
22,2018-05-13 10:11:04,MBB,CONNECTED,External Chg 0 Charger 2 Connected,,,,,,,,,,,,,,,,
23,2018-05-13 10:11:04,BMS,DEBUG,Module scheme changed from Stopped mode to Charging mode,,,,,,,,,,,,,,,,
24,2018-05-13 10:11:04,BMS,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,
25,2018-05-13 10:11:04,MBB,INFO, Disabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,
26,2018-05-13 10:11:05,MBB,INFO, Enabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,
27,2018-05-13 10:11:15,MBB,CHARGING,Charging,37,36,,-63,,,,,,,9%,Yes,,,,
28,2018-05-13 10:11:55,MBB,OFF,Key Off,,,,,,,,,,,,,,,,
29,2018-05-13 10:21:15,MBB,CHARGING,Charging,37,36,,-86,,,,,,,21%,Yes,,,,
30,2018-05-13 10:31:15,MBB,CHARGING,Charging,38,37,,-86,,,,,,,32%,Yes,,,,
31,2018-05-13 10:41:15,MBB,CHARGING,Charging,39,38,,-83,,,,,,,44%,Yes,,,,
32,2018-05-13 10:51:15,MBB,CHARGING,Charging,42,40,,-82,,,,,,,55%,Yes,,,,
33,2018-05-13 11:01:15,MBB,CHARGING,Charging,44,43,,-78,,,,,,,66%,Yes,,,,


What I'll try to do next is segment the data where there are charge sequences and ride sequences, so that the events are somewhat blended or interpolated. At the very least, each charge section and ride section could get an identifier so it's easier to extract one sequence and plot or analyze that, without writing a lot of time-range logic.


Any comments/suggestions?

[BrianTRice@gmail.com]

I've updated the tool to extract more fields reliably, and added a small test suite so I can keep track of all the behavior as I make changes.

Here's the example above, updated through the new tool revision:

entry,timestamp,component,event_type,event_level,event,PackTemp (h),PackTemp (l),Vpack,BattAmps,MotTemp,AmbTemp,Odo,vmod,minsys,vcap,prechg,MinCell,MaxPackTemp,PackSOC,MbbChgEn,BmsChgEn,batt curr,Reset,serial,Code,Error Code,Data,Bmvolts,Amps,RPM,CapV
1,2018-05-21 21:12:20,MBB,,,Disarmed,21,20,113.044,2,26,20,48809km,,,,,,,,,,,,,,,,,,,
2,2018-05-13 10:06:37,Controller,DISCONNECTED,,Sevcon CAN Link Down,,,,,,,,,,,,,,,,,,,,,,,,,,
3,2018-05-13 10:06:38,Controller,CONNECTED,,Sevcon CAN Link Up,,,,,,,,,,,,,,,,,,,,,,,,,,
4,2018-05-13 10:06:40,MBB,,,Contactor Welded,,,,,,,,,,,,,,,,,,,,,,,,,,
5,2018-05-13 10:06:40,Controller,OFF,,Sevcon Turned Off,,,,,,,,,,,,,,,,,,,,,,,,,,
6,2018-05-13 10:06:42,Controller,ON,,Sevcon Turned On,,,,,,,,,,,,,,,,,,,,,,,,,,
7,2018-05-13 10:06:42,Controller,DISCONNECTED,,Sevcon CAN Link Down,,,,,,,,,,,,,,,,,,,,,,,,,,
8,2018-05-13 10:06:43,Controller,CONNECTED,,Sevcon CAN Link Up,,,,,,,,,,,,,,,,,,,,,,,,,,
9,2018-05-13 10:06:43,Controller,,DEBUG,Sevcon Contactor Drive ON.,,,,,,,,,,,,,,,,,,,,,,,,,,
10,2018-05-13 10:06:43,Battery,,,Module 00,,,,,,,,93.175,93.197,86.75,93.10437349074323%,,,,,,,,,,,,,,,
11,2018-05-13 10:06:43,Battery,,DEBUG,Module 00 Contactor is now Closed,,,,,,,,,,,,,,,,,,,,,,,,,,
12,2018-05-13 10:06:43,External Charger,,INFO,Enabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,,,,,,,,,,,
13,2018-05-13 10:10:25,External Charger,DISCONNECTED,,External Chg 0 Charger 2 Disconnected,,,,,,,,,,,,,,,,,,,,,,,,,,
14,2018-05-13 10:10:25,Battery,,DEBUG,Module scheme changed from Charging mode to Stopped mode,,,,,,,,,,,,,,,,,,,,,,,,,,
15,2018-05-13 10:10:25,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,
16,2018-05-13 10:10:32,Battery,,DEBUG,Module scheme changed from Stopped mode to Running mode,,,,,,,,,,,,,,,,,,,,,,,,,,
17,2018-05-13 10:10:32,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,
18,2018-05-13 10:10:35,MBB,RIDING,,Riding,37,36,93.271,1,43,18,46213km,,,,,,,,,,,,,,,,,,,
19,2018-05-13 10:10:35,MBB,LIMIT,,Batt Dischg Cur Limited    105 A (15.217391304347826%),,,,,,,,,,,,,3.28,37,,,,,,,,,,,,,
20,2018-05-13 10:10:42,Battery,,DEBUG,Module scheme changed from Running mode to Stopped mode,,,,,,,,,,,,,,,,,,,,,,,,,,
21,2018-05-13 10:10:42,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,
22,2018-05-13 10:11:04,External Charger,CONNECTED,,External Chg 0 Charger 2 Connected,,,,,,,,,,,,,,,,,,,,,,,,,,
23,2018-05-13 10:11:04,Battery,,DEBUG,Module scheme changed from Stopped mode to Charging mode,,,,,,,,,,,,,,,,,,,,,,,,,,
24,2018-05-13 10:11:04,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,
25,2018-05-13 10:11:04,External Charger,,INFO,Disabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,,,,,,,,,,,
26,2018-05-13 10:11:05,External Charger,,INFO,Enabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,,,,,,,,,,,
27,2018-05-13 10:11:15,MBB,CHARGING,,Charging,37,36,,-63,,,,,,,,,,9%,Yes,No,,,,,,,,,,
28,2018-05-13 10:11:55,MBB,OFF,,Key Off,,,,,,,,,,,,,,,,,,,,,,,,,,
29,2018-05-13 10:21:15,MBB,CHARGING,,Charging,37,36,,-86,,,,,,,,,,21%,Yes,No,,,,,,,,,,
30,2018-05-13 10:31:15,MBB,CHARGING,,Charging,38,37,,-86,,,,,,,,,,32%,Yes,No,,,,,,,,,,
31,2018-05-13 10:41:15,MBB,CHARGING,,Charging,39,38,,-83,,,,,,,,,,44%,Yes,No,,,,,,,,,,
32,2018-05-13 10:51:15,MBB,CHARGING,,Charging,42,40,,-82,,,,,,,,,,55%,Yes,No,,,,,,,,,,
33,2018-05-13 11:01:15,MBB,CHARGING,,Charging,44,43,,-78,,,,,,,,,,66%,Yes,No,,,,,,,,,,


Here's a highlight of the changes in this example:
https://github.com/zero-motorcycle-community/zero-log-data-extractor/commit/aa3c2bfd1b4331833e794da79e7c73ea6c5770a8

[Hans2183]

So this is working upon the output from this project then I suppose? https://github.com/zero-motorcycle-community/zero-log-parser

Haven't tried it yet but I understood this is not working for SRF. Do you know if someone is working on that?

[BrianTRice@gmail.com]

So this is working upon the output from this project then I suppose? https://github.com/zero-motorcycle-community/zero-log-parser

Haven't tried it yet but I understood this is not working for SRF. Do you know if someone is working on that?

Yes, that's correct, or any of the other log parsers including Zero's official proprietary log decoder.

There is no SRF log parser yet that Zero doesn't control, and some engineers unofficially indicated an unwillingness to share decoded SRF logs with us. I'm inclined to dispute that on a number of legal grounds.


FWIW, I have been shown decoded SRF logs, and they are substantially different even structurally (what counts as a log event is on a different level of detail), so a data extraction tool for that would probably be a different script entirely.

[BrianTRice@gmail.com]

As of changes I pushed today, I've filled out the JSON export functionality, addressed data field extraction from every known MBB message, and started testing and improving the functionality for BMS logs.

As usual in my line of work, converting to tabular form was its own kind of QA loop, where I can summarize a data range or distribution to quickly find parsing deficiencies.

My preferred tool for working with the JSON is jq, a scripting tool, but any program can deal with it in a neater way than the CSV.

Now I'll reevaluate how well I can make the results interpolate better so that the ride and charge events show continuous state transitions instead of divvy up the data between alternating events, or events in BMS vs MBB logs.

The goal after that will be to stitch/join data from other sensors (gps, weather) to make a coherent picture of the motorcycle as a system as it rides or charges.

[BrianTRice@gmail.com]

I've published a set of visualizations of a week's worth of logs on my 2016 DSR. This is for a week where I traveled from SF down to LA, rode around a while, and then returned.

I used Tableau because I know it (even worked on it for a few years) but this is just an exploratory phase while I work out how best to use the data:

https://public.tableau.com/profile/brian.t.rice1320#!/vizhome/ZeroMotorcyclesExampleLogAnalysis/RideDataTimeline

[BrianTRice@gmail.com]

I've updated the data extractor again to capture more types of events (thanks partly to someone who sent in high-resolution data with a 6kW Charge Tank).

I've also extended the data format with "segment" data, which identifies contactor-closed time segments with the activity that happens within them, like riding or charging. This makes it much easier to group the events by the activity involved. I'm working on better demonstrations of what to do with this, but the examples on the Github page have been updated thus:


entry,segment_id,segment_activity,timestamp,component,event_type,event_level,event,vmod,maxsys,minsys,diff,vcap,prechg,Module,PackTemp (h),PackTemp (l),PackSOC,Vpack,MotAmps,BattAmps,Mods,MotTemp,CtrlTemp,AmbTemp,MotRPM,Odo,MinCell,MaxPackTemp,MbbChgEn,BmsChgEn,batt curr,Reset,serial,ImpedanceKOhms,Cell,PV,Allowed diff,pack cap,PackTemp h,l,lcell,Max charge,max discharge,Code,Error Reg,Error Code,Data,Bmvolts,Cmvolts,Amps,RPM,CapV
1,0,STOPPED,2018-05-13 10:06:43,Controller,,DEBUG,Sevcon Contactor Drive ON.,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
2,1,STARTED,2018-05-13 10:06:43,Battery,,,Module Closing Contactor,93.175,93.197,93.197,0.000,86.750,93%,00,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
3,2,STARTED,2018-05-13 10:06:43,Battery,,,Module Closing Contactor,93.175,93.197,93.197,0.000,86.750,93%,00,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
4,2,STARTED,2018-05-13 10:06:43,Battery,,DEBUG,Module Contactor is now Closed,,,,,,,00,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
5,2,STARTED,2018-05-13 10:06:43,External Charger,ENABLING,INFO,Enabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
6,2,STARTED,2018-05-13 10:10:25,External Charger,DISCONNECTED,,External Chg 0 Charger 2 Disconnected,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
7,2,STARTED,2018-05-13 10:10:25,Battery,,DEBUG,Module scheme changed from Charging mode to Stopped mode,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
8,2,STARTED,2018-05-13 10:10:25,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
9,2,STARTED,2018-05-13 10:10:32,Battery,,DEBUG,Module scheme changed from Stopped mode to Running mode,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
10,2,STARTED,2018-05-13 10:10:32,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
11,3,RIDING,2018-05-13 10:10:35,MBB,RIDING,,Riding,,,,,,,,37,36,9%,93.271,108,1,10,43,23,18,0,46213km,,,,,,,,,,,,,,,,,,,,,,,,,,
12,3,RIDING,2018-05-13 10:10:35,MBB,LIMIT,,Batt Dischg Cur Limited,,,,,,,,,,15.217391304347826%,,,105,,,,,,,3.28,37,,,,,,,,,,,,,,,,,,,,,,,,
13,3,RIDING,2018-05-13 10:10:42,Battery,,DEBUG,Module scheme changed from Running mode to Stopped mode,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
14,3,RIDING,2018-05-13 10:10:42,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
15,3,RIDING,2018-05-13 10:11:04,External Charger,CONNECTED,,External Chg 0 Charger 2 Connected,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
16,4,CHARGING,2018-05-13 10:11:04,Battery,CHARGING,DEBUG,Module scheme changed from Stopped mode to Charging mode,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
17,4,CHARGING,2018-05-13 10:11:04,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
18,4,CHARGING,2018-05-13 10:11:04,External Charger,DISABLING,INFO,Disabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
19,4,CHARGING,2018-05-13 10:11:05,External Charger,ENABLING,INFO,Enabling External Chg 0 Charger 2,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
20,4,CHARGING,2018-05-13 10:11:15,MBB,CHARGING,,Charging,,,,,,,,37,36,9%,94.750,,-63,01,,,18,,,,,Yes,No,,,,,,,,,,,,,,,,,,,,,,
21,4,CHARGING,2018-05-13 10:11:55,MBB,OFF,,Key Off,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
22,4,CHARGING,2018-05-13 10:21:15,MBB,CHARGING,,Charging,,,,,,,,37,36,21%,101.313,,-86,01,,,19,,,,,Yes,No,,,,,,,,,,,,,,,,,,,,,,
23,4,CHARGING,2018-05-13 10:31:15,MBB,CHARGING,,Charging,,,,,,,,38,37,32%,103.378,,-86,01,,,20,,,,,Yes,No,,,,,,,,,,,,,,,,,,,,,,
24,4,CHARGING,2018-05-13 10:41:15,MBB,CHARGING,,Charging,,,,,,,,39,38,44%,104.692,,-83,01,,,20,,,,,Yes,No,,,,,,,,,,,,,,,,,,,,,,
25,4,CHARGING,2018-05-13 10:51:15,MBB,CHARGING,,Charging,,,,,,,,42,40,55%,107.101,,-82,01,,,21,,,,,Yes,No,,,,,,,,,,,,,,,,,,,,,,
26,4,CHARGING,2018-05-13 11:01:15,MBB,CHARGING,,Charging,,,,,,,,44,43,66%,110.499,,-78,01,,,22,,,,,Yes,No,,,,,,,,,,,,,,,,,,,,,,
27,4,CHARGING,2018-05-13 11:09:02,MBB,ON,,Key On,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
28,4,CHARGING,2018-05-13 11:11:15,MBB,CHARGING,,Charging,,,,,,,,45,45,77%,114.291,,-75,01,,,22,,,,,Yes,No,,,,,,,,,,,,,,,,,,,,,,
29,4,CHARGING,2018-05-13 11:15:05,External Charger,DISCONNECTED,,External Chg 0 Charger 2 Disconnected,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
30,4,CHARGING,2018-05-13 11:15:05,Battery,,DEBUG,Module scheme changed from Charging mode to Stopped mode,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
31,4,CHARGING,2018-05-13 11:15:05,Battery,,DEBUG,Module mode Change Does Not Require Disconnect,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
32,4,CHARGING,2018-05-13 11:15:15,MBB,,,Disarmed,,,,,,,,45,45,80%,114.333,0,0,01,29,20,22,0,46213km,,,,,,,,,,,,,,,,,,,,,,,,,,
33,4,CHARGING,2018-05-13 11:15:15,Battery,,DEBUG,Module scheme changed from Stopped mode to Running mode,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

[BrianTRice@gmail.com]

I wrote a Python version of the VIN decoder which takes a little more effort to sort out the vehicle's capabilities:

./decode_vin.py 538SDDZ57KCBxxxxx
Manufacturer:   Zero Motorcycles
Plant Location:   Santa Cruz, CA
Year:   2019
Platform:   SDS
Model:   DS
Motor Power:   21kW
Motor Size:   75-7
Pack Capacity:   14.4





./decode_vin.py 538SD9Z37GCGxxxxx
Manufacturer:   Zero Motorcycles
Plant Location:   Santa Cruz, CA
Year:   2016
Platform:   SDS
Model:   DSR
Motor Power:   16kW
Motor Size:   75-7R
Pack Capacity:   13.0



I've also fixed up the Python log parser to run more reliably on various formats, AND to parse 2019 logs which had a different header version and format.