Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[BrianTRice@gmail.com]

Greg Hassler runs the online parser, which had stopped working a few months ago while he tried upgrading the core parser code to the latest Python I'd been working on while in surgery recovery.


This weekend, we figured out the problem and got the latest code deployed:
http://home.hasslers.net/zerologparser/log_parser.php


So, this is now working as intended again, with updated README instructions. See the original code (which now requires Python 3 vs Python 2 previously) if you want to run it yourself:
https://github.com/zero-motorcycle-community/zero-log-parser/


I have a better framework in place for testing and rolling out new changes to the parser, which has always been a challenge, because the test inputs are log files that belong to owners, which we don't republish or even use without permission. The test harness in that repository works for whatever logs you have on hand, then.


If you have a log file that the parser chokes on, we would love it if you let us know which file it was so we can add it to our test collection.


I believe that the 2017+ model year log header is now mostly passing, but still needs further tweaks. For SRF/SRS logs, there is a totally new format and no examples of decoded logs, so although I've worked out some basics, I need to roll out actual code to expose the contents to owners in a reasonable way.

[JaimeC]

Thank you for your hard work on this, Brian.  How's your recovery going?

[valnar]

This is fantastic news.  I just tried it.

Thanks Brian.

[BrianTRice@gmail.com]

Thank you for your hard work on this, Brian.  How's your recovery going?

Thanks. I'm waiting on my final surgery to get actual teeth implants, and unfortunately because of the state lockdown, I'm not sure when I'll get a date. But everything with insurance is clearing fine, so I'll pay out my yearly deductible and wait.

The last surgery installed bone / hardware to support teeth implants, and the pain from that has been dull but quite bearable. I can now hike fully over the hill I live on in SF, which is a considerable one, so that took a lot of practice. (My aerobic exercise is limited by doctor-mandated "no heavy lifting" instructions, because the blood vessels in my mouth are sensitive for weeks after every surgery.)

[gt13013]

Hello. Many thanks for the job!!

I compared it to my old offline parser and I can clearly see many improvements.
New information appears, that was not seen by the old parser.
Negative temperatures (in °C) gave errors in the old parser and now they are correctly decoded.

But I detected some small problems:
1) date is not correct (in fact there is a 9 hours shift in my case)
2) percentiles are given with too many digits (example : 92.11538461538461% )

More details here: https://www.dropbox.com/sh/gef8xjymn9gu8j9/AADfZY_SanxpHexPlrx86kIBa?dl=0

Gerard

 

[BrianTRice@gmail.com]

Hello. Many thanks for the job!!

I compared it to my old offline parser and I can clearly see many improvements.
New information appears, that was not seen by the old parser.
Negative temperatures (in °C) gave errors in the old parser and now they are correctly decoded.

But I detected some small problems:
1) date is not correct (in fact there is a 9 hours shift in my case)
2) percentiles are given with too many digits (example : 92.11538461538461% )

More details here: https://www.dropbox.com/sh/gef8xjymn9gu8j9/AADfZY_SanxpHexPlrx86kIBa?dl=0

Thanks! I'll speak to each point:

• I don't think we're doing anything with time that would cause mis-interpretation (timezones, probably). Were the timestamps correct before? (In which case, the timezone of the server probably interfered so I'll figure that out). If they're the same as before, then it's the internal clock that's not localized, which is something I've seen many times.
• Percentages were reworked, but yes there's no reason for more significant digits than 4. I'll make a fix for that.

[BrianTRice@gmail.com]

I found and fixed the percentage display issue; it's nice that I have a test framework where I can check what a piece of code does to about a hundred example logs that I've collected or gotten permission for.


I've informed Greg Hassler who will update the site when he gets time.

[gt13013]

I don't think we're doing anything with time that would cause mis-interpretation (timezones, probably). Were the timestamps correct before? (In which case, the timezone of the server probably interfered so I'll figure that out). If they're the same as before, then it's the internal clock that's not localized, which is something I've seen many times.

The timestamps have always been good with my previous version of the parser.
I give all the files in the dropbox link and you can check by yourself. The same log gives good timestamps with the old offline parser and bad ones with the new online parser.

In fact I just realize that the difference between the two parsers is 8 hours at the beginning of my log file, and 9 hours at the end (see the two winmerge screenshots). This is certainly due to the fact that we changed from winter time to summer time in between...

[BrianTRice@gmail.com]

Okay, I'll use those files to test and fix the bug, assuming you don't mind. Thanks for the report.

[gt13013]

Okay, I'll use those files to test and fix the bug, assuming you don't mind. Thanks for the report.

Yes, you can use my files. No problem. Thanks.

[BrianTRice@gmail.com]

I cannot figure out why or how the time handling regressed. That was one part of the processing I did not touch. It's possible that the server configuration changed, resulting in local-time being misconfigured, but the logic says to take the MBB timestamp bytes, subtract 7 hours unconditionally (making it Pacific Time Zone in Daylight Savings Time), and then treat that as UTC/GMT, which should not be susceptible to local-time issues.

The parser has a provision to interpret the bytes another way, but it's hard-configured to use the method above. Both methods do not produce what you saw before at all, and neither does removing the GMT-7 correction.

As usual, with programming around timezones, there's always some weird mystery. I'll update if I find a real answer. Maybe from Python 2 to Python 3, there's some annoyingly implicit change.

[gt13013]

the logic says to take the MBB timestamp bytes, subtract 7 hours unconditionally (making it Pacific Time Zone in Daylight Savings Time)

Hi Brian,
I don't understand that point. If it was the case, it would mean that the MBB timestamp is always UTC time. How can it be like that? The MBB has a way to synchronize with real time? It has a radio controlled clock?

Since the clock of my bike always shifts from the good time, I was supposing it has no way to synchronize with the true time.
And consequently, I was thinking that the only time known by the bike was the time entered by the user, i.e. the local time in that area.

Or perhaps all Zero models do not behave the same concerning the timestamp...

[BrianTRice@gmail.com]

The Dash / instrument cluster has its own clock and time setting. The MBB log event timestamp can be unrelated. I’m not sure of more than that right now.

I could certainly attempt to decode the log according to the system’s local time, but Greg Hassler’s server won’t necessarily provide that from the web browser/client. So I can only resolve the code that way for people running the Python code themselves.

I can think of more interesting ways to improve this, but they involve more time investment than I can budget, since I have multiple priorities even when it comes to decoding Zero data. We’ll see whether I can come up with something helpful.

[Curt]

Since the FXS has no timezone setting, the actual physical clock can only be set to local time. Zero appears to have added an arbitrary 7 hour offset to the timestamp. There is no logic in that. If they wanted it to be closer to a Unix timestamp in UTC, that would only be correct for bikes in California during the summer. So one can only subtract the mistake (7*60*60) and assume local time.

Then, for computational purposes, pretend the bike, logs, server, parser and viewer all exist in Greenwich. The script correctly does that using gmtime(). It can't use localtime() because that would be converting an already-local time to local.

I can confirm that strftime('%m/%d/%Y %H:%M:%S', gmtime(time())) produces the same output in python2 and python3 in any server time zone. (Interestingly, %Z output differs between python2 and python3.)

Maybe SR/Fs modernized clock settings with timezone or cellular or GPS? No idea.

[gt13013]

Suggestion: it could be simpler to let an optional "hour shift" parameter in the parser and the viewer.
In such a way that the interested user could add or remove some hours to adjust the result to his local time.