Support for Gamrin Descent MK1

[Dirk Hohndel]

Could be painful to develop your own fit parser in C from scratch...

I'm working on exactly that for libdivecomputer.
I'm actually starting to get some data, and have a structure for the
parser definition that seems not entirely unreasonable. Fingers
crossed.
But I'm missing a lot of the message ID definitions (and I haven't
gotten to a lot of the field numbers within them).

This discussion started on our user forum... I think we should move this
to the developer mailing list. Lionel, Wojciech, Sébastian, are you on this
mailing list as well? Or should we just keep copying you?

Linus, do you have specific IDs/fields that look like they have valid data
but that you are missing?

When I dumped the files using the Python library I see a ton of undefined
fields, but the vast majority seem to have no data in them...

/D

[Dirk Hohndel]

(again, moving this to the developer mailing list where I think it belongs)

I just returned from vacation where I had possibility to supplement some missing events in my FIT files. I mean – forcing the alarms, violating MODs, breaking ceilings, etc. Nothing harmful of course...

I just love that statement... "violating MODs, breaking ceilings" ... "Nothing harmful"
I assume and hope that you did this by giving your dive computer incorrect information about the gases that you were diving with :-)
Also, did you get a chance to simulate gas changes?

I dived during the day and trying to find some time to code and analyze FIT files at night. The effect is quite obvious - a lot of new information gathered and a lot of chaotic code written.

I know exactly how that feels :-)

I wanted to hardly refactor the script before showing it in public. Especially because current idea (collect the data and update Subsurface log on the fly) is stupid and causes a lot of conditional code. Additionally this is only temporary solution, created for my own needs, before support for Garmin is ready in Subsurface.
Today’s activity in this thread convinced me to share the code as is. Some developers may found interesting conditions in function “message_processor” as it contains all known for me, reverse engineered Garmin records and fields I needed to use.

Excellent. Thank you!

But after all - my script imports data from multiple fit files at once, creates new Subsurface logs or updates existing, supports sites, computers, cylinders creation, consolidates sites using defined radius around GPS coordinates, can include apnea dives, filters out dives shorter than defined time, lists FIT files summaries (to easy pick interesting ones), contains a lot of already recognized events

Some extra data I had no chance to treat other way is imported to Subsurface Extra Info tab. The most important one for me is gradient factor setting that can be set per-dive in Garmin but only globally in Subsurface.

It's globally set for the calculation that we do when you display the calculated ceiling. We do report the GF used by the dive computer in extra data if that information is available from the dive computer (and if we know how to parse it).

FIT is flexible and has consistent API. Unfortunately Garmin writes there a lot of data that is not documented. For instance, the last problem I had to resolve was time offset [s] stored in ‘device_settings’ message. It is required to calculate local time as internally Garmin uses UTC time. It turned out that it’s signed integer stored in uint32 field using “two's complement” format. I had to do a lot of tests before I recognized it.

That's the bread and butter of reverse engineering dive computer data. The Oceanic format where the bits for certain values are seemingly randomly distributed across multiple bytes appears to be one of the worst in this context...

Once (not for the current version) I tested the script also under Windows using python 3.7 and it worked fine. Among others, for this purpose there are a few changes in the code to substitute missing shell-level file wildcards expansion when called from shells as crippled as CMD.
https://github.com/xplwowi/fit2subs
<https://github.com/xplwowi/fit2subs>
You will find there also postulated source FIT files with GPS and diving activities inside.

That is outstanding. Thank you!

/D

[Wojciech Więckowski]

wt., 28 sie 2018 o 21:25 Dirk Hohndel <***@hohndel.org
<mailto:***@hohndel.org>> napisał(a):

(again, moving this to the developer mailing list where I think it
belongs)

I just love that statement... "violating MODs, breaking ceilings"
... "Nothing harmful"
I assume and hope that you did this by giving your dive computer
incorrect information about the gases that you were diving with :-)


Knowlege is precious, health priceless. It's not a problem to use two
computers and set some false data of O2 fraction on that one intended
for testing.

Also, did you get a chance to simulate gas changes?


Gas changes are present in some fit files I'm sharing. e.g. this one
2018-08-13-13-48-26.fit

There is also ready, created outside Subsurface log file, that has these
events marked:
https://github.com/xplwowi/fit2subs/blob/master/logs/out.xml

Generally finding this was easy. MK1 has 6 zero-indexed slots available
for mixes. Watch lets selecting only these that has enabled flag set.
In *event* #57 *data* field contains index of mix (cylinder) just
switched to.


@Dirk, sorry - initially replied only to you from browser where "reply
all" button was hidden. Now I'm also subscribed to the mailing list.

[Linus Torvalds]

https://github.com/xplwowi/fit2subs
You will find there also postulated source FIT files with GPS and diving activities inside.

Thanks, I'm definitely interested, because I'm getting to the point
where my FIT parser skeleton code is ready to actually maybe become
more than a skeleton.

And you shamed me into actually pushing my (INCOMPLETE!) work out too.

It's the 'garmin-descent' branch of

https://github.com/torvalds/libdc-for-dirk.git

and the way it currently works is that you build this version of
libdivecomputer together with the current git version of subsurface,
and you can download directly from your Garmin Descent.

And by "download directly" I mean "not really download at all, but
test". Because right now it doesn't actually fill any dive information
with the data, it just prints it out for my debugging purposes.

So I just capture 'stderr', and see things like this:

FILE_serial: ecee9feb
FILE_creation_time: 35de2636
FILE/7: ffffffff
FILE_manufacturer: 1
FILE_product: a2c
FILE/5: ffff
FILE_file_type: 4
FILE_CREATOR/0: 0104
FILE_CREATOR/1: ff
EVENT/253: 35de2636
EVENT/3: 00000000
EVENT/15: ffffffff
EVENT/0: 00
EVENT/1: 00
EVENT/4: 00
DEVICE_INFO/253: 35de2636
DEVICE_INFO/3: ecee9feb
DEVICE_INFO/7: ffffffff
DEVICE_INFO/8: ffffffff
...

where things like "FILE_serial" means that it actually has figured out
that it's the serial number field (although I have no idea what the
rule is for turning the value into a real Garmin serial number), but
something like "DEVICE_INFO/3" just means that I haven't even filled
in the data for field 3 for the DEVICE_INFO msg.

The parser is probably broken, and currently it doesn't even try to
handle the compressed record format because I've not actually seen one
yet.

And as mentioned, it doesn't actually fill any dive data at all, so
just do a "download" and then "cancel" after you see the dive list.
Don't actually accept it, you'll just get empty data. It's the debug
printout that is the interesting part right now.

I'll take a look at your fit files and your record numbers at some
point, but I need to take a break from looking at FIT files for now.

Linus

[Linus Torvalds]

On Tue, Aug 28, 2018 at 12:52 PM Linus Torvalds <

Post by Linus Torvalds
It's the 'garmin-descent' branch of
https://github.com/torvalds/libdc-for-dirk.git
and the way it currently works is that you build this version of
libdivecomputer together with the current git version of subsurface,
and you can download directly from your Garmin Descent.

Now the downloading actually works, and does something.

Wojciech, this is what your FIT data looks like when the current branch
downloads it directly.

[image: Garmin-5.png]

(although I don't know if images will actually show up correctly on the
google groups thing).

I don't import everything - things like gas mixes and changes (and any
events) are currently just ignored.

But the basics certainly work.

Linus

[Wojciech Więckowski]

Post by Linus Torvalds
On Tue, Aug 28, 2018 at 12:52 PM Linus Torvalds

Post by Linus Torvalds
It's the 'garmin-descent' branch of
https://github.com/torvalds/libdc-for-dirk.git
and the way it currently works is that you build this version of
libdivecomputer together with the current git version of subsurface,
and you can download directly from your Garmin Descent.

Now the downloading actually works, and does something.
Wojciech, this is what your FIT data looks like when the current
branch downloads it directly.
Garmin-5.png
(although I don't know if images will actually show up correctly on
the google groups thing).
I don't import everything - things like gas mixes and changes (and any
events) are currently just ignored.
But the basics certainly work.
                           Linus

Wow - I'm impressed. I just strarted testing dctool compiled from your
branch. In comment for commit modifying common.c I informed you that:
src/libdivecomputer.symbols has no symbol dc_usb_storage_open added.
It's not exported by libdivecomputer and linking dctools fails with
undefined reference... error.

I'm reading your commits messages and observing code changes.
There are a lot of GPS coordinates stored in LAP records but I see no
reason to support them as whole dive summary is stored in SESSION
record. Laps are created after surfacing, spending there less than 1
minute (time can be defined) and immersing again. Laps don't affect
sample timers, what is obvious as they are related to Garmin's epoch.

/WW

[Linus Torvalds]

Wow - I'm impressed. I just strarted testing dctool compiled from your branch.

Ahh. I haven't used dctool in a long while. It's of dubious use, but I
guess for testing it's fine, and it's easier to build than subsurface
is.
Ok, you seem to have done it on the github comment system, so I missed it.

I don't actually use github that way - I have all notifications turned
off, because they are way too noisy for me. So I basically use github
as a hosting place, without ever looking at the other things github
offers.

And yeah, I have no idea how src/libdivecomputer.symbols is supposed
to work, I think it's some odd Windows thing, so I missed it entirely.

Is that the only symbol that needs adding?

I'm reading your commits messages and observing code changes.

I'm done for a while. It's not perfect, but it is "good enough" for
me, and I'm traveling a bit the rest of the week.

So I might get back to it to finish up some odds and ends on the
weekend, but for now it is what it is. Comments welcome, but it's
probably best to just email me directly than to use the github comment
system. Or comment on github, but send me an email telling me to go
look.

There are a lot of GPS coordinates stored in LAP records but I see no
reason to support them as whole dive summary is stored in SESSION
record. Laps are created after surfacing, spending there less than 1
minute (time can be defined) and immersing again. Laps don't affect
sample timers, what is obvious as they are related to Garmin's epoch.

As you'll find out, I wrote the parser basically by just looking at
what fields I could find in your and Dirk's FIT files, and then
looking at patterns.

I didn't even filter by activity, which is why it currently happily
creates a "dive" even from non-dive activity. It won't have any actual
*dive* data, but it can have random other data - like GPS points.

So there's definitely a fair amount of "cleanup and polishing" for
this to be a *good* Garmin downloader, but it seems to mostly work
already.

I'll be diving for a few days in a week or two, and then I'll actually
get to test the Garmin Descent myself. Right now I've just been
working off other peoples data.

Linus

[Dirk Hohndel]

A first test version that includes support for downloading (via USB, on a computer) from the Garmin Descent is available.
There are a few shortcomings, still, but I think it's reasonably functional already. The two biggest issues are
a) no events are downloaded - this is simply code missing in our parsing algorithm and should be added reasonably soon.
b) the downloader right now offers ALL activities, it doesn't filter for dive activities. So unless you remember when your dives started, you may end up with weird super short/shallow "dives" in your dive log. Again, something I expect us to fix pretty soon.

You can find binaries here: https://github.com/Subsurface-divelog/subsurface/releases/tag/continuous <https://github.com/Subsurface-divelog/subsurface/releases/tag/continuous>

For Windows simply grab the subsurface-4.8.1-...exe - that's an installer
For Mac we don't have a signed DMG as we do for releases, instead there's a Subsurface-4.8.1-...app.zip - which you can download, unpack, and then manually run (your Mac may complain about it being unsigned, depending on your security settings... worst case you may have to run the Contents/MacOS/Subsurface file directly
For Linux there's an AppImage there.

I'd be very interested in feedback how this works out for you.

Thanks

/D

[Linus Torvalds]

Post by Dirk Hohndel
I'd be very interested in feedback how this works out for you.

NOTE! If somebody finds oddities, parse errors, has some case that
doesn't work at all, etc, please do send me and the list

- a description of what the expected result is (ie "the Garmin app
says the max depth was 76 ft)

- a description of what subsurface did wrong (ie "subsurface reports
that dive as a 401 mile depth dive")

- the "Activity" file that matches the dive that got mis-parsed. Or
just all your activity files from the dive computer, and I'll figure
it out.

but as Dirk says, the current downloader is known to be incomplete wrt
event information and has that oddity of "all activities are recorded
as dives".

If you *do* have some event you care deeply about, and that you
actually know is there in the dive file, you can also send me that
kind of "this thing XYZ is important to me" information. It's likely
really easy to add certain event parsing, I just got fed up at looking
at FIT files, and my motivation to look at them was pretty low. BNut
if you point out particular events and have an example FIT file for
them, I'll be motivated to add at least those ones.

In a couple of weeks, I should have dive data of my own, and I'll
probably finish this off then.

And yes, while the downloader works, if you use your Garmin for other
things too, you will see silly 0-minute dives etc without a real
profile for things that are supposed to be
running/walking/cycling/whatever activities when you download.

For now, just uncheck them from the dive download list. They'll
usually stand out as being zero minutes long and having a zero depth
(although with swimming, I think you'll actually get a really boring
"dive" profile).

Linus