DS 1.2.0 Trace CSV Format not compatible with Excel

IoT Modules Development tools
1

Ref: viewtopic.php?f=108&t=5123&start=0

There’s a few issues with the DS 1.2.0 Traces CSV format that make it incompatible with MS Excel (and, I suspect, other tools also):

1. Date format

Because all fields are 2 digits, Excel can’t tell which is the year (human readers may also get it wrong due to US/RoW confusion about mm/dd or dd/mm)

A 4-digit year needs to be used.

2. Time format

Excel doesn’t recognise this as a time, because of the colon between seconds & decimal seconds.
It needs to be a decimal point here.

3. Time width
The width of the decimal seconds varies between 1 & 3 digits - which breaks the tabulation, and makes the plain text file hard to read.; eg,

11/02/04,15:32:55:984 - 137;ADL;7;[L3] same PLMN
11/02/04,15:32:55:98 - 138;ADL;7;02 03 04 01 00 FF 
11/02/04,15:32:56:0 - 001;ADL;7;[L3] plmn_change_timer_cb
11/02/04,15:32:56:0 - 002;L3RR;3;==>C RRC_TIM_PERIODIC_SI
11/02/04,15:32:56:15 - 003;L3RR;3;<==C PH_BCCH_REQ on 27
11/02/04,15:32:56:150 - 004;L3RR;3;C: S4 P0 N1 M10

This also means that the above problems can’t be worked around by importing as fixed-width fields.

4. Quoting
The Trace strings themselves need to be “quoted” in some way - so that semicolons within the text are not takn as delimiters!
Or, again, the preceding fields need to be fixed width.

2

Correct me if I’m wrong, but all of this can be configured in the DS preferences: Window > Preferences > Developer Studio > Target Management > Traces

3

Oh yes - so it is! :blush:

4

When the traces are displayed in the Studio, it manages to get each one on a single row.

But AT responses in the recorded file get broken across lines; eg,

11/02/04,15:47:58:93 - 007;ADL;7;l3_info_rsm_cb
11/02/04,15:47:58:109 - 008;ADL;7;Chan Lev BS MCC MNC  LAC CI
11/02/04,15:47:58:109 - 009;ADL;7;  75  15 60 234 15F 00BF 1588
11/02/04,15:47:58:109 - 010;ADL;7;RSM end.
11/02/04,15:47:58:125 - 011;ATI;1;ati_SndUnsolicitedAtRsp: Len=12
11/02/04,15:47:58:125 - 012;ATI;1;
+CREG: 2

11/02/04,15:47:58:125 - 013;ATI;1;ati_SndUnsolicitedAtRsp: Len=13
11/02/04,15:47:58:125 - 014;ATI;1;
+CGREG: 0

11/02/04,15:47:58:140 - 015;ADL;1;+CREG: 2 state received 
11/02/04,15:47:58:140 - 016;ADL;1;Force RSSI=99
11/02/04,15:47:58:140 - 017;ADL;1;+CREG: 2 Change registration state from 5
5

We already have logged an evolution request to handle “special characters” in trace strings.
E.g. print “” or “” explicit keywords, in order to:

  1. recognize them in the Traces view (instead of displaying weird squares characters :wink:)
  2. avoid this line break issue in the logged files
6

It would be really useful if you could find a font that can represent these codes in a single character space - as the VT100 et al used to:

vt100.net/dec/vt220/glyphs
vt100.net/docs/vt220-rm/table2-16a.html

Yes, that would be good - I currently tend to use both TRACE and DUMP where I suspect there may be a mix of prointable and non-printable characters…

7

With quoting enabled, I get this kind of thing:

This means that Excel still can’t use the time - because of those suffixes.

8

We’ll see to add an option to remove the counter suffix when persisting traces in a file.

9

Might also be better to have the date separate from the time?

10

Sounds difficult to make this working along with the capability to configure the timestamp format… If you don’t need the date, just remove it from the timestamp…

11

See: https://forum.sierrawireless.com/t/getting-ds-1-2-0-to-remember-my-layout/4842/7

I generally find that the date just wastes space in the screen display, but I do want it recorded in the log file.

Having separate date & time fields for both screen & file seems good to me

12

Ok, this is what we’ve decided to implement in 2.1.0:

  • Use two fields in preferences for timestamp configuration:
    • One for time (default “HH:mm:ss:SSS”; prints UNIX raw millisecond timestamp if empty)
    • One for date (default “yyyy/MM/dd”; doesn’t print anything if empty)
  • UI behavior: use “time” format field + " - XXX" suffix for traces counter (date is not displayed; seems clearly not useful in the UI)
  • Log behavior: persist date, time and counter on three different columns