DS 1.2.0 Trace CSV Format not compatible with Excel


#1

Ref: viewtopic.php?f=108&t=5123&start=0

There’s a few issues with the DS 1.2.0 Traces CSV format that make it incompatible with MS Excel (and, I suspect, other tools also):

1. Date format

Because all fields are 2 digits, Excel can’t tell which is the year (human readers may also get it wrong due to US/RoW confusion about mm/dd or dd/mm)

A 4-digit year needs to be used.

2. Time format

Excel doesn’t recognise this as a time, because of the colon between seconds & decimal seconds.
It needs to be a decimal point here.

3. Time width
The width of the decimal seconds varies between 1 & 3 digits - which breaks the tabulation, and makes the plain text file hard to read.; eg,

11/02/04,15:32:55:984 - 137;ADL;7;[L3] same PLMN
11/02/04,15:32:55:98 - 138;ADL;7;02 03 04 01 00 FF 
11/02/04,15:32:56:0 - 001;ADL;7;[L3] plmn_change_timer_cb
11/02/04,15:32:56:0 - 002;L3RR;3;==>C RRC_TIM_PERIODIC_SI
11/02/04,15:32:56:15 - 003;L3RR;3;<==C PH_BCCH_REQ on 27
11/02/04,15:32:56:150 - 004;L3RR;3;C: S4 P0 N1 M10

This also means that the above problems can’t be worked around by importing as fixed-width fields.

4. Quoting
The Trace strings themselves need to be “quoted” in some way - so that semicolons within the text are not takn as delimiters!
Or, again, the preceding fields need to be fixed width.


#2

Correct me if I’m wrong, but all of this can be configured in the DS preferences: Window > Preferences > Developer Studio > Target Management > Traces


#3

Oh yes - so it is! :blush:


#4

When the traces are displayed in the Studio, it manages to get each one on a single row.

But AT responses in the recorded file get broken across lines; eg,

11/02/04,15:47:58:93 - 007;ADL;7;l3_info_rsm_cb
11/02/04,15:47:58:109 - 008;ADL;7;Chan Lev BS MCC MNC  LAC CI
11/02/04,15:47:58:109 - 009;ADL;7;  75  15 60 234 15F 00BF 1588
11/02/04,15:47:58:109 - 010;ADL;7;RSM end.
11/02/04,15:47:58:125 - 011;ATI;1;ati_SndUnsolicitedAtRsp: Len=12
11/02/04,15:47:58:125 - 012;ATI;1;
+CREG: 2

11/02/04,15:47:58:125 - 013;ATI;1;ati_SndUnsolicitedAtRsp: Len=13
11/02/04,15:47:58:125 - 014;ATI;1;
+CGREG: 0

11/02/04,15:47:58:140 - 015;ADL;1;+CREG: 2 state received 
11/02/04,15:47:58:140 - 016;ADL;1;Force RSSI=99
11/02/04,15:47:58:140 - 017;ADL;1;+CREG: 2 Change registration state from 5

#5

We already have logged an evolution request to handle “special characters” in trace strings.
E.g. print “” or “” explicit keywords, in order to:

  1. recognize them in the Traces view (instead of displaying weird squares characters :wink:)
  2. avoid this line break issue in the logged files

#6

It would be really useful if you could find a font that can represent these codes in a single character space - as the VT100 et al used to:

vt100.net/dec/vt220/glyphs
vt100.net/docs/vt220-rm/table2-16a.html

Yes, that would be good - I currently tend to use both TRACE and DUMP where I suspect there may be a mix of prointable and non-printable characters…


#7

With quoting enabled, I get this kind of thing:

This means that Excel still can’t use the time - because of those suffixes.


#8

We’ll see to add an option to remove the counter suffix when persisting traces in a file.


#9

Might also be better to have the date separate from the time?


#10

Sounds difficult to make this working along with the capability to configure the timestamp format… If you don’t need the date, just remove it from the timestamp…


#11

See: https://forum.sierrawireless.com/t/getting-ds-1-2-0-to-remember-my-layout/4842/7

I generally find that the date just wastes space in the screen display, but I do want it recorded in the log file.

Having separate date & time fields for both screen & file seems good to me


#12

Ok, this is what we’ve decided to implement in 2.1.0:

  • Use two fields in preferences for timestamp configuration:
    • One for time (default “HH:mm:ss:SSS”; prints UNIX raw millisecond timestamp if empty)
    • One for date (default “yyyy/MM/dd”; doesn’t print anything if empty)
  • UI behavior: use “time” format field + " - XXX" suffix for traces counter (date is not displayed; seems clearly not useful in the UI)
  • Log behavior: persist date, time and counter on three different columns