FX30S unexpected file and folder loss 16.10.1.m3

IoT Gateways FX30
1

Hi

On FX30S with 16.10.1.m3_f5d280aff1eaa96ea8459ca6effa50c5

After receiving an AT command error and a power cycle. SSH’d into the FX30S - the fx30s demanded a password change (no password originally set )

Following successful login - all files and data in
/home/root
were erased.

Question?
Is there somewhere else that files and data can be stored without risk of the system erasing them? (and not a legato application)

2

Hi John,
This issue sounds very similar to the one posted here:

Could you please clarify:

  1. Are you the friend that is referenced in the other post? (I’d like to know the frequency of this occurrance)
  2. What AT command was executed?
  3. When you say power cycle, did you pull the power? If so, did you get an AT command response before the power was pulled?
  4. I can see you’re running stock Legato, but have you changed any other components on the system? What is the output of “fwupdate query” and “cm info”?
  5. I’m not aware of any situation that will erase files from the /home/root folder. To answer your question, this is the location that should be stored without risk. I will consult further internally how this could possibly happen.

BR,
Chris

3

Hi Chris,

No I am not friend of the person in other post.

  1. AT command which were executed were:
    AT, ATI,AT+CPIN?,AT+CGDCONT(for network configuration as I am using cellular network),AT!GSTATUS?
    2.Yes, by saying power cycle its removing the power cable. AT command were responding before the power was pulled. Only cm data was giving network status as “No”.
  2. I haven’t tried the “fwupdate query” but “cm info” is giving Sierra_FX30S information(IMEI,FSN etc.).
    4.But I and my friend both the same issue and lost all the data from /home/root and other directory location as well.

Will be a great help to resolve the issue.

Thanks & Regards,
Alka Rani

4

Thanks for responding - answers to question below

  1. Are you the friend that is referenced in the other post? (I’d like to know the frequency of this occurrence)

Not so far as I know

  1. What AT command was executed?

AT+CMGS - using PDU mode
It failed with an undocumented error - I didn’t at the time write it down - just assumed the FW doesn’t support PDU mode. The error could have been ERROR 250

  1. When you say power cycle, did you pull the power? If so, did you get an AT command response before the power was pulled?

Yes the AT command was completed probably 1/2 an hour before the power removal. I mentioned the AT command because it’s the only different activity that was performed and it behaved abnormally

  1. I can see you’re running stock Legato, but have you changed any other components on the system? What is the output of “fwupdate query” and “cm info”?

It’s stock - I’d like to modify it but I think the source on source is not correct - but that’s another question

Firmware Version: SWI9X15Y_07.12.09.00 r34123 CARMD-EV-FRMWR1 2017/04/26 23:34:19
Bootloader Version: SWI9X15Y_07.12.09.00 r34123 CARMD-EV-FRMWR1 2017/04/26 23:34:19
Linux Version: 3.14.29ltsi-a00e464379_499694411b #2 PREEMPT Tue Jun 12 16:49:03 PDT 2018
FX30 Version: R14.0.4.002

Device: FX30S
IMEI: 353984080034778
FSN: LL647501020410
Firmware: SWI9X15Y_07.12.09.00 r34123 CARMD-EV-FRMWR1 2017/04/26 23:34:19
Bootloader: SWI9X15Y_07.12.09.00 r34123 CARMD-EV-FRMWR1 2017/04/26 23:34:19
priIdPn: 9906770
priIdRev: 01.00
skuId: 1103232

  1. I’m not aware of any situation that will erase files from the /home/root folder. To answer your question, this is the location that should be stored without risk. I will consult further internally how this could possibly happen.

Help on this will be very useful as the project cannot be commercially deployed until we find a resolution or work around

John

5

Hi @rkirk,

Do you have any idea about the cause of this issue? Could it be EFS corruption?

The fix for LXSWIREF-248 was backported into the R14.0.4.002 release.

Thanks,
Chris

6

Hi

Anyone any ideas how this could happen and how to mitigate against it?

Cheers

John

7

Hi John,
I suspect the issue is caused by pulling the power. Is it possible to power down gracefully wherever possible?

BR,
Chris

8

Hi Chris

Random power supply input loss is inevitable in 99.999% of industrial use cases. As is power supply brownout. Have these use cases been tested for? what were the results? I can’t see any warnings in the FX30 user manual?

Random catastrophic loss of user data (say in /home/root ) and the loss of overlay Legato user apps are serious issues (for us) - the product will no longer function for the end customer - probably requiring a site visit or at least causing data loss.

I guess we could build app code / scripts into the Read only flash (Yocto --> cwe) but this doesn’t workaround the following

  • How to reliably replace apps in the field? - with the current system there is a chance that the app will revert to the previous (Read only) version
  • where to store database data ? /home/root has proved to be not safe - is there somewhere better?

A potential workaround is to supply a UPS with every FX30S - but this seems an expensive option where most likely the battery will fail at some point exposing the system to the issue again.

The failure occurred when no user apps were running. So I guess the failure was caused by some planned behaviour in the core FW/ OS / Legato?

Thanks

John

9

Hi John,
I completely understand. The patch from LXSWIREF-248 was specifically added to this release to address this issue so I’m very surprised we haven’t heard of any occurrences until now.

So it’s either the patch isn’t sufficient or there’s another unknown issue.

Could you please confirm the patch has indeed been applied? I can provide instructions later if you need.

The only other suggestion would be go seek support through your official channel.

BR,
Chris

10

Hi Chris

Let me know the process required to check the patch. At the time of the failure the FX30S was running the latest binary image downloaded from the source and installed [Edit delete this -> with the one click installer]

please ignore this comment it is incorrect
[We have had another report of a project using a WP7607-1 current Legato image - where user loaded legato apps vanish following repeated power cycling. This is a much more recent Legato with different hardware. Of course it could be a totally unrelated issue.]

Kind regards

John

11

Hi John,
I looked at the whole thread again and I can see that you were running the correct version of Legato in your original posting, but I’m confused at your latest statement:

At the time of the failure the FX30S was running the latest binary image downloaded from the source and installed with the one click installer

The FX30 and FX30S images are not available for one click installers. So now I’m wondering, is it possible there has been some mix up of running stock WP or stock Legato instead of the FX30 released versions when the issue occurred?

The fix for issue LXSWIREF-248 was purposefully added to the R14.0.4.002 FX30 release in Legato 16.10.1.m3, which is a maintenance branch based of Legato 16.10.3. This is why I’m surprised you’re seeing powerdown corruption with this fix in place.

If you’re building your own image, can you re-download the Legato 16.10.1.m3 source tarfile, extract it and do a diff on your workspace Legato folder?

Also, you can do: “cat legato/version” in your workspace to check the version (though it doesn’t guarantee source code has not been modified within).

For your last statement, the apps vanishing is expected behaviour as the Legato framework attempts to detect rogue applications causing the system to break and boot loop. So if your app, or you manually reboot multiple times (I think it’s 4 or more) within 30 mins, Legato will remove extra apps that were installed.

To get around this, you can mark the system as “good”:

BR,
Chris

12

Hi Chris

You are correct about the one click installer I didn’t upgrade the FX30S a colleague did using firmware from here

https://source.sierrawireless.com/resources/airlink/software_downloads/fx30-firmware/fx30-firmware-3g/

mcu-rmfw-boot-yocto-legato_wp85.cwe

I don’t know what method he actually used but I think it could have been one click if he constructed an installer by adding the above cwe to a one click installer.

I can’t now query the hardware for the version used when the failure occurred because I have since overwritten it with a custom Yocto image as part of an ongoing project. However the version data (above post) was obtained from the unit at the time that the FX30S deleted the folder

Marking FW stuff or Legato overlay stuff good

The host legato toolchain supplied with the current Yocto FX30S image (URL above) doesn’t have

fwupdate downloadOnly yocto-legato_wp85.cwe 192.168.2.2
fwupdate install 192.168.2.2
fwupdate markGood 192.168.2.2

How should the fw be marked good on the FX30S?

The text from here has been deleted and a new post has been created

Thanks

John

13

Hi John,
This makes good sense now. The unit running R13.1.3.001 does not have the powerdown fix. Could you please update your device to R14.0.4.002 and let me know if you experience any further corruption after powerdown?

Sorry I don’t know the answers to all your questions about Legato, it would be best to post those on the Legato forum.

You can however, ssh to your device or use serial console and use “fwupdate markGood” from the command line.

BR,
Chris

14

Hi Chris

Where in this thread is R13.1.3.001 mentioned? apart from here (my response) and your last response

The FX30S which failed reported (documented near the start of this thread) FX30S stock Version: R14.0.4.002 . As far as I am aware this thread is all about loss of user folders with stock FX30S R14.0.4.002

Please forget my comments about the WP76 - I checked a bit deeper into that and I’m convinced the issue was something totally different which is being addressed by the development team - I have amended my comments (above)

Am I misunderstanding something?

Cheers

John

15

Hi John,

Sorry for the confusion, a misunderstanding on my part. I’m not sure how I concluded you were using R13.

Could you please clarify with your team:

  1. How was the one-click created?
  2. When the corruption occurred, are you absolutely certain the FX30 was running R14.0.4.002? I’m wondering if there’s a chance it happened to be running a stock WP image that was installed using the WP one-click. The reason I suspect the device may not have been running the correct version is that other customers who constantly power down their units have not reported any issues since upgrading to R14.0.4.002.

BR,
Chris

16

Hi Chris
As per my first post this is the version reported by the modem immediately after the data / file loss

16.10.1.m3_f5d280aff1eaa96ea8459ca6effa50c5

I checked with the guy who upgrade the FX30S - this is the procedure used

root@fx30:~# fwupdate download mcu-rmfw-boot-yocto-legato_wp85.cwe
Connecting to service ...
Download started ...
Download successful; please wait for modem to rese

Cheers

John

17

Thanks John. Could you please open a ticket through your support channel?
BR,
Chris