i work on a project, which need a secure data transfer between a webserver and the Q2686. If I use server authentication and SSL encryption everything works fine.
Now I try to use client authentication with a valid client certificate and the corresponding key. The wip_SSLInitOpts and the wip_SSLCreate methods work fine and the Q2686 connects to the server. After I send some bytes to the server, the Q2686 reboots. It seems that the SSLLibrary needs to much memory for handle the server AND client certificates at the same time.
Here is a debug output only with server authentication:
[INFO ] SSLClientCreate
[DEBUG] ConnectionID:0
[DEBUG] XXX.XXX.XXX.XXX
[DEBUG] Port:443
[WIP] new TCPCLIENT 0x180c9694
[SSL][MEM] Memory profile: 10792 bytes reserved, 2224 useful; max = 12668/4016
36 segments between 2 and 3 bytes long
2 segments between 4 and 7 bytes long
60 segments between 8 and 15 bytes long
174 segments between 16 and 31 bytes long
12 segments between 32 and 63 bytes long
9 segments between 64 and 127 bytes long
8 segments between 128 and 255 bytes long
4 segments between 256 and 511 bytes long
1 segments between 512 and 1023 bytes long
[SSL] subevh_handshake function
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] Handshaking waiting for more network exchanges (state==0)
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] [BIO =>] read 7 bytes on TCP
[SSL] [BIO =>] read 826 bytes on TCP
[SSL] Handshaking waiting for more network exchanges (state==0)
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] [BIO =>] read 297 bytes on TCP
[SSL] Handshaking waiting for more network exchanges (state==0)
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 1 bytes on TCP
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 36 bytes on TCP
[SSL] Handshaking completed state(0)
[INFO ] TCP open
[INFO ] TCPClientSend
[SSL] writeOpts: 321 bytes written
[SSL][MEM] Memory profile: 66537 bytes reserved, 53237 useful; max = 96953/83457
50 segments between 2 and 3 bytes long
2 segments between 4 and 7 bytes long
89 segments between 8 and 15 bytes long
270 segments between 16 and 31 bytes long
16 segments between 32 and 63 bytes long
17 segments between 64 and 127 bytes long
19 segments between 128 and 255 bytes long
7 segments between 256 and 511 bytes long
1 segments between 512 and 1023 byte
2 segments between 4096 and 8191 bytes long
2 segments between 16384 and 32767 bytes long
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 24 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 1700 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 1169 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 633 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 97 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 21 bytes on TCP
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 36 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 1590 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 1059 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 523 bytes on TCP
[SSL] wip_read() can’t fill buffer, WIP_CEV_READ re-armed
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 525 bytes on TCP
[SSL] just received WIP_CEV_PEER_CLOSE on TCP socket
[SSL] Shutdown process needs more network data…
[INFO ] TCP Peer close
[WIP] closing CHANNEL 0x180c9614
[WIP] closing TCPCLIENT 0x180c9694
[WIPSSL] SSL Library closed
Here is a debug output with server authentication and client authentication:
[INFO ] SSLClientCreate
[DEBUG] ConnectionID:0
[DEBUG] XXX.XXX.XXX.XXX
[DEBUG] Port:443
[WIP] new TCPCLIENT 0x180cc714
[SSL][MEM] Memory profile: 15054 bytes reserved, 2678 useful; max = 16500/4130
50 segments between 2 and 3 bytes long
2 segments between 4 and 7 bytes long
86 segments between 8 and 15 bytes long
250 segments between 16 and 31 bytes long
17 segments between 32 and 63 bytes long
17 segments between 64 and 127 bytes long
14 segments between 128 and 255 bytes long
5 segments between 256 and 511 bytes long
1 segments between 512 and 1023 bytes long
[SSL] subevh_handshake function
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] Handshaking waiting for more network exchanges (state==0)
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] [BIO =>] read 7 bytes on TCP
[SSL] [BIO =>] read 826 bytes on TCP
[SSL] Handshaking waiting for more network exchanges (state==0)
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] [BIO =>] read 297 bytes on TCP
[SSL] Handshaking waiting for more network exchanges (state==0)
[SSL] subevh_handshake function
[SSL] subevh_handshake: WIP_CEV_OPEN
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 1 bytes on TCP
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 36 bytes on TCP
[SSL] Handshaking completed state(0)
[INFO ] TCP open
[INFO ] TCPClientSend
[SSL] writeOpts: 321 bytes written
[SSL][MEM] Memory profile: 70799 bytes reserved, 53691 useful; max = 101215/83911
64 segments between 2 and 3 bytes long
2 segments between 4 and 7 bytes long
115 segments between 8 and 15 bytes long
346 segments between 16 and 31 bytes long
21 segments between 32 and 63 bytes long
25 segments between 64 and 127 bytes long
25 segments between 128 and 255 bytes long
8 segments between 256 and 511 bytes long
1 segments between 512 and 1023 bytes long
2 segments between 4096 and 8191 bytes long
2 segments between 16384 and 32767 bytes long
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 24 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 5 bytes on TCP
[SSL] [BIO =>] read 1700 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 1169 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 633 bytes on TCP
[SSL] pause the reading on transport layer, waiting for more data
[SSL] [BIO =>] read 97 bytes on TCP
…after this line the modem resets
It seems to be a bug in the SSL-Library. Anyone an Idea?
thanks for your reply!
Yes, I tried a couple of stack sizes (up to 64kByte) but the reboot behavior stays the same.
As soon as I comment out the client certificate, everything works fine.
Of course the server answers with error 403, because of the missing client certificate.
Is there another possibility to change a configuration property exept the stack size?
Is it possible to discover the reason for an exeption that triggers a reboot?
In my case, the stack was not a problem (8KB). When I had less than 100KB of RAM (i.e. heap) free, I also got exceptions/reboots when handshaking SSL. I ended making sure I always have 150KB heap free for SSL and since then haven’t encountered this particular problem.
this sounds interesting! How you can make sure, that SSL have always 150kByte Heap free for his memory wasting? Also for me, it increasingly likely that the problems comes from a wrong memory management inside the SSL-Library.
BTW using the adl_errSubscribe(…) was also not helpful because unfortunately there are no errors thrown by the SSL-Lib before the restart happens.
Sure, as you could expect there were some bugs in SSL library, mainly causing memory leaks when a connection was broken. After a few broken connections I was running out of memory when handshaking a new connection. Fortunately, in firmware 7.44 (i.e. OpenAT 2.34) I don’t see this error anymore. That is, I think SSL tends to allocate a lot of memory while connecting, but it seems to release it afterwards. I haven’t verified it though.
As to the first question. I don’t use too much heap memory in my application, so it’s not a problem of always leaving at least 150KB of free heap. See the results of adl_memGetInfo() to see how much heap memory you have at the start (!) of your application. Then, you have to estimate how much you use (i.e. the biggest buffers you allocate, etc.), but unfortunately there’s no API to check how much memory you, or the libraries, have allocated while the program is running. I’ve written some wrappers for my malloc/free to count the allocated bytes of heap memory. This is also useful to hunt for memory leaks. Not the greatest tool, but better than nothing.
I encountered this SSL problem because I had large static buffers in the program, so that the region for heap memory was quite small (AFAIR 100KB during application start). The solution was to decrease their size so that I get around 190-200KB of heap at the start.
All you can do, is just do some experiments, write down the numbers and finnaly add some safe marigin.
a wrapper for malloc/free to count the allocated bytes of heap… I think this is a good idea!
It would be nice, if the SSL plugin developers would do the same to.
I will also write this kind of wrapper to have a workaround for using client certificates. At this moment, it is not clear if we will use it in the final project. For the first step server authentication is enough - and this works fine. (probably as long as enough heap is available…)
That reminded me that I haven’t tested this issue when using client certificate, as we’re not currently using them, but have this as an option. Probably using both client and server authentication requires more heap.
I was wondering how did you generate the CA cert, server/client certs and keys. I an using this method which I posted to serverfault.com/questions/244408 … rtificates but I am getting a lot of problems with it when testing it with an openssl server and client.
I also wondering if you have any other issues with the SSL comms. I have got one of the suppliers to test the same code on a Q2687 over in OZ to connect to my OpenSSL server and that works fine but when I test the same setup in NZ, I always get a certificate problem. I have also tested the same certs and keys (provided by sierra wireless) on an openssl server and client and that seems to be working correctly (no faults). I am not sure if it is a problem with the APN but Vodafone so far has not been helpful. I also tried the Extreme part but the supplier always gets certificate problem over in OZ when connecting to my OpenSSL server. When I try it, I can’t connect to the XT network as all using the same hardware sent by the supplier.
Okay I figured it out the reason for the fault. Just need to generate the certs. This forum is pretty useless when no-one from Sierra Wireless respond to any technical question.
there were some bugs in SSL library, mainly causing memory leaks when a connection was broken. After a few broken connections I was running out of memory when handshaking a new connection. Fortunately, in firmware 7.44 (i.e. OpenAT 2.34) I don’t see this error anymore. That is, I think SSL tends to allocate a lot of memory while connecting, but it seems to release it afterwards. I haven’t verified it though.