Dualstack errors with MC7354

IoT Modules MC/EM Series
1

I am attempting to setup a dualstack interface on Verizon with the MC7354 using libqmi and the Linux kernel 3.13 qmi_wwan driver. Independently, IPv4 and IPv6 work successfully using DHCP for v4 and SLAAC for v6. I can bring up both WDS clients and successfully start both network calls. I enable the net-802-3 and net-no-qos-header options to qmicli. The modem is using the latest VzW software: SWI9X15C_05.05.16.03 r22385

The problem arises when doing a simple ping on IPv6 after DHCP has been started. If I simply start both networks, perform the IPv6 autoconfiguration, and ping 2607:f8b0:4004:806::1005 (google.com), the ping succeeds. After starting DHCP, the ping response has a garbled 802.3 header. The MAC address and ethernet type fields are invalid as shown in the following tcpdump output. The data in the packet is correct but the header prevents interpretation by the software. IPv4 continues to be usable after IPv6 gets into this state.

09:09:07.027155 3e:3a:cf:a5:18:52 2:50:f3:0:8:0 86dd 118: 2600:1009:b023:940:3c3a:cfff:fea5:1852 > 2607:f8b0:4004:806::1005: icmp6: echo request
0x0000 6000 0000 0040 3aff 2600 1009 b023 0940 ....@:.&....#.@ 0x0010 3c3a cfff fea5 1852 2607 f8b0 4004 0806 <:.....R&...@... 0x0020 0000 0000 0000 1005 8000 4db5 07bb 0016 ..........M..... 0x0030 f05c b03b 0000 0000 0000 0000 0000 0000 .\.;............ 0x0040 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0050 0000 .. 09:09:07.133879 eb:b8:7:63:95:27 fb:d7:2b:6a:0:45 f60c 118: 0x0000 6000 0000 0040 3a32 2607 f8b0 4004 0806…@:2&…@…
0x0010 0000 0000 0000 1005 2600 1009 b023 0940 …&…#.@
0x0020 3c3a cfff fea5 1852 8100 4cb5 07bb 0016 <:…R…L…
0x0030 f05c b03b 0000 0000 0000 0000 0000 0000 …;…
0x0040 0000 0000 0000 0000 0000 0000 0000 0000 …
0x0050 0000 …

Any help?

2

Good to know. This really eliminates all other options than a firmware bug. At least I don’t see any other alternatives.

And that’s rules out a simple firmware upgrade…

OK, even if this is fixed in the future there is obviously devices out there with the bug. Which means that we should try to work around it in the driver, if possible.

Thanks a lot for the packet dumps. That makes it easier to understand the problem, and to design some solution for it.

Regarding solutions: I believe it is possible to fix up these packets in qmi_wwan_rx_fixup(), by simply ignoring the bogus header and create a new one the same way we fixup packets without a header. But after trying to create a simple two-line fix, I’ve realized that it will require some refactoring of the existing fixup code. We need to verify that the ethertype is one of the 3 expected types, which is simple. But we need to keep the existing behaviour for raw IP packets, which means that we cannot just test the ethertype. We’ll probably have to apply that test only to the packets failing the simple IP header test (assuming IP if the first nibble is ‘4’ or ‘6’).

It might take a while before I have time to propose a patch doing this. But I don’t think it’s very difficult if you want to give it a shot yourself. It’s just a bit of work thinking it all over, testing and verifying the results.

3

I happened to read the release notes for the SIerra supported Linux drivers (
developer.sierrawireless.com/Res … 0Note.aspx )
and the following recent change (in version SierraLinuxQMIdriversS2.20N2.26) caught my interest:

Looking at the driver I see code similar to what I tried to describe above in the GobiNetDriverLteRxFixup() function (which they have borrowed from qmi_wwan :slight_smile: ):

/* special handling for corrupted Ethernet header packet if any */
    if ((skb->data[ETH_HLEN] & 0xF0) == 0x40)
    {
        /* check if need to correct the IPV4 Ethernet header or not */
        if (FixEthFrame(dev, skb, 1))
        {
           /* pass through */
           return 1;
        }
    }
    else if ((skb->data[ETH_HLEN] & 0xF0) == 0x60)
    {
        /* check if need to correct the IPV6 Ethernet header or not */
        if (FixEthFrame(dev, skb, 0))
        {
           /* pass through */
           return 1;
        }
    }

The actual fixup code looks like this. A similar approach should work fine for qmi_wwan too, but will need some style fixes before it is acceptable in the mainline Linux kernel:

/* check the packet if the Etherenet header is corrupted or not, if yes,
 * correct the Ethernet header with replacing destination address and ip protocol.
 * if no, do nothing
 */
int FixEthFrame(struct usbnet *dev, struct sk_buff *skb, int isIpv4)
{
    __be16 proto;
    u16 total_len, payload_len;

    /* All MAC-48 multicast identifiers prefixed "33-33", do not overwrite the MAC address if it is not corrupted */    
    if ((skb->data[0] == MAC48_MULTICAST_ID) && (skb->data[1] == MAC48_MULTICAST_ID))
    {
        proto = ((skb->data[ETH_HLEN-2] << 8) & 0xff) |(skb->data[ETH_HLEN-1]);
        /* check the IP type field, if it is correct, we can consider this is not a corrupted packet */
        if (proto == skb->protocol)
        {
            DBG( "multicast MAC address: destination matched, pass through ");
            /* correct packet, pass through */
            return 1;
        }
        else
        {
            DBG( "multicast MAC address: destination mismatched, IPV%s header modified:", isIpv4 == 1 ? "4":"6");
            ResetEthHeader(dev, skb, isIpv4);
            return 1;
        }
    }
    else if (memcmp(&skb->data[0], &dev->net->dev_addr[0], ETH_ALEN) == 0)
    {
        /* MAC address is correct, no need to overwrite, pass through */
        DBG( "correct packet, pass through ");
        return 1;
    }
    else
    {
        if (isIpv4)
        {
            /* ipv4 */
            total_len = ((skb->data[ETH_HLEN+IPV4HDR_TOT_UPPER] << 8) & 0xff) | (skb->data[ETH_HLEN+IPV4HDR_TOT_LOWER]);
            DBG( "ipv4 header: total length = %d\n", total_len);
            /* total length includes IP header and payload, hence it plus Ethernet header length should be equal to
               the skb buffer length if the Etherent header is presented in the skb buffer*/
            if (skb->len >= (total_len+ETH_HLEN))
            {
                DBG( "IPv4 header modified: ");
                ResetEthHeader(dev, skb, isIpv4);
                return 1;
            }
        }
        else
        {
            /* ipv6 */
            payload_len = ((skb->data[ETH_HLEN+IPV6HDR_PAYLOAD_UPPER] << 8) & 0xff) | (skb->data[ETH_HLEN+IPV6HDR_PAYLOAD_LOWER]);
            DBG( "ipv6 header: payload length = %d\n", payload_len);
            /* for IPV6, the playload length does not include ipv6 header */
            if (skb->len >= (payload_len+ETH_HLEN+IPV6HDR_LENGTH))
            {
                DBG( "IPv6 header modified: ");
                ResetEthHeader(dev, skb, isIpv4);
                return 1;
            }
        }
    }
    return 0;
}

Note that this also seems to verify and fixup IPv4 frames as well. I don’t know if that is just for completeness or whether they’ve actually encountered the same firmware bug affecting IPv4. I do still find it sort of both worrying and amusing that they choose to add workarounds to drivers instead of fixing the firmware bugs, but…

In any case, it would have been nice of those working on this driver started communicating with the open source community. We have obviously a lot to learn from eachother. Consider that my xmas wishlist.

4

I have ported the changes from the latest Sierra drivers to qmi_wwan, and they appear to work from a simple ping test. I’ve attached a patch to the driver.

Jeff
qmi_wwan.patch.txt (4.95 KB)