Bug 467004

This is actually a PPP problem, not a NetworkManager problem.  Fairly well documented, requires fixes to PPP.  See:

http://osdir.com/ml/linux.ppp/2007-11/msg00006.html
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445711

Thanks a lot for the pointers.  The thing is that in my case it _always_ gets incorrect DNS servers.  Still, it looks like the same bug.

What do you think about those fixes? are they going to be upstream?

This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

I can confirm this bug on updated Fedora 10 with Huawei E220. Modem works out of box but resolv.conf contains 10.11.12.13 and 10.11.12.14. Changing those values manually fix the problem however its not an option with NM VPNC plugin as they are rewritten there once again.

I believe this bug is fixed in ppp-2.4.5 upstream.

How can I test this? I would love to try it out and confirm the fix and not trash my F10 installation or be able to revert. I'm not very knowledgeable about fedora packaging but can donate a bit of my time to play with this issue if you give me some tips.

I found a needed patch here: http://marc.info/?l=linux-ppp&m=119559914711075&w=2
and corrected to make it work with "ppp-2.4.4-8.fc10.src.rpm". This fixes the issue. If anyone is interested below is the patch. I simply added it as the last one in the ppp.spec file and used hints from here to rebuild the package:

http://hacktux.com/fedora/source/rpm

Patch:

--- ppp-2.4.4/pppd/ipcp.c	2005-08-26 01:59:34.000000000 +0200
+++ ppp-2.4.4/pppd/ipcp.c.bogus-dns	2008-12-09 09:32:26.000000000 +0100
@@ -715,7 +715,8 @@ ipcp_cilen(f)
 #define LENCIADDRS(neg)		(neg ? CILEN_ADDRS : 0)
 #define LENCIVJ(neg, old)	(neg ? (old? CILEN_COMPRESS : CILEN_VJ) : 0)
 #define LENCIADDR(neg)		(neg ? CILEN_ADDR : 0)
-#define LENCIDNS(neg)		(neg ? (CILEN_ADDR) : 0)
+#define LENCIDNS(neg) LENCIADDR(neg)
+#define LENCIWINS(neg) LENCIADDR(neg)
 
     /*
      * First see if we want to change our options to the old
@@ -737,7 +738,9 @@ ipcp_cilen(f)
 	    LENCIVJ(go->neg_vj, go->old_vj) +
 	    LENCIADDR(go->neg_addr) +
 	    LENCIDNS(go->req_dns1) +
-	    LENCIDNS(go->req_dns2)) ;
+	    LENCIDNS(go->req_dns2) +
+            LENCIWINS(go->winsaddr[0]) +
+            LENCIWINS(go->winsaddr[1])) ;
 }
 
 
@@ -810,6 +813,19 @@ ipcp_addci(f, ucp, lenp)
 	} else \
 	    neg = 0; \
     }
+ 
+#define ADDCIWINS(opt, addr) \
+    if (addr) { \
+       if (len >= CILEN_ADDR) { \
+           u_int32_t l; \
+           PUTCHAR(opt, ucp); \
+           PUTCHAR(CILEN_ADDR, ucp); \
+           l = ntohl(addr); \
+           PUTLONG(l, ucp); \
+           len -= CILEN_ADDR; \
+        } else \
+           addr = 0; \
+    }
 
     ADDCIADDRS(CI_ADDRS, !go->neg_addr && go->old_addrs, go->ouraddr,
 	       go->hisaddr);
@@ -823,6 +839,10 @@ ipcp_addci(f, ucp, lenp)
 
     ADDCIDNS(CI_MS_DNS2, go->req_dns2, go->dnsaddr[1]);
 
+    ADDCIWINS(CI_MS_WINS1, go->winsaddr[0]);
+
+    ADDCIWINS(CI_MS_WINS2, go->winsaddr[1]);
+
     *lenp -= len;
 }
 
@@ -1159,6 +1179,15 @@ ipcp_nakci(f, p, len, treat_as_reject)
 		try.neg_addr = 1;
 	    no.neg_addr = 1;
 	    break;
+        case CI_MS_WINS1:
+        case CI_MS_WINS2:
+            if (cilen != CILEN_ADDR)
+               goto bad;
+            GETLONG(l, p);
+            ciaddr1 = htonl(l);
+            if (ciaddr1)
+               try.winsaddr[citype == CI_MS_WINS2] = ciaddr1;
+            break;
 	}
 	p = next;
     }
@@ -1275,6 +1304,21 @@ ipcp_rejci(f, p, len)
 	try.neg = 0; \
     }
 
+#define REJCIWINS(opt, addr) \
+    if (addr && \
+        ((cilen = p[1]) == CILEN_ADDR) && \
+        len >= cilen && \
+        p[0] == opt) { \
+        u_int32_t l; \
+        len -= cilen; \
+        INCPTR(2, p); \
+        GETLONG(l, p); \
+        cilong = htonl(l); \
+        /* Check rejected value. */ \
+        if (cilong != addr) \
+            goto bad; \
+        try.winsaddr[opt == CI_MS_WINS2] = 0; \
+    }
 
     REJCIADDRS(CI_ADDRS, !go->neg_addr && go->old_addrs,
 	       go->ouraddr, go->hisaddr);
@@ -1288,6 +1332,10 @@ ipcp_rejci(f, p, len)
 
     REJCIDNS(CI_MS_DNS2, req_dns2, go->dnsaddr[1]);
 
+    REJCIWINS(CI_MS_WINS1, go->winsaddr[0]);
+
+    REJCIWINS(CI_MS_WINS2, go->winsaddr[1]);
+
     /*
      * If there are any remaining CIs, then this packet is bad.
      */

The patch is applied in the rawhide. When there is no more problem I'll backport it to F-10.

*** Bug 449884 has been marked as a duplicate of this bug. ***

If anyone still sees this bug, please try this patch:

http://www.archivum.info/linux-kernel@vger.kernel.org/2008-07/msg00238.html

Although this problem is not strictly a PPP one, a patch to pppd might
help. It was mentioned in comp.os.linux.networking discussion[1] and did
the trick for a Huawei E220 USB and an Option PC card on Debian etch
which misses a few other ppp patches related to GPRS and DNS.

http://groups.google.com/group/comp.os.linux.networking/browse_thread/thread/40dc4ddde9f0b40d#

-Mikko

*** Bug 492067 has been marked as a duplicate of this bug. ***

re-opening; seems the problem still isn't completely fixed with latest ppp.  It's certainly a *ton* better for me with T-Mobile USA after this ppp update, but apparently not completely squashed.

For NM users: you can enter manual DNS servers for your 3G connection in the connection editor, there's an "IPv4" tab into which you want to put cf "4.2.2.1 4.2.2.2" etc.

This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Hi,
I've found an info joined to this bug. Please try use:

"connect-delay 5000" in /etc/ppp/options to fix.

The second thing I've built scratch build of ppp with an extension in the bogus_dns patch mentioned above. That is reachable on:

http://koji.fedoraproject.org/koji/taskinfo?taskID=1721764

Let me know results of both solution. If the first one will work It would be
fine to avoid adding additional patch.

Thanks, Jiri

*** Bug 496010 has been marked as a duplicate of this bug. ***

FWIW, I don't have access to the piece of hardware which was problematic in my tests so unfortunately I cannot check whether those suggestions would help. 

Thanks.

What stage does connect-delay work in?  Does it just pause pppd for 5 seconds or something before trying to start the negotiation?

I think the problem still exists. I recently got a Huawei E180v from Swisscom, and have noticed that the internet was unbearably slow under linux, but fine under windows. I have thus compared the dns values, and it turns out 4.2.2.1 and 4.2.2.2 are getting picked up. I'll attach the connection log.

Created attachment 378355 [details]
relevant part of /var/log/messages

This is the relevant part of /var/log/messages.

Created attachment 378359 [details]
connection data under fedora

Created attachment 378360 [details]
connection data under win xp

And here are the connection parameters under Windows XP SP3, with Swisscom's Unlimited Data Manager installed.
As you can see, not only DNS addresses are different, but also the subnet mask and the default gateway. Not sure how the others influence the connection, though, since after I gave the right DNS addresses by hand everything seems to work fine.
This is all on Fedora 12 x86_64, with the following packages installed:
NetworkManager-1:0.7.996-7.git20091113.fc12.x86_64
ppp-2.4.4-13.fc12.x86_64

(In reply to comment #18)
> I think the problem still exists. I recently got a Huawei E180v from Swisscom,
> and have noticed that the internet was unbearably slow under linux, but fine
> under windows. I have thus compared the dns values, and it turns out 4.2.2.1
> and 4.2.2.2 are getting picked up. I'll attach the connection log.  

These values (which are known-good nameservers but obviously aren't correct for everyone) are substituted by NetworkManager when ppp returns bogus nameservers, or no nameservers.  It indicates a problem with ppp or between ppp and your dongle's firmware.  Using 4.2.2.1 and 4.2.2.2 at least allows you to get on the internet.

If you want to use custom DNS servers, you can do so from the IPv4 tab of your 3G connection in nm-connection-editor.

The question is, then, why is ppp returning bogus nameservers - the 10.11.12.13 values appear in the log:
Dec 14 22:01:44 snowball pppd[3270]: Could not determine remote IP address: defaulting to 10.64.64.64
Dec 14 22:01:44 snowball pppd[3270]: local  IP address 10.121.99.166
Dec 14 22:01:44 snowball pppd[3270]: remote IP address 10.64.64.64
Dec 14 22:01:44 snowball pppd[3270]: primary   DNS address 10.11.12.13
Dec 14 22:01:44 snowball pppd[3270]: secondary DNS address 10.11.12.14
Moreover, I checked the TCP/IP properties under Windows and the addresses are supposed to be picked up automatically - although it might be the case that it's the UDM which provides them in some non-standard manner.

(In reply to comment #23)
> The question is, then, why is ppp returning bogus nameservers - the 10.11.12.13
> values appear in the log:
> Dec 14 22:01:44 snowball pppd[3270]: Could not determine remote IP address:
> defaulting to 10.64.64.64
> Dec 14 22:01:44 snowball pppd[3270]: local  IP address 10.121.99.166
> Dec 14 22:01:44 snowball pppd[3270]: remote IP address 10.64.64.64
> Dec 14 22:01:44 snowball pppd[3270]: primary   DNS address 10.11.12.13
> Dec 14 22:01:44 snowball pppd[3270]: secondary DNS address 10.11.12.14
> Moreover, I checked the TCP/IP properties under Windows and the addresses are
> supposed to be picked up automatically - although it might be the case that
> it's the UDM which provides them in some non-standard manner.  

Exactly, and that's what this bug was originally filed for.  Nobody is quite sure why ppp isn't able to negotiate correct DNS servers with the modem.  Remember that the PPP session does *not* happen between the computer and the network; it happens between the computer and the 3G device's firmware because PPP itself is not actually used in 3G networks.  This is why the card's firmware can have an effect on the issue.  Some devices work better than others; Huawei and ZTE have this bogus DNS problem more frequently than Sierra or Option hardware for example.

I formerly used an Huawei E220 und ZTE MF626 together with wvdial (as of current debian testing). I regularly got this annoying '10.11.12.14' bug described here.
A few weeks ago I deciced to call pppd directly to get more useful debug data in case the problem shows up again.

pppd debug ttyUSB2 460800 nodetach defaultroute noipdefault noauth lock usepeerdns connect '\''chat -V \"\" at \"\" at OK atz OK AT+CPMS=? OK at+cgdcont=1,\\\"IP\\\",\\\"internet.eplus.de\\\" OK atdt*99# CONNECT'\''

It's funny: since I do that I haven't got this error until now with both E220 and ZTE626

forgot to say:

at the time I still used 'wvdial' I tried the workaround from above:

> "connect-delay 5000" in /etc/ppp/options to fix

this definitively didn't 'fix' the issue for me

The bogus DNS addresses come from the modem, most likely a Qualcomm one, when it has not yet received proper addresses from the GSM/3G network and a PPP client is asking for one.

Workaround is to add a long delay before PPP asks for DNS addresses, but that delay can be really long and depends on network coverage. Another workaround is to use static DNS server addresses.

IMO, proper fix is to find out how to query network registration and configuration
status from the modem with AT commands. Unfortunately I'm not aware of any Qualcomm extensions for this.

Other fix is to not accept these phony addresses during PPP address negotiations. Eventually the device will give the proper addresses.

An easy way to repeat the problem is to use udev rules to fire up the PPP link
when a USB modem is connected. Since the modem draws power from USB and takes a while to boot the modem and after that authenticate and setup the GSM/GPRS/EDGE/3G network connections, the bogus addresses appeared almost every time.

-Mikko

*** Bug 547999 has been marked as a duplicate of this bug. ***

if a solution becomes available it probably will apply to here also:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445711

(In reply to comment #27)
> The bogus DNS addresses come from the modem, most likely a Qualcomm one, when
> it has not yet received proper addresses from the GSM/3G network and a PPP
> client is asking for one.
> 
> Workaround is to add a long delay before PPP asks for DNS addresses, but that
> delay can be really long and depends on network coverage. Another workaround is
> to use static DNS server addresses.
> 
> IMO, proper fix is to find out how to query network registration and
> configuration
> status from the modem with AT commands. Unfortunately I'm not aware of any
> Qualcomm extensions for this.

modem-manager will use standard GSM queries to determine registration status and won't attempt connection until the modem has registered; NM/MM have pretty much always done this and this has still been a problem.  I don't think registration status has much to do with the problem at all.

When setting up the GPRS context between the firmware and the network, the "create PDP context response" from the network includes both the terminal's IP address /and/ DNS servers (the two GSN fields of the GTP-U "Create PDP context response" message I believe).  There aren't any other relevant messages that return DNS information to the firmware; though it could be that the network tries to update the PDP context and sends along the valid DNS servers later, but that would be pretty stupid and the modem should handle that.

I don't think this is really an issue between the modem and the network, I think it's more of an issue internally in the firmware in the firmware's implementation of PPP.  Since on Windows most devices don't actually use PPP (they use Qualcomm proprietary mechanisms for command signaling and data transfer) I don't think Windows will run into this as often.

> Other fix is to not accept these phony addresses during PPP address
> negotiations. Eventually the device will give the proper addresses.

I think that's the right place to handle it, and I thought that PPP had already fixed that but I guess not.

Looking at the ppp git tree, it seems only patch missing from pppd is this:

http://www.archivum.info/linux-kernel@vger.kernel.org/2008-07/00238/Re:_PPP_difficulties_regarding_connection_establishment_and_bogus_DNS_received

which uses the DNS server addresses from the peers last response. Current default is to use the addresses from the first response, which are bogus in this case.

I tested the patch back in January 2009 when I had easy access to affected hardware. Logs and other stuff, including AT command logs from Vodafone Connection Manager from Vista are mentioned in:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=445711#51

as mentioned above I start pppd directly with debug opt to get more information in case the error strikes again. After about 30 successful pppd usages I now got my first '10.11.12.13' - and what a surprise not with HUAWAI or ZTE - it got it with my new Option GI0451.

After a second call to pppd the connection was setup again but with correct name servers.

Maybe the debug output gives some useful hints about the root cause of the problem?

device: Option GI0451
debug output:

at
OK
at
OK
atz
OK
AT+CPMS=?
+CPMS: ("SM","MT","ME","SR"),("SM","MT","ME","SR"),("SM","MT","ME","SR")

OK
at+cgdcont=1,"IP","pinternet.interkom.de"
OK
atdt*99#
CONNECTSerial connection established.
using channel 1
Using interface ppp0
Connect: ppp0 <--> /dev/ttyHS3
sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x1433c5eb> <pcomp> <accomp>]
rcvd [LCP ConfReq id=0x8 <asyncmap 0x0> <auth chap MD5> <magic 0x2f2a702b> <pcomp> <accomp>]
sent [LCP ConfNak id=0x8 <auth pap>]
rcvd [LCP ConfAck id=0x1 <asyncmap 0x0> <magic 0x1433c5eb> <pcomp> <accomp>]
rcvd [LCP ConfReq id=0x9 <asyncmap 0x0> <auth pap> <magic 0x2f2a702b> <pcomp> <accomp>]
sent [LCP ConfAck id=0x9 <asyncmap 0x0> <auth pap> <magic 0x2f2a702b> <pcomp> <accomp>]
sent [LCP EchoReq id=0x0 magic=0x1433c5eb]
sent [PAP AuthReq id=0x1 user="bajada" password=<hidden>]
rcvd [LCP DiscReq id=0xa magic=0x2f2a702b]
rcvd [LCP EchoRep id=0x0 magic=0x2f2a702b 14 33 c5 eb]
rcvd [PAP AuthAck id=0x1 ""]
PAP authentication succeeded
sent [CCP ConfReq id=0x1 <deflate 15> <deflate(old#) 15> <bsd v1 15>]
sent [IPCP ConfReq id=0x1 <compress VJ 0f 01> <addr 0.0.0.0> <ms-dns1 0.0.0.0> <ms-dns3 0.0.0.0>]
rcvd [LCP ProtRej id=0xb 80 fd 01 01 00 0f 1a 04 78 00 18 04 78 00 15 03 2f]
Protocol-Reject for 'Compression Control Protocol' (0x80fd) received
rcvd [IPCP ConfNak id=0x1 <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14> <ms-wins 10.11.12.13> <ms-wins 10.11.12.14>]
sent [IPCP ConfReq id=0x2 <compress VJ 0f 01> <addr 0.0.0.0> <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14>]
rcvd [IPCP ConfNak id=0x2 <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14> <ms-wins 10.11.12.13> <ms-wins 10.11.12.14>]
sent [IPCP ConfReq id=0x3 <compress VJ 0f 01> <addr 0.0.0.0> <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14>]
rcvd [IPCP ConfNak id=0x3 <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14> <ms-wins 10.11.12.13> <ms-wins 10.11.12.14>]
sent [IPCP ConfReq id=0x4 <compress VJ 0f 01> <addr 0.0.0.0> <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14>]
rcvd [IPCP ConfNak id=0x4 <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14> <ms-wins 10.11.12.13> <ms-wins 10.11.12.14>]
sent [IPCP ConfReq id=0x5 <compress VJ 0f 01> <addr 0.0.0.0> <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14>]
rcvd [IPCP ConfNak id=0x5 <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14> <ms-wins 10.11.12.13> <ms-wins 10.11.12.14>]
sent [IPCP ConfReq id=0x6 <compress VJ 0f 01> <addr 0.0.0.0>]
rcvd [IPCP ConfNak id=0x6 <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14> <ms-wins 10.11.12.13> <ms-wins 10.11.12.14>]
sent [IPCP ConfReq id=0x7 <compress VJ 0f 01> <addr 0.0.0.0>]
rcvd [IPCP ConfNak id=0x7 <ms-dns1 10.11.12.13> <ms-dns3 10.11.12.14> <ms-wins 10.11.12.13> <ms-wins 10.11.12.14>]
sent [IPCP ConfReq id=0x8 <compress VJ 0f 01> <addr 0.0.0.0>]
rcvd [IPCP ConfReq id=0x4]
sent [IPCP ConfNak id=0x4 <addr 0.0.0.0>]
rcvd [IPCP ConfRej id=0x8 <compress VJ 0f 01>]
sent [IPCP ConfReq id=0x9 <addr 0.0.0.0>]
rcvd [IPCP ConfReq id=0x5]
sent [IPCP ConfAck id=0x5]
rcvd [IPCP ConfNak id=0x9 <addr 10.35.175.137>]
sent [IPCP ConfReq id=0xa <addr 10.35.175.137>]
rcvd [IPCP ConfAck id=0xa <addr 10.35.175.137>]
Could not determine remote IP address: defaulting to 10.64.64.64
Cannot determine ethernet address for proxy ARP
local  IP address 10.35.175.137
remote IP address 10.64.64.64
primary   DNS address 10.11.12.13
secondary DNS address 10.11.12.14
Script /etc/ppp/ip-up started (pid 2703)
Script /etc/ppp/ip-up finished (pid 2703), status = 0x0

after all my conclusion is we don't talk about a problem of a specific device or firmware. I got hits with:

HAWAEI E220
ZTE MF626
Option GI0451

Nor it is an issue of a specific provider. I got hits with:

O2 Germany
EPLUS Germany

(In reply to comment #33)
> after all my conclusion is we don't talk about a problem of a specific device
> or firmware. I got hits with:
> HAWAEI E220
> ZTE MF626
> Option GI0451
> Nor it is an issue of a specific provider. I got hits with:
> O2 Germany
> EPLUS Germany  

My conclusion is that this problem affects Qualcomm GSM/GPRS/EDGE/3G chipsets which are used by many mobile data cards. I've seen the phony DNS servers on Huawei E220 USB modem and Vodafone Mobile Connect 3G/GPRS PCMCIA datacard made by Option. Both have Qualcomm 3G/GPRS chips, Option even says "Qualcomm 3G/CDMA".

I suggest trying the patch linked in my previous post.

(In reply to comment #34)
> I suggest trying the patch linked in my previous post.  

great, thank you for this. I also skimmed the posts around this patch.
It appears to be a simple but effective solution. Just my taste!
Your patch matches exactly the footprint of all my debug outputs.
I've got a few of them in the meantime.

So from now I will use a patched pppd and report the results...

I can confirm that the patch from comment #31 solves the problem (it needs whitespace fixes, though). On the other hand, the connection seems to take longer, but this might be caused by the fact that I'm not in .ch now and tests are done using roaming.

(In reply to comment #35)
> So from now I will use a patched pppd and report the results...  

I just want to confirm, that the patch from https://bugzilla.redhat.com/show_bug.cgi?id=467004#c31 solves the problem for me too. No failures after some more 40 pppd logins. Thanks!

It seems like the longer connection times were indeed caused by roaming - I'm now back in .ch and the connection gets established without a significant delay.

I was amiss I thought the patch from comment #31 is in F11. I'm sorry - next release will contain it.

Would you mind building a fixed package for F-12 as well? It is also affected.

(In reply to comment #40)
> Would you mind building a fixed package for F-12 as well? It is also affected.  

I'll probably modify the patch to remove ifdefs. Then I'll put it to F12 and rawhide. Although I suppose the main action is updating selinux-policy.

I hope I'll do it during week-end cos I'm currently a bit in a stress.

Jiri

Sorry, I'm totally crazy. I thought about another bug that I closed today together with this one.

As far as I know the bug should be fixed in current F12 release. I will check for it.

Jiri

Seems that you're right, I tried ppp-2.4.5 from devel (it required NetworkManager rebuild) and the problem seems to be gone. Please coordinate with Dan and push the fixed ppp to stable :)

This problem also exists in RHEL5. Very annoying :-/

I am using the following /etc/ppp/ip-up.local script to force my ISP's DNS servers:

----
#!/bin/bash

if [ "$DNS1" -o "$DNS2" ]; then
    echo "### File created by /etc/ppp/ip-up.local on $(date)" >/etc/resolv.conf

    if [ "${DNS1:0:3}" == "10." -o "${DNS1:0:3}" == "10." ]; then
        DNS1="212.224.255.252"
        DNS2="212.65.63.218"
    fi

    for NAMESERVER in $DNS1 $DNS2; do
        logger -t ppp "Configuring nameserver $NAMESERVER in /etc/resolv.conf"
        echo "nameserver $NAMESERVER" >>/etc/resolv.conf
    done
fi
----