Bug 1298732

Summary: regression: Fails to connect to Red Hat VPN
Product: Red Hat Enterprise Linux 7 Reporter: Colin Walters <walters>
Component: NetworkManagerAssignee: Dan Williams <dcbw>
Status: CLOSED ERRATA QA Contact: Desktop QE <desktop-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: alex, atodorov, bgalvani, dcbw, fzdarsky, huzaifas, lkundrak, lrintel, markus, nobody+bgollahe, nobody, oliver, psimerda, ptalbert, rbonser, rkhan, rmcgover, rsawhill, rstrode, tcameron, thaller, trevor.hemsley, tsorense, vbenes, walters
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-03 19:06:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Colin Walters 2016-01-14 20:48:06 UTC 
I had been using NetworkManager-vpnc-0.9.9.0-6.git20140428.el7 with the Red Hat corporate VPN successfully.  http://koji.fedoraproject.org/koji/buildinfo?buildID=701213 disconnects immediately during authentication.

Stracing is pretty strange, looks like the plugin decides to kill itself:

000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\300\33\30\0\0\0\0\0\300\33\30\0\0\0\0\0\300\33\30\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\274[\33\0\0\0\0\0\274[\33\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0000g\33\0\0\0\0\0000g;\0\0\0\0\0000g;\0\0\0\0\0pQ\0\0\0\0\0\0\20\233\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\200\233\33\0\0\0\0\0\200\233;\0\0\0\0\0\200\233;\0\0\0\0\0\360\1\0\0\0\0\0\0\360\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0000g\33\0\0\0\0\0000g;\0\0\0\0\0000g;\0\0\0\0\0\20\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0P\345td\4\0\0\0\334\33\30\0\0\0\0\0\334\33\30\0\0\0\0\0\334\33\30\0\0\0\0\0<h\0\0\0\0\0\0<h\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0000g\33\0\0\0\0\0000g;\0\0\0\0\0000g;\0\0\0\0\0\3208\0\0\0\0\0\0\3208\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0P\217\177\365\371\200/\342\376R\375\237]\31\320\3272\325ZV\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0 \0\0\0\0\0\0\0\363\3\0\0\t\0\0\0\0\1\0\0\16\0\0\0\0000\20D\240 \2\1\210\3\346\220\305E\214\0\300\0\10\0\5\200\0`\300\200\0\r\212\f\0\4\20\0\210D2\10.@\210P<, \0162H&\204\300\214\4\10\0\2\2\16\241\254\32\4f\300\0\3002\0\300\0P\1 \201\10\204\v  ($\0\4 P\0\20X\200\312DB(\0\6\200\20\30B\0 @\200\0\tP\0Q\212@\20\0\0\0\0\10\0\0\21\20", 832) = 832
17311 fstat(3,  <unfinished ...>
17311 <... fstat resumed> {st_mode=S_IFREG|0755, st_size=2107816, ...}) = 0
17311 mmap(NULL, 3932736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0 <unfinished ...>
17311 <... mmap resumed> )              = 0x7f04c18e7000
17311 mprotect(0x7f04c1a9d000, 2097152, PROT_NONE <unfinished ...>
17144 kill(17311, SIGTERM <unfinished ...>
17311 <... mprotect resumed> )          = 0
17311 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=17144, si_uid=0} ---
17144 write(2, "** Message: Terminated vpnc daemon with PID 17311.\n", 51) = 51
17311 +++ killed by SIGTERM +++
17144 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=17311, si_status=SIGTERM, si_utime=0, si_stime=0} ---
Comment 1 Colin Walters 2016-01-14 20:48:30 UTC 
Workaround: yum downgrade https://kojipkgs.fedoraproject.org//packages/NetworkManager-vpnc/0.9.9.0/6.git20140428.el7/x86_64/NetworkManager-vpnc-0.9.9.0-6.git20140428.el7.x86_64.rpm https://kojipkgs.fedoraproject.org//packages/NetworkManager-vpnc/0.9.9.0/6.git20140428.el7/x86_64/NetworkManager-vpnc-gnome-0.9.9.0-6.git20140428.el7.x86_64.rpm
Comment 2 Oliver Ilian 2016-02-04 13:35:33 UTC 
do we have any update on this. This makes updating Satellite channels for my ~ 2000 really complicated, as I have to remove the broken NetworkManager-vpnc rpms manually after a channel sync

output from /var/log/messages:

Feb  4 14:31:59 ohaessle NetworkManager[10175]: <info>  Starting VPN service 'vpnc'...
Feb  4 14:31:59 ohaessle NetworkManager[10175]: <info>  VPN service 'vpnc' started (org.freedesktop.NetworkManager.vpnc), PID 11002
Feb  4 14:31:59 ohaessle NetworkManager[10175]: <info>  VPN service 'vpnc' appeared; activating connections
Feb  4 14:31:59 ohaessle NetworkManager[10175]: <info>  VPN plugin state changed: init (1)
Feb  4 14:32:09 ohaessle NetworkManager[10175]: <info>  VPN plugin state changed: starting (3)
Feb  4 14:32:09 ohaessle NetworkManager[10175]: <info>  VPN connection '1_test' (ConnectInteractive) reply received.
Feb  4 14:32:09 ohaessle NetworkManager[10175]: <info>  VPN plugin state changed: stopped (6)
Feb  4 14:32:09 ohaessle NetworkManager[10175]: <info>  VPN plugin state change reason: unknown (0)
Feb  4 14:32:09 ohaessle NetworkManager: ** Message: vpnc started with pid 11013
Feb  4 14:32:09 ohaessle NetworkManager: ** Message: Terminated vpnc daemon with PID 11013.
Feb  4 14:35:02 ohaessle systemd: Started Session 93 of user root.
Feb  4 14:35:02 ohaessle systemd: Starting Session 93 of user root.
Feb  4 14:35:10 ohaessle NetworkManager[10175]: <info>  VPN service 'vpnc' disappeared
Comment 3 Dan Williams 2016-02-24 14:54:33 UTC 
When you have time, here are some debug steps:

1) killall -TERM nm-vpnc-service
2) /usr/libexec/nm-vpnc-service --debug --persist
3) attempt a reconnect, wait for the failure, grab the logs
4) SANITIZE THE LOGS!!!

email them to me, or attach as a private attachment to bugzilla so we can analyze.
Comment 4 Dan Williams 2016-02-24 14:56:47 UTC 
Also, does this still happen with https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-2919d6d7d9 from Nov 2015?
Comment 5 Oliver Ilian 2016-02-25 07:59:11 UTC 
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2015-2919d6d7d9 was what was synced by now to our satellite channel and did not work. Will provide logs shortly.
Comment 8 Oliver Ilian 2016-02-25 10:31:27 UTC 
I also tried the http://koji.fedoraproject.org/koji/buildinfo?buildID=701213 on a fresh RHEL 7.1 and it worked without issues. Updated to RHEL 7.2 and it disconnected directly (stopped working)
Comment 9 Dan Williams 2016-02-26 18:57:10 UTC 
I debugged this a bunch today.  There are two causes:

1) the vpnc version in EPEL apparently doesn't have the patches I wrote in 2014 for interactive connect

2) the NM VPN service helper library is mishandling the error that the plugin returns when it knows vpnc cannot connect interactively, which causes the immediate failure
Comment 10 Dan Williams 2016-02-26 22:24:07 UTC 
Patches for NetworkManager's vpn-service library helpers posted here:

https://mail.gnome.org/archives/networkmanager-list/2016-February/msg00091.html
Comment 12 Colin Walters 2016-03-01 19:51:29 UTC 
Works for me.

https://mail.gnome.org/archives/networkmanager-list/2016-March/msg00006.html
Comment 13 Thomas Haller 2016-03-02 10:31:52 UTC 
Dan's patch got merged upstream:

master: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=abc700c5c71f474730f703c648b0b8dab455d7ba
nm-1-0: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=ba7359441b565c5ac6b0524b6aa04b155f0a9123
Comment 15 Oliver Ilian 2016-04-14 07:42:17 UTC 
Can we also have a backport for RHEL 7.2 with this Patch?
Comment 16 Oliver Ilian 2016-04-14 07:44:06 UTC 
*** Bug 1296220 has been marked as a duplicate of this bug. ***
Comment 17 Oliver Ilian 2016-04-14 07:44:23 UTC 
*** Bug 1293890 has been marked as a duplicate of this bug. ***
Comment 18 Oliver Ilian 2016-04-14 07:44:38 UTC 
*** Bug 1302755 has been marked as a duplicate of this bug. ***
Comment 19 Dan Williams 2016-04-14 19:26:01 UTC 
(In reply to Oliver Haessler from comment #15)
> Can we also have a backport for RHEL 7.2 with this Patch?

I think instead we should disable the interactive mode in NetworkManager-vpnc that causes this issue for EPEL until 7.3 is released.
Comment 23 Trevor Hemsley 2016-04-25 00:44:38 UTC 
Perhaps these packages should be pulled from EPEL until this is sorted out? I just yum updated my working 7.2 system and got new versions of NetworkManager-vpnc and -gnome and they broke my previously working connections. Reverting those packages to the previous versions immediately fixes the problem.

Broken combination:
Installed Packages
NetworkManager-vpnc.x86_64                            1:1.0.8-1.el7                                 @epel     
NetworkManager-vpnc-gnome.x86_64                      1:1.0.8-1.el7                                 @epel     
vpnc.x86_64                                           0.5.3-22.svn457.el7                           @epel     
vpnc-consoleuser.x86_64                               0.5.3-22.svn457.el7                           @epel     
vpnc-script.noarch                                    0.5.3-22.svn457.el7                           @epel
Comment 27 errata-xmlrpc 2016-11-03 19:06:26 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2581.html
Comment 28 Red Hat Bugzilla 2023-09-14 03:16:09 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days