Bug 458600

Summary: Verify command failed to execute: openssl verify -CAfile /etc/openvpn/PCAcert.pem
Product: [Fedora] Fedora EPEL Reporter: Robert Scheck <redhat-bugzilla>
Component: openvpnAssignee: Steven Pritchard <steve>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: medium    
Version: el5CC: earthbase2008, jlieskov, robert.scheck, thoger
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-11-02 22:50:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 458612    

Description Robert Scheck 2008-08-10 19:37:25 UTC
Description of problem:
I recently upgraded from openvpn-2.1-0.17.rc2.el5.1 (EPEL stable?) to the
latest openvpn-2.1-0.27.rc9.el5 (EPEL testing?) and now the OpenVPN server
seems to behave somehow odd/buggy/strange. From the log:

Aug 10 21:15:45 int-fw openvpn[22164]: x.y.z.a:1194 Re-using SSL/TLS context
Aug 10 21:15:45 int-fw openvpn[22164]: x.y.z.a:1194 LZO compression initialized
Aug 10 21:15:45 int-fw openvpn[22164]: x.y.z.a:1194 Control Channel MTU parms [ L:1542 D:166 EF:66 EB:0 ET:0 EL:0 ]
Aug 10 21:15:45 int-fw openvpn[22164]: x.y.z.a:1194 Data Channel MTU parms [ L:1542 D:1450 EF:42 EB:135 ET:0 EL:0 AF:3/1 ]
Aug 10 21:15:45 int-fw openvpn[22164]: x.y.z.a:1194 Local Options hash (VER=V4): '3f08d474'
Aug 10 21:15:45 int-fw openvpn[22164]: x.y.z.a:1194 Expected Remote Options hash (VER=V4): '02af3434'
Aug 10 21:15:45 int-fw openvpn[22164]: x.y.z.a:1194 TLS: Initial packet from x.y.z.a:1194, sid=e21a82a3 42077483
Aug 10 21:15:47 int-fw openvpn[22164]: x.y.z.a:1194 Verify command failed to execute: openssl verify -CAfile /etc/openvpn/PCAcert.pem 2 /C=DE/ST=Baden-Wuerttemberg/L=Stuttgart/O=PRIVATE/OU=Certification_Authority/CN=Root_Certification_Authority/emailAddress=pca@localhost
Aug 10 21:15:47 int-fw openvpn[22164]: x.y.z.a:1194 Exiting

After this, the OpenVPN server instance is gone away. Restarting brings no
change into this.

The file /etc/openvpn/server.conf contains the line:
tls-verify "openssl verify -CAfile /etc/openvpn/PCAcert.pem"

Version-Release number of selected component (if applicable):
openvpn-2.1-0.27.rc9

How reproducible:
Everytime, see above.

Actual results:
Unusable OpenVPN daemon since last update.

Expected results:
Working OpenVPN daemon again even with latest update state.

Additional info:
I've already set "script-security 3" in the configuration file or tried 
--script-security 3 as parameter when starting the daemon, no success. Only
downgrading to the older version was helpful and made sucessful OpenVPN
connections. This bug report IS NOT a duplicate of bug #458594, these are
two different problems.

Comment 1 Robert Scheck 2008-08-10 21:43:02 UTC
I digged in a bit into this crap and came to the conclusion, that this bug
report describes a regression caused by OpenVPN 2.1rc9. When using OpenVPN
2.1rc8, everything works fine. The only change, I would say is the cause for
this bug report is the replacement of system() -> execvp() which seems to
change the <STDIN> | tls-verify behaviour e.g. by different pipe behaviour
or by different line endings/foldings.

I know, that you're only downstream maintainer, but get please rid of this
bug report as soon as possible, as this is really a big issue in many many
VPN setups.

BTW, in 2.1rc8 the line which looks hooked up in 2.1rc9 looks like this:

Aug 10 23:34:04 int-fw openvpn[23225]: x.y.z.a:1194 VERIFY SCRIPT OK: depth=2, /C=DE/ST=Baden-Wuerttemberg/L=Stuttgart/O=PRIVATE/OU=Certification_Authority/CN=Root_Certification_Authority/emailAddress=pca@localhost

Comment 2 Robert Scheck 2008-08-10 22:09:09 UTC
Ah, BTW a my VPN configuration for example (nothing special, but just to
have it here as well):

dev tun0
port 1194
verb 3
mode server
server 10.254.0.0 255.255.255.0
tls-server
dh dh2048.pem
ca CA.pem
cert servercert.pem
key serverkey.pem
tls-auth serverhmac.key
tun-mtu 1500
tls-verify "openssl verify -CAfile /etc/openvpn/PCAcert.pem"
comp-lzo
keepalive 10 60
script-security 3

Comment 3 Robert Scheck 2008-08-13 18:41:16 UTC
-- COPY OF TWO COMMENTS FROM BUG REPORT #458594 TO HERE ---

Comment #4 From Jan Lieskovsky 2008-08-11 08:25:53 EDT

Hello Robert,

> Okay, there are now two problems for me with the latest OpenVPN you pushed
> into EPEL:
> 
> 1. Automagic "--script-security 3" in initscript passing to openvpn once
>    the configuration is not mentioning ^script-security.
> 2. tls-verify "openssl verify -CAfile /etc/openvpn/PCAcert.pem" doesn't
>    work any longer
>

Could you please specify in more detail what means "doesn't work any longer"?
It is:

a, the certificate verification as a service is not working any more
   (is broken)?

b, the certificate got invalid between upgrading between versions of the
   openvpn package (due miss to update some part in the certificate DN part)?

If a, is the case appearing here, it could be a potential security issue.

> The first problem we've to follow here. And IMPORTANT: I think, we need
> "--script-security 3" to have perfect backward compatibility; otherwise I
> saw some things failing here. For the second problem, I'm going to open
> another bug report now.

Have you already reported the new bug for the second issue?

Jan



Comment #5 From Tomas Hoger 2008-08-13 12:58:19 EDT

Comment #4 was previously mistakenly marked as private...

Robert, can you please give more details about what you mean by:

 2. tls-verify "openssl verify -CAfile /etc/openvpn/PCAcert.pem" doesn't
    work any longer

Is some invalid certificate getting accepted, or then OpenVPN tries to run this
external command, it blows up?

Thank you!
-- COPY OF TWO COMMENTS FROM BUG REPORT #458594 TO HERE ---

Comment 4 Robert Scheck 2008-08-13 18:54:39 UTC
Okay...here we go now: As written in the description, the certificate
verification itself doesn't simply work and this breaks down the whole
OpenVPN server for the first OpenVPN connect try by a VPN user (the
OpenVPN daemon just says "exiting" as per the log).

The certificate is still valid, at least it seems so and downgrading
makes the whole stuff working again. So it's IMHO neither a) nor b), I
would say, Jan. But breaking down the VPN server independent which of
the certificates (working with the old version) is used after the VPN
upgrade looks somehow like possible DoS for me.

No, OpenVPN "just" seems to blow up and exits the whole daemon, so no
other connects are possible, Tomas. Maybe some kind of self-protection,
I don't know. As it seems to me, the STDIN/argument behaviour by the
switch from system() -> execvp() is what I would make responsible for;
at least when I'm reading the changes as per the release changelog.

If I'm comparing (non-working)

Aug 10 21:15:47 int-fw openvpn[22164]: x.y.z.a:1194 Verify command failed to execute: openssl verify -CAfile /etc/openvpn/PCAcert.pem 2
/C=DE/ST=Baden-Wuerttemberg/L=Stuttgart/O=PRIVATE/OU=Certification_Authority/CN=Root_Certification_Authority/emailAddress=pca@localhost

and (working)

Aug 10 23:34:04 int-fw openvpn[23225]: x.y.z.a:1194 VERIFY SCRIPT OK: depth=2,
/C=DE/ST=Baden-Wuerttemberg/L=Stuttgart/O=PRIVATE/OU=Certification_Authority/CN=Root_Certification_Authority/emailAddress=pca@localhost

it looks like that the "2" is somewhere just around in the command as well as 
the DN string - executing the first command shown in the log also causes a
failure for me of course, thus I'm guessing that's a broken STDIN/STOUT and/or
argument handling. And I'm not aware, that openssl was/is changed by the vpn
update.

Comment 5 Steven Pritchard 2008-08-13 19:19:57 UTC
Have you tried specifying /usr/bin/openssl instead of just "openssl", by any chance?

Comment 6 Robert Scheck 2008-08-13 21:18:14 UTC
Yes, I tried this. But unluckily doesn't work as well (just the same):

Aug 13 23:12:18 int-fw openvpn[15187]: x.y.z.a:1194 TLS: Initial packet from x.y.z.a:1194, sid=3dbe1281 e7e9ca9b
Aug 13 23:12:20 int-fw openvpn[15187]: x.y.z.a:1194 Verify command failed to execute: /usr/bin/openssl verify -CAfile /etc/openvpn/PCAcert.pem 2 /C=DE/ST=Baden-Wuerttemberg/L=Stuttgart/O=PRIVATE/OU=Certification_Authority/CN=Root_Certification_Authority/emailAddress=pca@localhost
Aug 13 23:12:20 int-fw openvpn[15187]: x.y.z.a:1194 Exiting

Comment 7 Robert Scheck 2008-08-17 12:15:16 UTC
Any news?

Comment 8 Jan Lieskovsky 2008-08-25 08:37:37 UTC
Steve, any news on this issue?

Comment 9 Robert Scheck 2008-10-13 18:24:36 UTC
Can't see any difference here with OpenVPN 2.1rc13, still broken.

Comment 10 James Yonan 2008-10-14 05:15:21 UTC
I don't understand what you are trying to accomplish with this line:

tls-verify "openssl verify -CAfile /etc/openvpn/PCAcert.pem"

If you want an OpenVPN server to verify connecting clients against a CA, use the "ca" directive.  The openssl verify line, as you have stated it above, is going to look for a certificate as input on STDIN, and verify that against the CA cert PCAcert.pem.  The problem is that tls-verify does not use these semantics.  tls-verify does not send anything to the STDIN of the executed command.  Instead the tls-verify semantics above would cause openssl to be executed with your stated arguments plus two more arguments:  the verify_level (integer) and the X509 subject (string).  But this is not compatible with the semantics of the "openssl verify" command (see the man page for tls-verify semantics.

My guess is that your tls-verify directive above was previously a no-op, because tls-verify always appears to return a success status code.  With 21-rc9 and later, because we are now calling scripts or executables directly with execve() rather than through system and the shell, I'm thinking that the error you are seeing has something to do with the fact that execve is not seeing the openssl binary (remember to fully specify the pathname, don't depend on PATH).  But even when you get the openssl verify command to execute properly via "tls-verify", I believe your usage above is broken.  It would help to know what you are trying to accomplish with the tls-verify call.

James

Comment 11 Robert Scheck 2008-10-14 06:23:00 UTC
I don't know what the given tls-verify was intended for, I didn't setup that 
OpenVPN instance, but as somebody did who reported already multiple security issues against OpenVPN in the past, I think, the guy knows what he does.

And if you read correctly above, specifying the full path doesn't solve this
issue, so it is simply a regression (independent of it makes sense or not).

Comment 12 James Yonan 2008-10-30 04:36:38 UTC
Can you be more specific about "multiple security issues"?  There have been relatively few security issues filed against OpenVPN over the years, and no know issues are currently outstanding.

James

Comment 13 Robert Scheck 2008-11-02 22:50:39 UTC
Okay, colleague killed this issue internally as no-op, so no further reason
for me to follow this any longer...

P.S.: James, I just wrote "in the past" (e.g. CAN-2006-1629 was discovered
by the same guy who configured that tls-verify). And yes, there's nothing 
outstanding currently.