Bug 1615617 - nuxwdog systemd - memory error when starting subCA
Summary: nuxwdog systemd - memory error when starting subCA
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: nuxwdog   
(Show other bugs)
Version: 7.5
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Dinesh Prasanth
QA Contact: Asha Akkiangady
Marc Muehlfeld
URL:
Whiteboard:
Keywords: TestCaseProvided
Depends On:
Blocks: 1618858
TreeView+ depends on / blocked
 
Reported: 2018-08-13 21:03 UTC by Asha Akkiangady
Modified: 2018-10-30 11:50 UTC (History)
10 users (show)

Fixed In Version: nuxwdog-1.0.3-8.el7
Doc Type: Bug Fix
Doc Text:
The *nuxwdog* service starts correctly when a sub-CA is installed Previously, if a sub-CA was installed, the *nuxwdog* service did not allocate enough memory. As a consequence, the service failed to start. This update fixes the problem. As a result, *nuxwdog* starts correctly in the mentioned scenario.
Story Points: ---
Clone Of:
: 1618858 (view as bug list)
Environment:
Last Closed: 2018-10-30 11:50:43 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3329 None None None 2018-10-30 11:50 UTC

Description Asha Akkiangady 2018-08-13 21:03:38 UTC
Description of problem:
When subCA is installed and nuxwdog is enabled for that instance, the server fails to start. It seems some kind of unintended memory free operation is being performed causing the error.

Version-Release number of selected component (if applicable):
systemd-219-57.el7_5.2.x86_64
nuxwdog-1.0.3-7.el7.x86_64

How reproducible:


Steps to Reproduce:
1. Install RootCA, enable nuxwdog and start RootCA. Everything works fine.
2. Install SubCA, enable nuxwdog and start RootCA shows following error:

Actual results:

Aug 13 15:15:12 <hostname> nuxwdog[23862]: *** Error in `/bin/nuxwdog': free(): invalid next size (fast): 0x00000000021c2e70 ***
Aug 13 15:15:12 <hostname> nuxwdog[23862]: ======= Backtrace: =========
Aug 13 15:15:12 <hostname> nuxwdog[23862]: /lib64/libc.so.6(+0x81429)[0x7fd515447429]
Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x405698]
Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x4044ac]
Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x402d9d]
Aug 13 15:15:12 <hostname>[23862]: /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fd5153e83d5]
Aug 13 15:15:12 <hostname>[23862]: /bin/nuxwdog[0x40323f]
Aug 13 15:15:12 <hostname> nuxwdog[23862]: ======= Memory map: ========
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 00400000-0040a000 r-xp 00000000 fd:00 3309616                            /usr/bin/nuxwdog
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 00609000-0060a000 r--p 00009000 fd:00 3309616                            /usr/bin/nuxwdog
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 0060a000-0060b000 rw-p 0000a000 fd:00 3309616                            /usr/bin/nuxwdog
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 0060b000-0060c000 rw-p 00000000 00:00 0
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 021bd000-021de000 rw-p 00000000 00:00 0                                  [heap]
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd510000000-7fd510021000 rw-p 00000000 00:00 0
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd510021000-7fd514000000 ---p 00000000 00:00 0
Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd514929000-7fd514935000 r-xp 00000000 fd:00 33612770                   /usr/lib64/libnss_files-2.17.so



Expected results:
SubCA server should start successfully.

Additional info:

Further investigation shows that when nuxwdog is started from commandline by following the guidelines from here: http://www.dogtagpki.org/wiki/Nuxwdog/HOWTO#Running_nuxwdog the server starts successfully.

Root Cause of the Issue: The nuxwdog and systemd communication might be broken which causes this issue

Comment 1 Asha Akkiangady 2018-08-13 22:27:53 UTC
Sorry, I meant to start SubCA in step 2. Correct instructions:

(In reply to Asha Akkiangady from comment #0)
> Description of problem:
> When subCA is installed and nuxwdog is enabled for that instance, the server
> fails to start. It seems some kind of unintended memory free operation is
> being performed causing the error.
> 
> Version-Release number of selected component (if applicable):
> systemd-219-57.el7_5.2.x86_64
> nuxwdog-1.0.3-7.el7.x86_64
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1. Install RootCA, enable nuxwdog and start RootCA. Everything works fine.
> 2. Install SubCA, enable nuxwdog and start SubCA shows following error:
> 
> Actual results:
> 
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: *** Error in `/bin/nuxwdog':
> free(): invalid next size (fast): 0x00000000021c2e70 ***
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: ======= Backtrace: =========
> Aug 13 15:15:12 <hostname> nuxwdog[23862]:
> /lib64/libc.so.6(+0x81429)[0x7fd515447429]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x405698]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x4044ac]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x402d9d]
> Aug 13 15:15:12 <hostname>[23862]:
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fd5153e83d5]
> Aug 13 15:15:12 <hostname>[23862]: /bin/nuxwdog[0x40323f]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: ======= Memory map: ========
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 00400000-0040a000 r-xp 00000000
> fd:00 3309616                            /usr/bin/nuxwdog
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 00609000-0060a000 r--p 00009000
> fd:00 3309616                            /usr/bin/nuxwdog
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 0060a000-0060b000 rw-p 0000a000
> fd:00 3309616                            /usr/bin/nuxwdog
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 0060b000-0060c000 rw-p 00000000
> 00:00 0
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 021bd000-021de000 rw-p 00000000
> 00:00 0                                  [heap]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd510000000-7fd510021000 rw-p
> 00000000 00:00 0
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd510021000-7fd514000000 ---p
> 00000000 00:00 0
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd514929000-7fd514935000 r-xp
> 00000000 fd:00 33612770                   /usr/lib64/libnss_files-2.17.so
> 
> 
> 
> Expected results:
> SubCA server should start successfully.
> 
> Additional info:
> 
> Further investigation shows that when nuxwdog is started from commandline by
> following the guidelines from here:
> http://www.dogtagpki.org/wiki/Nuxwdog/HOWTO#Running_nuxwdog the server
> starts successfully.
> 
> Root Cause of the Issue: The nuxwdog and systemd communication might be
> broken which causes this issue

Comment 3 Jan Synacek 2018-08-14 06:51:09 UTC
Aug 13 15:15:12 <hostname> nuxwdog[23862]: *** Error in `/bin/nuxwdog': free(): invalid next size (fast): 0x00000000021c2e70 ***

Comment 4 Dinesh Prasanth 2018-08-14 16:30:26 UTC
When `/bin/nuxwdog` is started normally from commandline (without the use of `systemctl` command), it works as expected. This error appears only when the `nuxwdog` is started using systemd.

Comment 5 Michal Schmidt 2018-08-16 11:03:20 UTC
Could you provide more exact steps to reproduce? For someone who knows systemd, but not the other components involved.

Comment 6 Dinesh Prasanth 2018-08-16 13:55:49 UTC
The error is kind of reproducible only in NetHSM-OCS environment. NetHSM-OCS is a hardware for key/cert store which requires an OCS (Operator Card) and password (that nuxwdog tries to get from user through the prompt) to access.

I was trying to debug and analyze the issue and found few things:

* Instead of starting nuxwdog with `systemctl start pki-tomcatd-nuxwdog@<instance>`, if i just do `/bin/nuxwdog -f <path to nuxwdog.conf>`, there is no memory error and everything works as expected
* When .service file for pki-tomcatd-nuxwdog was edited to start with valgrind to check for memory errors, there is no memory error and everything works as expected
* When I attached the `gdb` tool to check for the stack trace, I found the following:
(1) the issue was with this free: https://github.com/dogtagpki/nuxwdog/blob/master/src/com/redhat/nuxwdog/wdpwd.cpp#L288
(2) When I tried to comment out the above free, I got a new malloc error in this line: https://github.com/dogtagpki/nuxwdog/blob/master/src/com/redhat/nuxwdog/wdservermessage.cpp#L97  
stack trace is available here: https://paste.fedoraproject.org/paste/9P04rUYneL4Mt0ovLM7K1Q

This is running on RHEL-7.5

Comment 7 Michal Schmidt 2018-08-16 14:06:51 UTC
(In reply to Dinesh Prasanth from comment #6)
> * When I attached the `gdb` tool to check for the stack trace, I found the
> following:
> (1) the issue was with this free:
> https://github.com/dogtagpki/nuxwdog/blob/master/src/com/redhat/nuxwdog/
> wdpwd.cpp#L288

250   char *keyname = (char *) malloc(strlen(pwdname) + strlen(KEY_PREFIX));
251   sprintf(keyname, "%s%s", KEY_PREFIX, pwdname);

The allocation is one byte too short, missing space for the '\0' string terminator.

Comment 8 Michal Schmidt 2018-08-16 14:11:11 UTC
As a side note, this pattern appear common in that code:
   if (X) free(X);

free(NULL) is a valid no-op, so there is no need for the check.

OTOH, I spotted several missing checks for NULL after malloc().

Comment 9 Dinesh Prasanth 2018-08-16 15:34:41 UTC
(In reply to Michal Schmidt from comment #7)
> 
> 250   char *keyname = (char *) malloc(strlen(pwdname) + strlen(KEY_PREFIX));
> 251   sprintf(keyname, "%s%s", KEY_PREFIX, pwdname);
> 
> The allocation is one byte too short, missing space for the '\0' string
> terminator.

Yeah. I think that's the culprit code. Thanks for pointing me in the right direction.

Also, the package is kind of old and needs to be thoroughly cleaned. I will keep your point in mind while doing that! Thank you!

Comment 11 Dinesh Prasanth 2018-08-16 21:18:28 UTC
Steps to reproduce:

As mentioned by the OP, this error occurred only in the NHSM-OCS environment. This is considered to be a heisenbug and it may/may not be reproducible. Regardless, this error should be solved.

Comment 12 Dinesh Prasanth 2018-08-16 21:20:09 UTC
Fixed in upstream master: https://github.com/dogtagpki/nuxwdog/commit/63d0cb4948d240068be52c5b0f701a9524320d4d

Comment 13 Fedora Update System 2018-08-17 16:27:14 UTC
nuxwdog-1.0.5-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-59a30f5bcf

Comment 15 Roshni 2018-09-05 19:52:42 UTC
[root@nocp1 ~]# rpm -qi nuxwdog
Name        : nuxwdog
Version     : 1.0.3
Release     : 8.el7
Architecture: x86_64
Install Date: Thu 30 Aug 2018 01:34:56 PM EDT
Group       : System Environment/Libraries
Size        : 103659
License     : LGPLv2 and (GPL+ or Artistic)
Signature   : RSA/SHA256, Tue 21 Aug 2018 02:40:55 AM EDT, Key ID 199e2f91fd431d51
Source RPM  : nuxwdog-1.0.3-8.el7.src.rpm
Build Date  : Tue 21 Aug 2018 02:03:38 AM EDT
Build Host  : x86-017.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.redhat.com/certificate_system
Summary     : Watchdog server to start and stop processes, and prompt for passwords


CA and subca installation using NHSM600-OCS and nuxwdog enabled was successful.

Comment 18 errata-xmlrpc 2018-10-30 11:50:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3329


Note You need to log in before you can comment on or make changes to this bug.