Sorry, I meant to start SubCA in step 2. Correct instructions:

(In reply to Asha Akkiangady from comment #0)
> Description of problem:
> When subCA is installed and nuxwdog is enabled for that instance, the server
> fails to start. It seems some kind of unintended memory free operation is
> being performed causing the error.
> 
> Version-Release number of selected component (if applicable):
> systemd-219-57.el7_5.2.x86_64
> nuxwdog-1.0.3-7.el7.x86_64
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1. Install RootCA, enable nuxwdog and start RootCA. Everything works fine.
> 2. Install SubCA, enable nuxwdog and start SubCA shows following error:
> 
> Actual results:
> 
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: *** Error in `/bin/nuxwdog':
> free(): invalid next size (fast): 0x00000000021c2e70 ***
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: ======= Backtrace: =========
> Aug 13 15:15:12 <hostname> nuxwdog[23862]:
> /lib64/libc.so.6(+0x81429)[0x7fd515447429]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x405698]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x4044ac]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: /bin/nuxwdog[0x402d9d]
> Aug 13 15:15:12 <hostname>[23862]:
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fd5153e83d5]
> Aug 13 15:15:12 <hostname>[23862]: /bin/nuxwdog[0x40323f]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: ======= Memory map: ========
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 00400000-0040a000 r-xp 00000000
> fd:00 3309616                            /usr/bin/nuxwdog
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 00609000-0060a000 r--p 00009000
> fd:00 3309616                            /usr/bin/nuxwdog
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 0060a000-0060b000 rw-p 0000a000
> fd:00 3309616                            /usr/bin/nuxwdog
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 0060b000-0060c000 rw-p 00000000
> 00:00 0
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 021bd000-021de000 rw-p 00000000
> 00:00 0                                  [heap]
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd510000000-7fd510021000 rw-p
> 00000000 00:00 0
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd510021000-7fd514000000 ---p
> 00000000 00:00 0
> Aug 13 15:15:12 <hostname> nuxwdog[23862]: 7fd514929000-7fd514935000 r-xp
> 00000000 fd:00 33612770                   /usr/lib64/libnss_files-2.17.so
> 
> 
> 
> Expected results:
> SubCA server should start successfully.
> 
> Additional info:
> 
> Further investigation shows that when nuxwdog is started from commandline by
> following the guidelines from here:
> http://www.dogtagpki.org/wiki/Nuxwdog/HOWTO#Running_nuxwdog the server
> starts successfully.
> 
> Root Cause of the Issue: The nuxwdog and systemd communication might be
> broken which causes this issue

Aug 13 15:15:12 <hostname> nuxwdog[23862]: *** Error in `/bin/nuxwdog': free(): invalid next size (fast): 0x00000000021c2e70 ***

When `/bin/nuxwdog` is started normally from commandline (without the use of `systemctl` command), it works as expected. This error appears only when the `nuxwdog` is started using systemd.

Could you provide more exact steps to reproduce? For someone who knows systemd, but not the other components involved.

The error is kind of reproducible only in NetHSM-OCS environment. NetHSM-OCS is a hardware for key/cert store which requires an OCS (Operator Card) and password (that nuxwdog tries to get from user through the prompt) to access.

I was trying to debug and analyze the issue and found few things:

* Instead of starting nuxwdog with `systemctl start pki-tomcatd-nuxwdog@<instance>`, if i just do `/bin/nuxwdog -f <path to nuxwdog.conf>`, there is no memory error and everything works as expected
* When .service file for pki-tomcatd-nuxwdog was edited to start with valgrind to check for memory errors, there is no memory error and everything works as expected
* When I attached the `gdb` tool to check for the stack trace, I found the following:
(1) the issue was with this free: https://github.com/dogtagpki/nuxwdog/blob/master/src/com/redhat/nuxwdog/wdpwd.cpp#L288
(2) When I tried to comment out the above free, I got a new malloc error in this line: https://github.com/dogtagpki/nuxwdog/blob/master/src/com/redhat/nuxwdog/wdservermessage.cpp#L97  
stack trace is available here: https://paste.fedoraproject.org/paste/9P04rUYneL4Mt0ovLM7K1Q

This is running on RHEL-7.5

(In reply to Dinesh Prasanth from comment #6)
> * When I attached the `gdb` tool to check for the stack trace, I found the
> following:
> (1) the issue was with this free:
> https://github.com/dogtagpki/nuxwdog/blob/master/src/com/redhat/nuxwdog/
> wdpwd.cpp#L288

250   char *keyname = (char *) malloc(strlen(pwdname) + strlen(KEY_PREFIX));
251   sprintf(keyname, "%s%s", KEY_PREFIX, pwdname);

The allocation is one byte too short, missing space for the '\0' string terminator.

As a side note, this pattern appear common in that code:
   if (X) free(X);

free(NULL) is a valid no-op, so there is no need for the check.

OTOH, I spotted several missing checks for NULL after malloc().

(In reply to Michal Schmidt from comment #7)
> 
> 250   char *keyname = (char *) malloc(strlen(pwdname) + strlen(KEY_PREFIX));
> 251   sprintf(keyname, "%s%s", KEY_PREFIX, pwdname);
> 
> The allocation is one byte too short, missing space for the '\0' string
> terminator.

Yeah. I think that's the culprit code. Thanks for pointing me in the right direction.

Also, the package is kind of old and needs to be thoroughly cleaned. I will keep your point in mind while doing that! Thank you!

Steps to reproduce:

As mentioned by the OP, this error occurred only in the NHSM-OCS environment. This is considered to be a heisenbug and it may/may not be reproducible. Regardless, this error should be solved.

Fixed in upstream master: https://github.com/dogtagpki/nuxwdog/commit/63d0cb4948d240068be52c5b0f701a9524320d4d

nuxwdog-1.0.5-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-59a30f5bcf

[root@nocp1 ~]# rpm -qi nuxwdog
Name        : nuxwdog
Version     : 1.0.3
Release     : 8.el7
Architecture: x86_64
Install Date: Thu 30 Aug 2018 01:34:56 PM EDT
Group       : System Environment/Libraries
Size        : 103659
License     : LGPLv2 and (GPL+ or Artistic)
Signature   : RSA/SHA256, Tue 21 Aug 2018 02:40:55 AM EDT, Key ID 199e2f91fd431d51
Source RPM  : nuxwdog-1.0.3-8.el7.src.rpm
Build Date  : Tue 21 Aug 2018 02:03:38 AM EDT
Build Host  : x86-017.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.
URL         : http://www.redhat.com/certificate_system
Summary     : Watchdog server to start and stop processes, and prompt for passwords


CA and subca installation using NHSM600-OCS and nuxwdog enabled was successful.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3329