Bug 907204

Created attachment 692459 [details]
disable signals during startup

(This is cherry-picked from upstream commit 3429a6d00ca9ec7e3c138ddbfcfb767b70bd4301)

If an interrupt occurs at startup, depending on the state of processes,
segfault can occur. The patch masks signals when processes are starting up
and enables it back again after that.

Actually, http://sourceforge.net/tracker/index.php?func=detail&aid=3416054&group_id=128809&atid=712784 provides a patch for easy debug that forces an immediate provider exit and a core dump by calling abort() in handlerSigSegv() in providerDrv.c. Because When a provider causes a SIGSEGV, the provider is not immediately terminated. This can cause *many* repeating messages and not always ends in a core file for debugging.

Upstream bug report is in http://sourceforge.net/tracker/?func=detail&aid=3416060&group_id=128809&atid=712784. I clarify it a bit more here.
It's an race condition between a early pthread_key_delete(__mm_key) in uninitGarbageCollector() and pthread_getspecific(__mm_key) used by provider message handler functions. During initializing, sfcb fork a child to start httpdaemon and startup sort of providers, such as ClassProvider and ProfileProvider. If Http(s)Port cannot be bind, SIGQUIT will be sent to parent. CMPI_BrokerExt_Ftab->destroyThreadKey(__mm_key) will run by sighandler while children would use CMPI_BrokerExt_Ftab->getThreadSpecific(__mm_key) to check if provider is started normally. And an early destroying __mm_key leads segfault occur.

I have cherry-pick that patch from git commit 3429a6d00ca9ec7e3c138ddbfcfb767b70bd4301. This commit has been contained in sfcb-1.4.3. The patch just masks signals when processes are starting up
and enables it back again after that.

If you fixed that segfault, A more serious bug would be easily detected. It's a race condition during service stop which is reported in upstream as "double free during stopProc - ID: 3497096" (http://sourceforge.net/tracker/?func=detail&aid=3497096&group_id=128809&atid=712784).

It's because uninit_sfcBroker() cannot invoked from so many places. I've cherry-picked the patch from upstream commit 4699356c2c7973994aa0ad280d1a848c984d9a0f. It's contained in sfcb-1.4.3. This patch ensures only stopBroker() is responsible for stoping the whole services and it can only be called once.

Created attachment 692499 [details]
synchronize provider state

It's cherry-picked from upstream commit 4699356c2c7973994aa0ad280d1a848c984d9a0f.

Synchronize provider state, fix double free during stopProc().

I've tested these two patches on both i686 and x86_64 El6.4 beta. It works.

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unable to address this
request at this time.

Red Hat invites you to ask your support representative to
propose this request, if appropriate, in the next release of
Red Hat Enterprise Linux.

Hi, will it be addressed in 6.5?

Hi team,

Will this bug be fixed in RHEL6.8?

Best regards,
Shiyu Wang
Technical Account Manager

Created attachment 1132811 [details]
updated patch

Hi,

I revised the original patches sent by the reporter again and updated them with latter upstream changes, see:

https://sourceforge.net/p/sblim/sfcb-tix/94/
https://sourceforge.net/p/sblim/sfcb-tix/95/

Resulting patch attached.

Still, I'm not very satisfied with it... It fixes the SIGSEGV indeed, but doesn't work as I would expected (and as latest upstream version works) all the time.

Sometimes it works fine (well... with a little disturbing message "-#- Serious provider id / provider process mismatch --- terminating process.") and sometimes, unfortunately much more often, sfcbd doesn't kill all it's processes at shutdown. Two processes are left. The first one hangs no wait in stopLogging (and I don't think that is a real problem - it should finish after the second one finishes too automatically). The second one hangs in spGetMsg at msgqueue.c:243. This seems to be somehow related to another upstream bug:

https://sourceforge.net/p/sblim/sfcb-tix/97/

Currently, I'm trying to backport the changes from that bug and test them.

Created attachment 1136496 [details]
proposed patch

Final patch. sfcb terminates correctly, reaps all it's processes.

Note that SELinux policy needs to be adjusted accordingly. Although it works fine running sfcb directly from terminal ('sfcbd' or 'sfcbd -d'), running it through 'service sblim-sfcb start' leads to errors with SELinux enforcing:

Mar 15 09:24:38 ibm-ls22-01 sfcb[24767]:
Mar 15 09:24:38 ibm-ls22-01 sfcb[24767]: -- Http Proc semaphore create key: 0x48036b34 failed: File exists
Mar 15 09:24:38 ibm-ls22-01 sfcb[24767]:      use "ipcrm -S 0x48036b34" to remove semaphore
Mar 15 09:24:38 ibm-ls22-01 sfcb[24767]: 
Mar 15 09:24:40 ibm-ls22-01 sfcb[24767]: -#- InternalProvider - 24783 provider exiting due to a SIGSEGV signal
Mar 15 09:24:40 ibm-ls22-01 kernel: sfcbd[24783] general protection ip:7f13392807b9 sp:7fff9d37d950 error:0 in libsfcBrokerCore.so.0.0.0[7f1339263000+59000]

Generated AVCs in SELinux permissive:
type=AVC msg=audit(1458031161.288:760): avc:  denied  { unix_read unix_write } for  pid=24831 comm="sfcbd" key=1208183604  scontext=unconfined_u:system_r:sblim_sfcbd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=sem
type=AVC msg=audit(1458031161.288:760): avc:  denied  { associate } for  pid=24831 comm="sfcbd" key=1208183604  scontext=unconfined_u:system_r:sblim_sfcbd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=sem
type=SYSCALL msg=audit(1458031161.288:760): arch=c000003e syscall=64 success=yes exit=98307 a0=48036b34 a1=1 a2=180 a3=7ffc9aa39550 items=0 ppid=24827 pid=24831 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=112 comm="sfcbd" exe="/usr/sbin/sfcbd" subj=unconfined_u:system_r:sblim_sfcbd_t:s0 key=(null)
type=AVC msg=audit(1458031161.289:761): avc:  denied  { destroy } for  pid=24831 comm="sfcbd" key=1208183604  scontext=unconfined_u:system_r:sblim_sfcbd_t:s0 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=sem
type=SYSCALL msg=audit(1458031161.289:761): arch=c000003e syscall=66 success=yes exit=0 a0=18003 a1=0 a2=0 a3=0 items=0 ppid=24827 pid=24831 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=112 comm="sfcbd" exe="/usr/sbin/sfcbd" subj=unconfined_u:system_r:sblim_sfcbd_t:s0 key=(null)
type=AVC msg=audit(1458031161.288:762): avc:  denied  { unlink } for  pid=24830 comm="sfcbd" name="sfcbLocalSocket" dev=sda3 ino=4990472 scontext=unconfined_u:system_r:sblim_sfcbd_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=sock_file
type=SYSCALL msg=audit(1458031161.288:762): arch=c000003e syscall=87 success=yes exit=0 a0=7f5be3015ba1 a1=7f5be3015bb8 a2=7f5be2da6c86 a3=7f5be2da69f0 items=0 ppid=1 pid=24830 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=112 comm="sfcbd" exe="/usr/sbin/sfcbd" subj=unconfined_u:system_r:sblim_sfcbd_t:s0 key=(null)

Verified on all architectures with sblim-sfcb-1.3.11-5.el6.  sfcbd does not segfault anymore.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0648.html