Bug 1357010

Summary: snapperd: abrtd detects error when snapperd finishes running on debug mode
Product: Red Hat Enterprise Linux 7 Reporter: Bruno Goncalves <bgoncalv>
Component: snapperAssignee: Ondrej Kozina <okozina>
snapper sub component: general QA Contact: Bruno Goncalves <bgoncalv>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: low CC: agk, okozina, prajnoha
Version: 7.3   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: snapper-0.2.8-3.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 07:29:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Bruno Goncalves 2016-07-15 13:25:41 UTC
Description of problem:
After running snapperd with -d parameter abrtd reports error:

# abrt-cli list
id 597b95d027aeb02afc5cf4c764032f295ea1b84f
reason:         snapperd killed by SIGABRT


Version-Release number of selected component (if applicable):
snapper-0.2.8-2.el7

How reproducible:
100%

Steps to Reproduce:
1. enable snapper -d
# cat /usr/share/dbus-1/system-services/org.opensuse.Snapper.service
# DBus service activation config
[D-BUS Service]
Name=org.opensuse.Snapper
Exec=/usr/sbin/snapperd -d
User=root

2. Create thin LV
3. Create FS on thin LV
4. Mount thin LV
# mount /dev/mapper/rhel-thin_lv /mnt/test
5. Create snapper config
# snapper -c bugtest create-config -f "lvm(xfs)" /mnt/test
6. Create data on mount point
cd /mnt/test
mkdir dir_1
touch dir_1/file_1
7. # snapper -c bugtest list
Type   | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+------+------+---------+-------------+---------
single | 0 |       |      | root |         | current     |   

8. # snapper -c bugtest create -t pre -p
1

9. # snapper -c bugtest create -t post --pre-num 1 -p
2

10. # snapper -c bugtest list
Type   | # | Pre # | Date                         | User | Cleanup | Description | Userdata
-------+---+-------+------------------------------+------+---------+-------------+---------
single | 0 |       |                              | root |         | current     |         
pre    | 1 |       | Fri 15 Jul 2016 09:09:39 EDT | root |         |             |         
post   | 2 | 1     | Fri 15 Jul 2016 09:09:44 EDT | root |         |             |     

11. # ps -ef | grep snapper
root      2402     1  0 09:09 ?        00:00:00 /usr/sbin/snapperd -d
root      2492  2114  0 09:09 pts/0    00:00:00 grep --color=auto snapper

12. wait 1 minute so snapperd finishes

13. # abrt-cli list
id 597b95d027aeb02afc5cf4c764032f295ea1b84f
reason:         snapperd killed by SIGABRT
time:           Fri 15 Jul 2016 09:10:49 EDT
cmdline:        /usr/sbin/snapperd -d
package:        snapper-0.2.8-2.el7


Actual results:
abrt founds error

Expected results:
no error should be reported by abrt

Additional info:
This problem does not happen if not running on debug mode

Comment 1 Ondrej Kozina 2016-08-04 11:23:25 UTC
The following bt was extracted from provided coredump, explanation will follow.

Program terminated with signal 6, Aborted.
#0  0x00007f37a16e91d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x00007f37a16e91d7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f37a16ea8c8 in __GI_abort () at abort.c:90
#2  0x00007f37a1fedab5 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007f37a1feba26 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:38
#4  0x00007f37a1feba53 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
#5  0x00007f37a1febc73 in __cxxabiv1::__cxa_throw (obj=0x7f3790001b00, tinfo=0x4321a0 <typeinfo for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >>, 
    dest=0x413a50 <boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >::~clone_impl()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:87
#6  0x0000000000414514 in boost::throw_exception<boost::lock_error> (e=...) at /usr/include/boost/throw_exception.hpp:67
#7  0x00007f37a3db4dee in lock (this=<optimized out>) at /usr/include/boost/thread/pthread/mutex.hpp:71
#8  lock_guard (m_=..., this=<optimized out>) at /usr/include/boost/thread/lock_guard.hpp:38
#9  simple_log_do (component="libsnapper", text="worker interrupted", func=0x7f379fd5abe0 "\350\r", line=129, file=0x438366 "Background.cc", level=4424550, this=<optimized out>, this=<optimized out>, this=<optimized out>) at Logger.cc:74
#10 snapper::callLogDo (level=level@entry=(anonymous namespace)::DEBUG, component="libsnapper", file=file@entry=0x438366 "Background.cc", line=line@entry=129, func=func@entry=0x43857d <Backgrounds::worker()::__FUNCTION__> "worker", text="worker interrupted")
    at Logger.cc:115
#11 0x00007f37a3db4609 in snapper::logStreamClose (level=level@entry=(anonymous namespace)::DEBUG, file=file@entry=0x438366 "Background.cc", line=line@entry=129, func=func@entry=0x43857d <Backgrounds::worker()::__FUNCTION__> "worker", stream=stream@entry=0x7f3790001830)
    at Log.cc:64
#12 0x0000000000427125 in Backgrounds::worker (this=0x648520 <backgrounds>) at Background.cc:129
#13 0x00007f37a3b3e27a in boost::(anonymous namespace)::thread_proxy (param=<optimized out>) at libs/thread/src/pthread/thread.cpp:165
#14 0x00007f37a26cfdc5 in start_thread (arg=0x7f379fd5b700) at pthread_create.c:308
#15 0x00007f37a17ab6ed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:72
#16 0x0000000000000000 in ?? ()

There's clash between lifetime of static object mutex in Logger.cc routine simple_log_do() and global Backgrounds object in Backgrounds.cc. Moreover the mutex is in namespace snapper whereas the Backgrounds object is in no (implicit?) namespace.

The Backgrounds object destructor is trying to write debug log message (so much for this bug being triggered only in debug mode) when static object mutex was already terminated which caused the program abort.

This bug is annoying, true, but I believe it's harmless in the same time since it's triggered only during daemon shutdown.

I have already simple workaround in my mind, but let me dig deeper in C++ std so i can propose correct fix for upstream as well.

Reproduced on snapper-0.1.7-10.el7, snapper-0.2.8-2.el7 as well as on fresh upstream 0.3.3. It is not a regression.

Comment 2 Ondrej Kozina 2016-08-08 12:41:47 UTC
Fix proposed upstream: https://github.com/openSUSE/snapper/pull/258

Comment 3 Ondrej Kozina 2016-08-10 09:23:48 UTC
Merged in upstream

Comment 5 Bruno Goncalves 2016-08-10 15:03:31 UTC
Tested snapper-0.2.8-3.el7 and I have not seen the problem any more.

Comment 8 errata-xmlrpc 2016-11-04 07:29:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2481.html