Bug 924359

Summary: [abrt] systemd-197-1.fc18.2: log_assert: Process /usr/lib/systemd/systemd-journald was killed by signal 6 (SIGABRT)
Product: [Fedora] Fedora Reporter: Peter Janes <bugzilla.redhat.com>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: cristian.ciupitu, johannbg, lnykryn, metherid, mschmidt, msekleta, notting, plautrba, systemd-maint, vpavlin, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: abrt_hash:a9bc20305723793df6e8133e2952ae8294c8d223
Fixed In Version: systemd-201-2.fc18.6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-05-16 03:01:31 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
File: backtrace
none
File: cgroup
none
File: core_backtrace
none
File: dso_list
none
File: environ
none
File: limits
none
File: maps
none
File: open_fds
none
File: proc_pid_status
none
File: smolt_data
none
File: var_log_messages
none
Ansible playbook for crashing journal
none
Python script that crashes the journal none

Description Peter Janes 2013-03-21 15:41:52 UTC
Description of problem:
I've been testing some Ansible playbooks, running lineinfile tasks against the local host.  All had been working well, but suddenly I was no longer able to execute the playbooks due to a "Connection refused" error.  System logs showed

Mar 21 11:32:23 peterj systemd[1]: systemd-journald.service: main process exited, code=dumped, status=6/ABRT

Restarting the iptables service fixes the problem temporarily, but I often can't run the playbook to completion because systemd-journald will crash during execution.

Version-Release number of selected component:
systemd-197-1.fc18.2

Additional info:
backtrace_rating: 4
cmdline:        /usr/lib/systemd/systemd-journald
crash_function: log_assert
executable:     /usr/lib/systemd/systemd-journald
kernel:         3.7.9-205.fc18.x86_64
uid:            0

Truncated backtrace:
Thread no. 1 (6 frames)
 #2 log_assert at src/shared/log.c:702
 #3 log_assert_failed at src/shared/log.c:707
 #4 dispatch_message_real at src/journal/journald-server.c:538
 #5 server_dispatch_message at src/journal/journald-server.c:795
 #6 server_process_native_message at src/journal/journald-native.c:260
 #7 process_event at src/journal/journald-server.c:1159

Comment 1 Peter Janes 2013-03-21 15:42:00 UTC
Created attachment 713940 [details]
File: backtrace

Comment 2 Peter Janes 2013-03-21 15:42:01 UTC
Created attachment 713941 [details]
File: cgroup

Comment 3 Peter Janes 2013-03-21 15:42:10 UTC
Created attachment 713942 [details]
File: core_backtrace

Comment 4 Peter Janes 2013-03-21 15:42:12 UTC
Created attachment 713943 [details]
File: dso_list

Comment 5 Peter Janes 2013-03-21 15:42:14 UTC
Created attachment 713944 [details]
File: environ

Comment 6 Peter Janes 2013-03-21 15:42:16 UTC
Created attachment 713945 [details]
File: limits

Comment 7 Peter Janes 2013-03-21 15:42:18 UTC
Created attachment 713946 [details]
File: maps

Comment 8 Peter Janes 2013-03-21 15:42:20 UTC
Created attachment 713947 [details]
File: open_fds

Comment 9 Peter Janes 2013-03-21 15:42:22 UTC
Created attachment 713948 [details]
File: proc_pid_status

Comment 10 Peter Janes 2013-03-21 15:42:24 UTC
Created attachment 713949 [details]
File: smolt_data

Comment 11 Peter Janes 2013-03-21 15:42:26 UTC
Created attachment 713950 [details]
File: var_log_messages

Comment 12 Michal Schmidt 2013-03-21 15:59:14 UTC
This is the assertion that failed:
        assert(n + N_IOVEC_META_FIELDS <= m);

n was 20
N_IOVEC_META_FIELDS is 17
m was 36

Comment 13 Cristian Ciupitu 2013-04-15 21:23:39 UTC
It also happened to me while testing some Ansible playbooks.

Comment 14 Michal Schmidt 2013-04-16 08:54:59 UTC
It would be nice if you could provide more exact steps to reproduce. I am not familiar with "Ansible playbooks", so going into details would be helpful.

Comment 15 Cristian Ciupitu 2013-04-16 14:23:38 UTC
Created attachment 736321 [details]
Ansible playbook for crashing journal

The playbook was tested with ansible-1.2-0.git201304142331.fc18.noarch and
systemd-201-2.fc18.2.x86_64. It installs cobbler & koan and configures
/etc/cobbler/settings (nothing is really changed). It uses the local
connection, so there's no need to start the SSH server.

To install ansible-1.2-0.git201304142331.fc18.noarch:

	% cd /tmp
	% git clone git://github.com/ansible/ansible.git
	% cd ansible
	% git checkout -b bug-924359 84fb92dac6bfd70b10fedf3f5a8758a684aa9dcb
	% make rpm # some packages might be required for this

	# yum install /tmp/ansible/rpm-build/ansible-1.2-0.git201304142331.fc18.noarch.rpm

To run the playbook:

	% cd /tmp
	% tar xJf /tmp/bug-924359-ansible-playbook.tar.xz

	# cd /tmp/bug-924359-ansible-playbook
	# ansible-playbook -i hosts site.yml

The journal should crash.

Comment 16 Cristian Ciupitu 2013-04-16 14:24:49 UTC
P.S. I'm sorry that I could make the playbook simpler, but the simpler versions didn't crash the journal.

Comment 17 Cristian Ciupitu 2013-04-16 16:16:50 UTC
Created attachment 736429 [details]
Python script that crashes the journal

The script for crashing the journal was tested with:
systemd-201-2.fc18.2.x86_64.rpm
systemd-libs-201-2.fc18.2.i686.rpm
systemd-libs-201-2.fc18.2.x86_64.rpm
systemd-python-201-2.fc18.2.x86_64.rpm
systemd-sysv-201-2.fc18.2.x86_64.rpm
python-2.7.3-13.fc18.x86_64.rpm

Comment 18 Michal Schmidt 2013-04-16 21:13:41 UTC
Thank you for the reproducer. I fixed the bug upstream:
http://cgit.freedesktop.org/systemd/systemd/commit/?id=f6422def2c10aa0dea1b872d2f187853e61bd015

Comment 19 Fedora Update System 2013-04-17 13:37:34 UTC
systemd-201-2.fc18.5 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/FEDORA-2013-5452/systemd-201-2.fc18.5

Comment 20 Fedora Update System 2013-04-18 02:42:20 UTC
Package systemd-201-2.fc18.5:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-201-2.fc18.5'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-5452/systemd-201-2.fc18.5
then log in and leave karma (feedback).

Comment 21 Fedora Update System 2013-05-07 13:44:24 UTC
systemd-201-2.fc18.6 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/FEDORA-2013-5452/systemd-201-2.fc18.6

Comment 22 Fedora Update System 2013-05-09 10:05:33 UTC
Package systemd-201-2.fc18.6:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing systemd-201-2.fc18.6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-5452/systemd-201-2.fc18.6
then log in and leave karma (feedback).

Comment 23 Fedora Update System 2013-05-16 03:01:31 UTC
systemd-201-2.fc18.6 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.