Bug 1369274

Summary: lsyncd crash due to improper shell wrapping in systemd unit
Product: [Fedora] Fedora EPEL Reporter: Dirk Dankhoff <bluthund23+redhat>
Component: lsyncdAssignee: Jason Taylor <jtfas90>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: epel7CC: ferdnyc, filip, jtfas90, lkundrak, martin, pwouters, scenek, troxor0
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lsyncd-2.2.2-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-28 01:17:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dirk Dankhoff 2016-08-22 22:37:26 UTC
Description of problem:
lsyncd is killed by its parent process (/usr/bin/sh) because of resource exhaustion after a lot of output has been generated (~2G). This happens because the shipped systemd service unit starts lsyncd inside a command substitution construct (backticks) which buffers all of its output (in this specific case for later evaluation by eval; which is probably also incorrect by itself, see additional info at the end).

journalctl -u lsyncd:
> /usr/bin/sh: xrealloc: cannot allocate 18446744071562067968 bytes (24576 bytes allocated).

Version-Release number of selected component (if applicable):
2.1.5-6.el7

How reproducible:
Always but highly dependent on the amount of output generated by the processes spawned by lsyncd; using a custom lsync onAction-syncer speeds up the repro tremendously (alternatively using deeply nested paths and verbose rsync logging it still takes a long time to trigger this)

Steps to Reproduce:
1. # mkdir -p /tmp/lsyncd
2. # cat >/etc/lsyncd.conf <<EOF
yes = {
  onCreate = "/usr/bin/yes",
}

sync {
  yes,
  source = "/tmp/lsyncd/"
}
EOF

3. # systemctl start lsyncd.service
4. # touch /tmp/lsyncd/a; while systemctl is-active -q lsyncd.service; do sleep 2; done; systemctl status lsyncd.service

Actual results:
System memory usage increases (used by /usr/bin/sh). The while-loop terminates after a few seconds because lsyncd is killed after its grandparent /usr/bin/sh has allocated ~2GiB of system memory and exits because xrealloc() fails to allocate more memory.

Expected results:
lsyncd continues to run indefinitely and independent of the amount of output generated. System memory usage doesn't increase because of lsyncd's output (at least not directly correlating to its total amount over the course of its runtime).

Additional info:
The systemd unit should probably simply use

ExecStart=/usr/bin/lsyncd -nodaemon $LSYNCD_OPTIONS /etc/lsyncd.conf

This also fixes that lsyncd's output is not logged to the journal using the current version of the unit (since the output never reaches systemd as its buffered by the shell for evaluation).

The 'eval ``'-construct seems to be in place to allow nested variables in /etc/sysconfig/lsyncd. But afaict the way it is used here is not as the original author of the header comment (in /etc/sysconfig/lsyncd) intended.

Comment 1 Fedora Update System 2017-01-14 14:32:22 UTC
lsyncd-2.2.1-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-5cf41b89e0

Comment 2 Fedora Update System 2017-01-14 15:10:19 UTC
lsyncd-2.2.1-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2d46983f2e

Comment 3 Fedora Update System 2017-01-15 10:50:00 UTC
lsyncd-2.2.1-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2d46983f2e

Comment 4 Fedora Update System 2017-01-15 11:19:47 UTC
lsyncd-2.2.1-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-5cf41b89e0

Comment 5 "FeRD" (Frank Dana) 2017-04-10 18:14:46 UTC
This was indeed fixed in 2.2.1-1 (which was unpushed due to bug #1415295), and subsequently released to stable as part of lsyncd-2.2.2-1.fc25.

Long and short, I believe this bug can now be closed?

Comment 6 Dirk Dankhoff 2017-04-10 20:53:36 UTC
While true for Fedora this bug still affects the package on EL7 EPEL.

A release of lsyncd 2.2.x on el7 probably requires a version bump of rsync on the core repos or its addition to the EPEL since the application's dependency on rsync >= 3.1.0 isn't met by the currently available core packages on el7 (see https://bodhi.fedoraproject.org/updates/lsyncd-2.2.1-1.el7#comment-550536).

Comment 7 Jason Taylor 2017-04-11 00:40:46 UTC
Hi Frank/Dirk,

As Dirk mentioned, this bug still effects the epel version. I haven't had a chance to get back to fixing the epel version.

I should be able to address it this week though. Thanks for checking back in on this and sorry for the delay.

JT

Comment 8 Fedora Update System 2017-04-12 13:43:52 UTC
lsyncd-2.2.2-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-fddfb1205f

Comment 9 Fedora Update System 2017-04-13 00:48:22 UTC
lsyncd-2.2.2-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-fddfb1205f

Comment 10 Fedora Update System 2017-04-28 01:17:23 UTC
lsyncd-2.2.2-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.