Bug 1072284

Summary:	beah runs tasks while systemd-readahead is still collecting readahead information
Product:	[Retired] Beaker	Reporter:	Alois Mahdal <amahdal>
Component:	beah	Assignee:	Amit Saha <asaha>
Status:	CLOSED CURRENTRELEASE	QA Contact:	tools-bugs <tools-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	0.15	CC:	aigao, asaha, dcallagh, ebaak, rmancy, tools-bugs, xjia
Target Milestone:	0.16.2
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-04-16 07:12:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alois Mahdal 2014-03-04 09:57:36 UTC

Before first test is launched, system is expected to be in a known stable state.  However, with recently launched test on RHEL7, it turns out that systemd-readahead-collect.service was running at the moment when first test started.

Purpose of this service is to collect data (FS access patterns) during boot which is then to be used by systemd-readahead-replay to pre-load often accessed files.  This, the service  is expected to run only during boot time.

This was found on RHEL7 machines when running /distribution/test_inspector with a new test[1].  2/3 receipts generated (supposedly) false positives due to this service running during first of tests but not during the last one.

  [1]: https://beaker.engineering.redhat.com/jobs/604206

~

From the service manpage[2]:

> systemd-readahead-collect.service is a service that
> collects disk usage patterns at boot time.
> systemd-readahead-replay.service is a service that
> replays this access data collected at the subsequent
> boot.  Since disks tend to be magnitudes slower than
> RAM, this is intended to improve boot speeds by
> pre-loading early at boot all data on disk that
> is known to be read for the complete boot process.
>
> systemd-readahead-done.service is executed a short
> while after boot completed and signals
> systemd-readahead-collect.service to end data
> collection. On this signal, this service will then
> sort the collected disk accesses and store information
> about them in /.readahead.

  [2]: http://www.freedesktop.org/software/systemd/man/systemd-readahead-replay.service.html

Comment 2 Dan Callaghan 2014-03-04 22:54:18 UTC

We should be able to add After=systemd-readahead-done.service to the beah.service unit. I guess that would be reasonable, since beah is not really a normal service (it starts doing a whole bunch of work immediately on boot, but it's not actually part of the boot process).

Comment 3 Amit Saha 2014-03-12 05:02:30 UTC

http://gerrit.beaker-project.org/#/c/2908/(In reply to Dan Callaghan from comment #2)
> We should be able to add After=systemd-readahead-done.service to the
> beah.service unit. I guess that would be reasonable, since beah is not
> really a normal service (it starts doing a whole bunch of work immediately
> on boot, but it's not actually part of the boot process).

http://gerrit.beaker-project.org/#/c/2908/

Comment 4 Dan Callaghan 2014-03-13 01:44:12 UTC

(In reply to Dan Callaghan from comment #2)

On second thought, I don't think this will work as is. It introduces a dependency cycle in the ordering: default.target -> beah-*.service -> systemd-readahead-done.service -> systemd-readahead-done.timer -> default.target.

One possibility would be to just make beah do nothing for 30 seconds after it's started, to give readahead collection time to stop.

Another possibility would be to introduce some new target (default-with-beah.target?) which depends on default.target, make beah-*.service WantedBy that new target, and change the system boot target to be default-with-beah.target instead of default.target. Then we can add After=systemd-readahead-done.service and it won't create a dependency cycle. The downside is we have to fiddle with the system boot target which is not very nice.

Any other ideas?

Comment 5 Dan Callaghan 2014-03-13 07:16:34 UTC

Alternatively, we could just disable readahead collection for Beaker recipes.

We already do this on RHEL6 because of bad interactions between readahead, auditd, and beah.

And readahead collection really doesn't make much sense consdering that a typical recipe is installed once, boots up, runs tasks, and then finishes. The readahead replay would never even be used in that case.

Comment 6 Dan Callaghan 2014-03-23 23:45:55 UTC

We haven't heard any objections about just disabling systemd-readahead so let's go ahead and do that.

We need to make sure there's a ks_meta to opt out of the disabling, and that it's consistent with how the RHEL6 readahead disabling works.

Comment 7 Amit Saha 2014-03-25 03:49:07 UTC

It turns out that it is indeed possible to achieve what we originally wanted here without resorting to an alternative boot target.

So, I  have uploaded a fresh patch for Beah's systemd unit files to that effect: 

Last patch here: http://gerrit.beaker-project.org/#/c/2908/

Lennart Pottering's reply to my query on systemd mailing list explaining how this is possible:

http://lists.freedesktop.org/archives/systemd-devel/2014-March/018162.html

This change should start the beah services after the readahead collection has completed.

Comment 8 Amit Saha 2014-03-28 06:27:29 UTC

Tip for verification:

* Provision a job with RHEL 7 / Fedora 18+
* Login to the system (by running a reservesys/ task, for e.g.) and check that the beah services (beah-srv, beah-beaker-backend and beah-fwd-backend) started *after* the systemd-readahead-collect service exited.

Relevant command:

$ systemctl status beah-srv beah-beaker-backend beah-fwd-backend systemd-readahead-collect

Comment 11 Dan Callaghan 2014-04-16 07:12:41 UTC

Beah 0.7.4 has been released.

Comment 12 Dan Callaghan 2014-09-30 06:23:54 UTC

This has not always worked properly and has caused us various problems since it was implemented (the latest is bug 1147807), so we are going to revert the dependency ordering hack in beah. Beaker will just disable systemd-readahead the same as it disables the readahead service on RHEL6.