Bug 1285863

Summary: systemd umounts any /dev/shm-based tmpfs mountpoints automagically nearly immediately after mount
Product: Red Hat Enterprise Linux 7 Reporter: Robert Scheck <redhat-bugzilla>
Component: systemdAssignee: systemd-maint
Status: CLOSED ERRATA QA Contact: Frantisek Sumsal <fsumsal>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.2CC: dzhukous, fkrska, fsumsal, gkulkarn, hannsj_uhl, jjarvis, jkachuck, jruemker, jscotka, lnykryn, mmielke, msekleta, ofamera, owilliam, pdhamdhe, rcyriac, redhat-bugzilla, robert.scheck, rupatel, santony, shjadhav, systemd-maint-list, systemd-maint, tspeetje, vvs
Target Milestone: rcKeywords: Patch
Target Release: 7.3   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 00:47:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1203710, 1285993, 1289485, 1295833, 1313485, 1364088    

Description Robert Scheck 2015-11-26 18:09:01 UTC
Description of problem:
systemd umounts any /dev/shm-based tmpfs mountpoints automagically
nearly immediately after mount since the upgrade to RHEL 7.2: Just
run:

mkdir -p /tmp/rsc2
watch -n 1 "mount /dev/shm /tmp/rsc2 -t tmpfs; mount | grep /tmp/; mount | grep /tmp/; mount | grep /tmp/"

The output will vary from one to three lines because the umount of that
tmpfs happens within 1 second after mounting.

Repeat the same on RHEL 7.1 including all updates: The mount points just 
increase each seconds but never get less like on RHEL 7.2.

Version-Release number of selected component (if applicable):
systemd-219-19.el7.x86_64
systemd-208-20.el7_1.6.x86_64

How reproducible:
Everytime, see above and below.

Actual results:
systemd umounts any /dev/shm-based tmpfs mountpoints automagically
nearly immediatelyafter mount.

Expected results:
Good old behaviour like in RHEL 7.1 so that /dev/shm-based tmpfs mount
points are not automagically umounted nearly immediately after mount.

Additional info:
Even systemd seems to perform the umount, I am not absolutely sure if
systemd is really the cause or if it's e.g. a kernel issue:

Nov 26 18:56:09 tux systemd: Unit tmp-rsc2.mount is bound to inactive unit dev-shm.device. Stopping, too.
Nov 26 18:56:09 tux systemd: Unmounting /tmp/rsc2...
Nov 26 18:56:09 tux systemd: Unmounted /tmp/rsc2.

The issue is bound to /dev/shm vs. tmpfs: Using tmpfs rather /dev/shm,
the issue is not reproducible on RHEL 7.2.

This behaviour was discovered while running a third party installer on
RHEL 7.2 (which succeeded on RHEL 7.1).

Comment 1 Robert Scheck 2015-11-26 18:11:25 UTC
Cross-filed case 01545770 on the Red Hat customer portal.

Comment 2 Lukáš Nykrýn 2015-11-27 06:50:15 UTC
Can you please try this test build?
https://people.redhat.com/lnykryn/systemd/bz1283579.1/

Comment 3 Robert Scheck 2015-11-27 14:56:02 UTC
Yes, that test build works. May we get this into a RHEL update soon, please?

Comment 4 Lukáš Nykrýn 2015-11-27 15:04:23 UTC
Thanks for testing!
It should be in rhel soon (if you want a more specific answer please ask on the customer portal)

Comment 8 Gaurav 2016-01-05 07:23:39 UTC
External Bug ID: Red Hat Customer Portal 01560271

Comment 10 Robert Scheck 2016-01-22 10:25:51 UTC
Why was the target release changed by somebody being not from Red Hat? We
would like to see the fix during the 7.2 series, not waiting for 7.3.

Comment 12 Kees 2016-01-22 10:40:42 UTC
We also run into the same issue. If you can provide a fix for 7.2 it would be great as this issue almost prevents us from using clearcase (and thus developing, because all our sources are in clearcase).

Comment 13 Hanns-Joachim Uhl 2016-01-22 11:00:45 UTC
Hello Red Hat / Joe, John,
... based on the previous comments I would like to request a 7.2.z zstream
for this bugzilla as soon as possible.
Thanks in advance for your support.

Comment 14 Michal Sekletar 2016-01-22 11:30:53 UTC
This should be fixed by next 7.2.z update. 

We received multiple bug reports describing the same issue but because reports were open to different confidential groups we could not really close as duplicates.

Comment 15 Hanns-Joachim Uhl 2016-01-22 11:47:36 UTC
(In reply to Michal Sekletar from comment #14)
> This should be fixed by next 7.2.z update. 
> 
.
... is there already a 7.2.z zstream bugzilla open ...? Please advise ...

Comment 16 Lukáš Nykrýn 2016-01-22 11:57:58 UTC
I am sorry, but bugzilla is an engineering tool and we (developers) can't give you any additional information. If you need any information about releases please use customer portal.

All I can say is that we know that this issues is important and we are addressing appropriately.

Meanwhile, if you want, you can try this test build:
https://people.redhat.com/lnykryn/systemd-219-19.el7_2.3/

Comment 17 Kees 2016-01-22 12:01:26 UTC
I've a valid customer subscription, I will ask using the customer portal (I've a number of other issues with clearcase, so this question will be added to ID 01521664).

Comment 18 Michal Sekletar 2016-01-22 12:07:31 UTC
(In reply to Hanns-Joachim Uhl from comment #15)
> (In reply to Michal Sekletar from comment #14)
> > This should be fixed by next 7.2.z update. 
> > 
> .
> ... is there already a 7.2.z zstream bugzilla open ...? Please advise ...

Yes, there is. But it is not open for IBM confidential group. I can ask for access to that bug on your behalf if you want.

https://bugzilla.redhat.com/show_bug.cgi?id=1285993

Comment 19 Robert Scheck 2016-01-22 12:08:21 UTC
Given ClearCase was mentioned as affected product, we are affected while
using the kernel module build script from Dialogic Diva System Release for
Linux (aka DIVAS4Linux).

Comment 20 Kees 2016-01-22 12:13:18 UTC
(In reply to Michal Sekletar from comment #18)
> (In reply to Hanns-Joachim Uhl from comment #15)
> > (In reply to Michal Sekletar from comment #14)
> > > This should be fixed by next 7.2.z update. 
> > > 
> > .
> > ... is there already a 7.2.z zstream bugzilla open ...? Please advise ...
> 
> Yes, there is. But it is not open for IBM confidential group. I can ask for
> access to that bug on your behalf if you want.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1285993

I'm not familiar with patching. Do you have a description how to use your build (in a RH 7.2 system)?

Comment 21 Hanns-Joachim Uhl 2016-01-22 12:15:38 UTC
(In reply to Michal Sekletar from comment #18)
> (In reply to Hanns-Joachim Uhl from comment #15)
> > (In reply to Michal Sekletar from comment #14)
> > > This should be fixed by next 7.2.z update. 
> > > 
> > .
> > ... is there already a 7.2.z zstream bugzilla open ...? Please advise ...
> 
> Yes, there is. But it is not open for IBM confidential group. I can ask for
> access to that bug on your behalf if you want.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1285993
.
... yes, please request to open the above 7.2.z bugzilla for the 
IBM confidential group ...

Comment 22 Robert Scheck 2016-01-22 12:23:51 UTC
(In reply to Kees from comment #20)
> I'm not familiar with patching. Do you have a description how to use your
> build (in a RH 7.2 system)?

LANG=C rpm -q libgudev1 systemd systemd-devel systemd-journal-gateway \
  systemd-libs systemd-networkd systemd-python systemd-resolved \
  systemd-sysv | grep -v "not installed"

For each installed package (e.g. systemd-219-19.el7.x86_64) install the
new package from the URL that Lukáš mentioned, so:

  rpm -Uvh https://people.redhat.com/…/systemd-219-19.el7_2.3.x86_64.rpm \
    https://people.redhat.com/…/systemd-libs-219-19.el7_2.3.x86_64.rpm \
    https://people.redhat.com/…/systemd-python-219-19.el7_2.3.x86_64.rpm \
    …

Afterwards you likely should reboot. Please keep in mind that above packages
are for testing purposes, they IMHO should not be used in production.

Comment 23 Kees 2016-01-22 12:27:42 UTC
(In reply to Robert Scheck from comment #22)
> (In reply to Kees from comment #20)
> > I'm not familiar with patching. Do you have a description how to use your
> > build (in a RH 7.2 system)?
> 
> LANG=C rpm -q libgudev1 systemd systemd-devel systemd-journal-gateway \
>   systemd-libs systemd-networkd systemd-python systemd-resolved \
>   systemd-sysv | grep -v "not installed"
> 
> For each installed package (e.g. systemd-219-19.el7.x86_64) install the
> new package from the URL that Lukáš mentioned, so:
> 
>   rpm -Uvh https://people.redhat.com/…/systemd-219-19.el7_2.3.x86_64.rpm \
>     https://people.redhat.com/…/systemd-libs-219-19.el7_2.3.x86_64.rpm \
>     https://people.redhat.com/…/systemd-python-219-19.el7_2.3.x86_64.rpm \
>     …
> 
> Afterwards you likely should reboot. Please keep in mind that above packages
> are for testing purposes, they IMHO should not be used in production.

Thanks Robert. When a final fix is released, will it then conflict with this update? Or should that work fine?

And I assume that there are no other prerequisites other than running 64-bit RH7.2?

Comment 24 Michal Sekletar 2016-01-22 12:50:18 UTC
(In reply to Hanns-Joachim Uhl from comment #21)

> ... yes, please request to open the above 7.2.z bugzilla for the 
> IBM confidential group ...

I've now requested access for IBM to both, original bugzilla and z-stream clone.

Comment 25 Robert Scheck 2016-01-22 13:59:11 UTC
(In reply to Kees from comment #23)
> Thanks Robert. When a final fix is released, will it then conflict with this
> update? Or should that work fine?

This should not conflict with an update if Red Hat changes the version number
of the RPM packages for the official update. It might cause issues if Red Hat
does not change the version number but performs changes inside of the packages.
As I am not a Red Hat employee, I can not comment about what will happen here.

However, there is "yum reinstall systemd systemd-libs systemd-python" which
could be run in such a case (just watch the errata mails about the version
once the next official systemd update happens). If the version number there
is newer, the packages will be simply replaced and everything is fine.

> And I assume that there are no other prerequisites other than running 64-bit
> RH7.2?

Right, but my example should be applicable to any supported RHEL platform.

Comment 26 Joseph Kachuck 2016-01-22 17:52:33 UTC
Hello IBM,
You should now have access to BZ:
https://bugzilla.redhat.com/show_bug.cgi?id=1285993

Thank You
Joe Kachuck

Comment 28 Kees 2016-01-27 10:59:24 UTC
I tried to test the provided patch on a virtual system.

The website https://people.redhat.com/lnykryn/systemd-219-19.el7_2.3/ is in an almost unreadable format and hides the platform details (path names are truncated). I obtained the files by running:

curl https://people.redhat.com/lnykryn/systemd-219-19.el7_2.3/ | grep x86_64 | sed 's/.*href=//' | sed 's/>.*$//' | sed 's/\"//g' | while read i; do
wget https://people.redhat.com/lnykryn/systemd-219-19.el7_2.3/$i;done

This resulted in 10 downloaded RPMs.

and tried to apply them by:
rpm -Uvh *.rpm

but got an error when installing these RPMs:

# rpm -Uvh *.rpm       
error: Failed dependencies:
        pkgconfig(glib-2.0) is needed by libgudev1-devel-219-19.el7_2.3.x86_64
        pkgconfig(gobject-2.0) is needed by libgudev1-devel-219-19.el7_2.3.x86_64


However, pkconfig is installed and no newer versions exists:

# rpm -qa | grep pkgconf
pkgconfig-0.27.1-4.el7.x86_64
pkgconfig-0.27.1-4.el7.i686

How to proceeed now?

Comment 29 Lukáš Nykrýn 2016-01-27 12:50:46 UTC
These packages are really just for testing and they are not supported by any means. If you are uncertain about something, you should wait for regular release.

But to answer your question, just update the packages you already have. I doubt that you have libgudev1-devel installed.

Comment 30 Kees 2016-01-27 13:20:10 UTC
(In reply to Lukáš Nykrýn from comment #29)
> These packages are really just for testing and they are not supported by any
> means. If you are uncertain about something, you should wait for regular
> release.
> 
> But to answer your question, just update the packages you already have. I
> doubt that you have libgudev1-devel installed.

I've libgudev installed:
# rpm -qa | grep libgudev
libgudev1-219-19.el7.x86_64

The libgudev1-devel and libgudev1 RPMs are on the previously mentioned URL ( libgudev1-devel-219-19.el7_2.3.x86_64.rpm and libgudev1-219-19.el7_2.3.x86_64.rpm).

The problem is that redhat customer support (by using my subscription) can not tell a delivery date, but my servers are still choking a CPU to 100%. So I don't know how long I have to wait for a regular release (and even don't know what is a 'normal' time frame, e.g. week, month or even longer).

Comment 31 Robert Scheck 2016-01-27 13:33:02 UTC
My example above tells to check which packages are currently installed and
only to update the installed ones (likely 3 packages on an average system),
run the "rpm -q .." above and update only these few packages, that should
work flawlessly.

Comment 32 Kees 2016-01-27 14:20:01 UTC
(In reply to Robert Scheck from comment #31)
> My example above tells to check which packages are currently installed and
> only to update the installed ones (likely 3 packages on an average system),
> run the "rpm -q .." above and update only these few packages, that should
> work flawlessly.

Not sure whether I understand you correctly. The URL provides 10 RPMs (for a number of platforms). 
1. Do you mean that I don't need them all?
2. To which few (3?) packages are you referring?

This is what is currently installed:
# rpm -qa  |egrep "libgudev1|systemd"|sort
libgudev1-219-19.el7.x86_64
systemd-219-19.el7.x86_64
systemd-libs-219-19.el7.x86_64
systemd-python-219-19.el7.x86_64
systemd-sysv-219-19.el7.x86_64

and these RPMs are downloaded (and about to be installed):
libgudev1-219-19.el7_2.3.x86_64.rpm
libgudev1-devel-219-19.el7_2.3.x86_64.rpm
systemd-219-19.el7_2.3.x86_64.rpm
systemd-devel-219-19.el7_2.3.x86_64.rpm
systemd-journal-gateway-219-19.el7_2.3.x86_64.rpm
systemd-libs-219-19.el7_2.3.x86_64.rpm
systemd-networkd-219-19.el7_2.3.x86_64.rpm
systemd-python-219-19.el7_2.3.x86_64.rpm
systemd-resolved-219-19.el7_2.3.x86_64.rpm
systemd-sysv-219-19.el7_2.3.x86_64.rpm

Sorry for my lack of knowledge. I'm developer, and clearcase admin, but not system admin.

Comment 33 Robert Scheck 2016-01-27 14:23:32 UTC
Only install/update the 5 packages that your grep returned, then it should fit.

Comment 34 Kees 2016-01-27 14:34:21 UTC
(In reply to Robert Scheck from comment #33)
> Only install/update the 5 packages that your grep returned, then it should
> fit.

That did it. Thanks for giving this clue.
Even without reboot, the CPU now got quiet. That's good news.
The rpm update did not complain about reboot. Do I then need to reboot (just to be sure)?

Comment 35 Robert Scheck 2016-01-27 14:52:39 UTC
Personally, I would do the reboot, because systemd is quite elementary. I
do not know if there is an official recommendation, this is really just what
I would do as a sysadmin (I treat systemd similar like kernel and glibc).

Comment 36 Lukáš Nykrýn 2016-01-27 15:41:31 UTC
Official recommendation is to reboot after update of systemd.

Comment 37 Lukáš Nykrýn 2016-01-27 15:43:54 UTC
Well to be precise systmed will reexecute itself after the update so you will run the new binary without reboot, but better safe than sorry.

Comment 42 Frantisek Sumsal 2016-08-16 13:30:23 UTC
Verified on RHEL 7.3 Beta with systemd-219-26.el7.

Old package:
:: [   PASS   ] :: Command 'mkdir -p '/tmp/tmp.dhbLqq7IW1/testpoint'' (Expected 0, got 0)
:: [   PASS   ] :: Command 'mount -t tmpfs /dev/shm '/tmp/tmp.dhbLqq7IW1/testpoint'' (Expected 0, got 0)
:: [   PASS   ] :: Command 'sleep 5' (Expected 0, got 0)
:: [   FAIL   ] :: Command 'mount -l | grep '/tmp/tmp.dhbLqq7IW1/testpoint'' (Expected 0, got 1)
:: [   FAIL   ] :: Command '[[ 0 -eq 1 ]]' (Expected 0, got 1)
:: [   LOG    ] :: Duration: 5s
:: [   LOG    ] :: Assertions: 3 good, 2 bad
:: [   FAIL   ] :: RESULT: Test

New package:
:: [   PASS   ] :: Command 'mkdir -p '/tmp/tmp.h28vzkoFAo/testpoint'' (Expected 0, got 0)
:: [   PASS   ] :: Command 'mount -t tmpfs /dev/shm '/tmp/tmp.h28vzkoFAo/testpoint'' (Expected 0, got 0)
:: [   PASS   ] :: Command 'sleep 5' (Expected 0, got 0)
:: [   PASS   ] :: Command 'mount -l | grep '/tmp/tmp.h28vzkoFAo/testpoint'' (Expected 0, got 0)
:: [   PASS   ] :: Command '[[ 1 -eq 1 ]]' (Expected 0, got 0)
:: [   LOG    ] :: Duration: 6s
:: [   LOG    ] :: Assertions: 5 good, 0 bad
:: [   PASS   ] :: RESULT: Test

Comment 44 errata-xmlrpc 2016-11-04 00:47:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2216.html