Bug 1418823

Summary: Upgrading from f23 to f24 with separate /var partition creates /var directory on root partition
Product: [Fedora] Fedora Reporter: Philip Prindeville <philipp>
Component: dnf-plugin-system-upgradeAssignee: Will Woods <wwoods>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 24CC: philipp, wwoods, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-08 19:37:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Philip Prindeville 2017-02-02 19:49:09 UTC
Description of problem:

Yesterday (2017-2-1) I did an upgrade of our mail server from F23 to F24 via "dnf system-upgrade --releasever=24" and it seemingly went okay.  This morning I was reading the logwatch emails from all of my machines and saw:

   var.mount: Directory /var to mount over is not empty, mounting anyway.: 1 Time(s)
   var.mount: Mount process exited, code=exited status=32: 1 Time(s)

so I edited /etc/fstab to mount the partition as /var.new, did an mkdir of that directory (obviously), and rebooted.

Sure enough, /var (on the root partition) contains files and directories touched yesterday, and there's /var/lib, /var/cache, /var/run, /var/spool, etc.

Given the enormous Charlie Foxtrot that happened with fedup (which, among other things, contributed to its very short shelf-life), I'd have thought this would be one of the things REALLY WELL TESTED before rolling out a replacement.

Apparently I've overestimated severely.


Version-Release number of selected component (if applicable):


How reproducible:

As per above

Steps to Reproduce:
1. Install F23, provision it with a separate /var, update it to most recent
2. Run "dnf system-upgrade --releasever=24"
3. Mount the /var partition elsewhere and look into the /var directory on the root partition
4. Gnash teeth and weep and repeat "fool me once, shame on you, fool me twice shame on me..."


Actual results:

Some or all of the upgrade happened in /var on the root partition.


Expected results:

The upgrade should have mounted and touched the /var partition instead


Additional info:

Comment 1 Philip Prindeville 2017-02-02 19:51:50 UTC
I have a primary server down right now so any comments about recovery would be appreciated.

Comment 2 Zbigniew Jędrzejewski-Szmek 2017-02-02 20:32:41 UTC
For the future: if you get the " Directory ... to mount over is not empty, mounting anyway" message, you can just mount the underlying partition at a second mount point:
  sudo mount -o x-mount.mkdir / --bind /tmp/root-copy
and then you can compare /var with /tmp/root-copy/var and so on.

Before we get to *why* /var wasn't mounted: it shouldn't make that much of a difference. The upgrade process probably created some directories there, but you should already have them in your old var. Did you see anything wrong or other error messages except from the one about non-empty dir?

As to *why*: this shouldn't happen... dnf-system-upgrade.service has a dependency on sysinit.target, which has a dependency on local-fs.target, which has a dependency on all mounts specified in fstab. Do you have any special configuration for /var? How does the entry in fstab look like?

Comment 3 Zbigniew Jędrzejewski-Szmek 2017-02-02 21:01:06 UTC
Ah, I forgot about /var/lib/rpm. You probably should copy that over to the real filesystem, if you still have the old one in the real filesystem.

Comment 4 Philip Prindeville 2017-02-02 21:14:31 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #2)
> Do you have any special configuration for /var? How does the entry in fstab look like?

Turns out that there wasn't a /var/lib/rpm so that was fortunate (see above).

Here's my /etc/fstab:

#
# /etc/fstab
# Created by anaconda on Wed Oct  6 07:29:58 2010
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=305deba5-a9dd-4112-a469-908dfd9d0a59 /                       ext4    defaults        1 1
UUID=7f5f9f37-c68e-4a30-82d5-78c472dac999 /boot                   ext4    defaults        1 2
UUID=8d134425-e74a-430e-bd7a-72c116924ff5 /home                   ext4    defaults        1 2
UUID=83aaa490-5bcb-4c77-8055-3524354b5a80 /var                    ext4    defaults        1 2
UUID=11416992-abc8-450d-8e34-a893eb11e110 swap                    swap    defaults        0 0

tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0


nothing exception (just that I'm not using LVM).

Comment 5 Zbigniew Jędrzejewski-Szmek 2017-02-02 21:54:30 UTC
So, what files do you have in the /var directory on the root filesystem?

Comment 6 Philip Prindeville 2017-02-02 22:10:08 UTC
Well, too late now to look again, but from memory it was:

cache
empty
lib
lock
log
spool
tmp

and lib contained (again from memory)

abrt
colord
geoclue

Comment 7 Zbigniew Jędrzejewski-Szmek 2017-02-02 22:31:57 UTC
Interesting. That's quite strange, because if /var/lib/rpm was updated properly, that means that /var must have been mounted when the upgrade was running.

Is it possible that you had those files in /var-on-the-root-filesystem before, and just didn't notice?

Comment 8 Philip Prindeville 2017-02-02 22:42:18 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #7)
> Is it possible that you had those files in /var-on-the-root-filesystem
> before, and just didn't notice?

Perhaps, but that wouldn't explain why some of them had been modified in the last 24 hours...

Comment 9 Fedora End Of Life 2017-07-26 00:14:07 UTC
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '24'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 10 Fedora End Of Life 2017-08-08 19:37:28 UTC
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.