Description of problem:
Yesterday (2017-2-1) I did an upgrade of our mail server from F23 to F24 via "dnf system-upgrade --releasever=24" and it seemingly went okay. This morning I was reading the logwatch emails from all of my machines and saw:
var.mount: Directory /var to mount over is not empty, mounting anyway.: 1 Time(s)
var.mount: Mount process exited, code=exited status=32: 1 Time(s)
so I edited /etc/fstab to mount the partition as /var.new, did an mkdir of that directory (obviously), and rebooted.
Sure enough, /var (on the root partition) contains files and directories touched yesterday, and there's /var/lib, /var/cache, /var/run, /var/spool, etc.
Given the enormous Charlie Foxtrot that happened with fedup (which, among other things, contributed to its very short shelf-life), I'd have thought this would be one of the things REALLY WELL TESTED before rolling out a replacement.
Apparently I've overestimated severely.
Version-Release number of selected component (if applicable):
As per above
Steps to Reproduce:
1. Install F23, provision it with a separate /var, update it to most recent
2. Run "dnf system-upgrade --releasever=24"
3. Mount the /var partition elsewhere and look into the /var directory on the root partition
4. Gnash teeth and weep and repeat "fool me once, shame on you, fool me twice shame on me..."
Some or all of the upgrade happened in /var on the root partition.
The upgrade should have mounted and touched the /var partition instead
I have a primary server down right now so any comments about recovery would be appreciated.
For the future: if you get the " Directory ... to mount over is not empty, mounting anyway" message, you can just mount the underlying partition at a second mount point:
sudo mount -o x-mount.mkdir / --bind /tmp/root-copy
and then you can compare /var with /tmp/root-copy/var and so on.
Before we get to *why* /var wasn't mounted: it shouldn't make that much of a difference. The upgrade process probably created some directories there, but you should already have them in your old var. Did you see anything wrong or other error messages except from the one about non-empty dir?
As to *why*: this shouldn't happen... dnf-system-upgrade.service has a dependency on sysinit.target, which has a dependency on local-fs.target, which has a dependency on all mounts specified in fstab. Do you have any special configuration for /var? How does the entry in fstab look like?
Ah, I forgot about /var/lib/rpm. You probably should copy that over to the real filesystem, if you still have the old one in the real filesystem.
(In reply to Zbigniew Jędrzejewski-Szmek from comment #2)
> Do you have any special configuration for /var? How does the entry in fstab look like?
Turns out that there wasn't a /var/lib/rpm so that was fortunate (see above).
Here's my /etc/fstab:
# Created by anaconda on Wed Oct 6 07:29:58 2010
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
UUID=305deba5-a9dd-4112-a469-908dfd9d0a59 / ext4 defaults 1 1
UUID=7f5f9f37-c68e-4a30-82d5-78c472dac999 /boot ext4 defaults 1 2
UUID=8d134425-e74a-430e-bd7a-72c116924ff5 /home ext4 defaults 1 2
UUID=83aaa490-5bcb-4c77-8055-3524354b5a80 /var ext4 defaults 1 2
UUID=11416992-abc8-450d-8e34-a893eb11e110 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
nothing exception (just that I'm not using LVM).
So, what files do you have in the /var directory on the root filesystem?
Well, too late now to look again, but from memory it was:
and lib contained (again from memory)
Interesting. That's quite strange, because if /var/lib/rpm was updated properly, that means that /var must have been mounted when the upgrade was running.
Is it possible that you had those files in /var-on-the-root-filesystem before, and just didn't notice?
(In reply to Zbigniew Jędrzejewski-Szmek from comment #7)
> Is it possible that you had those files in /var-on-the-root-filesystem
> before, and just didn't notice?
Perhaps, but that wouldn't explain why some of them had been modified in the last 24 hours...
This message is a reminder that Fedora 24 is nearing its end of life.
Approximately 2 (two) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 24. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora 'version'
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 24 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
Thank you for reporting this bug and we are sorry it could not be fixed.