Bug 796793 - Cannot usrmove: Something failed. Move back to the original state
Cannot usrmove: Something failed. Move back to the original state
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: dracut-maint
Fedora Extras Quality Assurance
:
Depends On:
Blocks: F17Blocker/F17FinalBlocker
  Show dependency treegraph
 
Reported: 2012-02-23 11:49 EST by Mamoru TASAKA
Modified: 2012-05-29 11:25 EDT (History)
4 users (show)

See Also:
Fixed In Version: dracut-017-40.git20120308.fc17
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-03-13 14:34:30 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
dmesg (90.23 KB, text/plain)
2012-02-23 11:49 EST, Mamoru TASAKA
no flags Details
dmesg with comment 5 (126.52 KB, text/plain)
2012-02-24 09:50 EST, Mamoru TASAKA
no flags Details
/etc/fstab (774 bytes, text/plain)
2012-02-24 09:56 EST, Mamoru TASAKA
no flags Details
/etc/dracut.conf (451 bytes, text/plain)
2012-02-24 09:58 EST, Mamoru TASAKA
no flags Details
upgrade failure (48.85 KB, application/octet-stream)
2012-05-29 09:05 EDT, Kapetanakis Giannis
no flags Details

  None (edit)
Description Mamoru TASAKA 2012-02-23 11:49:47 EST
Created attachment 565336 [details]
dmesg

Description of problem:
This night I tried usrmove but it does not seem to work.


Version-Release number of selected component (if applicable):
dracut-016-9.git20120217.fc17.noarch


How reproducible:
100%

Steps to Reproduce:
1. I think I followed https://fedoraproject.org/wiki/Upgrading_Fedora_using_yum#Fedora_16_-.3E_Fedora_17
  
Actual results:
[   11.048298] dracut: mv: cannot stat `/sysroot/usr/bin': No such file or directory
[   11.052074] dracut: Something failed. Move back to the original state
[   11.236577] dracut: Switching root


Expected results:
usrmove should succeed

Additional info:

$ LANG=C df -k
Filesystem                      1K-blocks     Used Available Use% Mounted on
rootfs                            1033516   648708    332380  67% /
devtmpfs                           370772        0    370772   0% /dev
tmpfs                              375108      228    374880   1% /dev/shm
/dev/mapper/VolGroup00-LogVol00   1033516   648708    332380  67% /
tmpfs                              375108      536    374572   1% /run
tmpfs                              375108        0    375108   0% /sys/fs/cgroup
tmpfs                              375108        0    375108   0% /media
/dev/sda8                          246959     4538    229571   2% /tmp
/dev/sda7                          495759   161223    308834  35% /var
tmpfs                              375108      536    374572   1% /var/run
tmpfs                              375108      536    374572   1% /var/lock
/dev/sda3                           76263    53101     19146  74% /boot
/dev/mapper/VolGroup00-LogVol02  24341772 19521772   3581372  85% /home
/dev/mapper/VolGroup00-LogVol01   9559104  7401676   1670824  82% /usr
none                               375108      228    374880   1% /dev/shm
Comment 1 Harald Hoyer 2012-02-23 12:12:37 EST
can you remove any other "rd*" parameter on your kernel command line, like rd.lvm... or rd.md... or rd.dm ? and retry ?
Comment 2 Mamoru TASAKA 2012-02-23 16:33:16 EST
I removed rd_plytheme=charge however still no luck.
Comment 3 Mamoru TASAKA 2012-02-23 21:02:43 EST
I just looked at http://dracut.git.sourceforge.net/git/gitweb.cgi?p=dracut/dracut;a=commitdiff;h=ae8b82e395c9530a66288f7a9e939242137d3f56 , and
my dmesg, then:

* dmesg says:
[    4.236182] dracut: inactive '/dev/VolGroup00/LogVol00' [1.00 GiB] inherit
[    4.245747] dracut: inactive '/dev/VolGroup00/LogVol02' [23.62 GiB] inherit
[    4.253656] dracut: inactive '/dev/VolGroup00/LogVol01' [9.28 GiB] inherit
[    4.468274] EXT4-fs (dm-0): mounting ext3 file system using the ext4 subsystem
[    4.498348] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[    4.619314] dracut: Checking ext3: /dev/VolGroup00/LogVol00
[    4.622943] dracut: issuing e2fsck -a  /dev/VolGroup00/LogVol00
[    4.775657] dracut: /dev/VolGroup00/LogVol00: clean, 17297/131072 files, 165939/262144 blocks
[    4.784133] dracut: Remounting /dev/VolGroup00/LogVol00 with -o rw
[    4.800158] EXT4-fs (dm-0): mounting ext3 file system using the ext4 subsystem
[    4.826932] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[    4.879457] dracut: Mounted root filesystem /dev/mapper/VolGroup00-LogVol00
[    5.058532] dracut: Make a copy of `/sysroot/usr/bin'.

So it seems that when usrmove begins, /usr is not yet mounted.

Then
+# merge / and /usr in new dir in /usr
+for dir in bin sbin lib lib64; do
+    rm -rf "$ROOT/usr/${dir}.usrmove-new"
+    [[ -L "$ROOT/$dir" ]] && continue
+    [[ -d "$ROOT/$dir" ]] || continue
+    echo "Make a copy of \`$ROOT/usr/$dir'."
+    [[ -d "$ROOT/usr/$dir" ]] \ <==== HERE
+        && cp -ax $CP_HARDLINK "$ROOT/usr/$dir" "$ROOT/usr/${dir}.usrmove-new"

So I guess at "<==== HERE" $ROOT/usr/$dir does not exist and "cp"ing files is skipped.

Then
+    mv "$ROOT/usr/$dir" "$ROOT/usr/${dir}.usrmove-old"
fails, because /usr is not mounted (if my guess is correct).
Comment 4 Mamoru TASAKA 2012-02-23 21:05:33 EST
By the way, can I check if /usr has enough space left for usrmove before usrmove begins?
Comment 5 Harald Hoyer 2012-02-24 08:28:17 EST
(In reply to comment #2)
> I removed rd_plytheme=charge however still no luck.

Please add "rd.lvm.vg=VolGroup00" to the kernel command line.

Do you have "hostonly" turned on for the initramfs?
Comment 6 Harald Hoyer 2012-02-24 08:32:59 EST
How does you /etc/fstab look like?

you can also add "rd.break rd.shell" for a dracut shell and inspect:

# ls -l /dev/mapper
# cat /etc/fstab

you /usr device and entry should be there
Comment 7 Harald Hoyer 2012-02-24 08:34:00 EST
The dracut usrmount module should be in the initramfs. Can you check?
$ lsinitrd |fgrep mount-usr
-rwxr-xr-x   1 root     root         1369 Feb 22 17:09 usr/lib/dracut/hooks/pre-pivot/50mount-usr.sh
Comment 8 Harald Hoyer 2012-02-24 08:46:37 EST
May I ask, why this is flagged as a blocker? I thought yum updates are not officially supported?

We can also state, that if you have /usr on a seperate partition it is only possible to upgrade with anaconda.
Comment 9 Mamoru TASAKA 2012-02-24 09:50:26 EST
Created attachment 565608 [details]
dmesg with comment 5

(In reply to comment #5) 
> Please add "rd.lvm.vg=VolGroup00" to the kernel command line.
With this I saw some progress. However now I see another issue (as I worried on comment 4), even though from df -k it seems that /usr has enough space.
 
> Do you have "hostonly" turned on for the initramfs?
If I can interpret this as if I have enabled hostonly="yes" on /etc/dracut.conf, yes.
Comment 10 Mamoru TASAKA 2012-02-24 09:56:46 EST
Created attachment 565609 [details]
/etc/fstab

(In reply to comment #6)
> How does you /etc/fstab look like?
attached.

(In reply to comment #7)
> The dracut usrmount module should be in the initramfs. Can you check?
> $ lsinitrd |fgrep mount-usr
> -rwxr-xr-x   1 root     root         1369 Feb 22 17:09
> usr/lib/dracut/hooks/pre-pivot/50mount-usr.sh

[tasaka1@localhost ~]$ lsinitrd |fgrep mount-usr
-rwxr-xr-x   1 root     root         1250 Feb 16 02:05 lib/dracut/hooks/pre-pivot/50mount-usr.sh

(In reply to comment #8)
> May I ask, why this is flagged as a blocker? I thought yum updates are not
> officially supported?
Whenever the reporter thinks he/she met some blocker issue, he/she can _propose_ the issue as blocker. Judgment is done by some Fedora committee.
Comment 11 Mamoru TASAKA 2012-02-24 09:58:13 EST
Created attachment 565611 [details]
/etc/dracut.conf

/etc/dracut.conf anyway
Comment 12 Harald Hoyer 2012-02-24 10:07:55 EST
(In reply to comment #9)
> Created attachment 565608 [details]
> dmesg with comment 5
> 
> (In reply to comment #5) 
> > Please add "rd.lvm.vg=VolGroup00" to the kernel command line.
> With this I saw some progress. However now I see another issue (as I worried on
> comment 4), even though from df -k it seems that /usr has enough space.

So, dracut cannot do anything here. As you can parse the error messages and the system boots afterwards, only you can assign more space to /usr. I am sorry.

For the other points, I extended the wiki page in regard to:
- turn off hostonly
- make sure dracut parameters for /usr are set
- should use anaconda anyway in this case
Comment 13 Harald Hoyer 2012-02-24 10:12:32 EST
(In reply to comment #12)
> (In reply to comment #9)
> > Created attachment 565608 [details]
> > dmesg with comment 5
> > 
> > (In reply to comment #5) 
> > > Please add "rd.lvm.vg=VolGroup00" to the kernel command line.
> > With this I saw some progress. However now I see another issue (as I worried on
> > comment 4), even though from df -k it seems that /usr has enough space.
> 
> So, dracut cannot do anything here. As you can parse the error messages and the
> system boots afterwards, only you can assign more space to /usr. I am sorry.
> 

Oh.. Indeed I found a bug. Can you try this:

--- /usr/lib/dracut/modules.d/30convertfs/convertfs.sh
+++ /usr/lib/dracut/modules.d/30convertfs/convertfs.sh
@@ -106,7 +106,7 @@ for dir in bin sbin lib lib64; do
     [[ -d "$ROOT/$dir" ]] || continue
     echo "Make a copy of \`$ROOT/usr/$dir'."
     [[ -d "$ROOT/usr/$dir" ]] \
-        && cp -ax $CP_HARDLINK "$ROOT/usr/$dir" "$ROOT/usr/${dir}.usrmove-new"
+        && cp -ax -l "$ROOT/usr/$dir" "$ROOT/usr/${dir}.usrmove-new"
     echo "Merge the copy with \`$ROOT/$dir'."
     [[ -d "$ROOT/usr/${dir}.usrmove-new" ]] \
         || mkdir -p "$ROOT/usr/${dir}.usrmove-new"
Comment 14 Harald Hoyer 2012-02-24 10:16:28 EST
of course, you will have to recreate the initramfs
Comment 15 Mamoru TASAKA 2012-02-24 10:49:08 EST
Will try. Wait a moment...
Comment 16 Mamoru TASAKA 2012-02-24 11:18:22 EST
(In reply to comment #13)

> Oh.. Indeed I found a bug. Can you try this:
> 
> --- /usr/lib/dracut/modules.d/30convertfs/convertfs.sh
> +++ /usr/lib/dracut/modules.d/30convertfs/convertfs.sh
> @@ -106,7 +106,7 @@ for dir in bin sbin lib lib64; do
>      [[ -d "$ROOT/$dir" ]] || continue
>      echo "Make a copy of \`$ROOT/usr/$dir'."
>      [[ -d "$ROOT/usr/$dir" ]] \
> -        && cp -ax $CP_HARDLINK "$ROOT/usr/$dir" "$ROOT/usr/${dir}.usrmove-new"
> +        && cp -ax -l "$ROOT/usr/$dir" "$ROOT/usr/${dir}.usrmove-new"
>      echo "Merge the copy with \`$ROOT/$dir'."
>      [[ -d "$ROOT/usr/${dir}.usrmove-new" ]] \
>          || mkdir -p "$ROOT/usr/${dir}.usrmove-new"

With this, now I have the following!!
[tasaka1@localhost ~]$ LANG=C ls -ald /lib /bin /sbin
lrwxrwxrwx 1 root root 7 Feb 25  2012 /bin -> usr/bin
lrwxrwxrwx 1 root root 7 Feb 25  2012 /lib -> usr/lib
lrwxrwxrwx 1 root root 8 Feb 25  2012 /sbin -> usr/sbin
Comment 17 Fedora Update System 2012-02-29 07:42:46 EST
dracut-017-17.git20120229.2.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-017-17.git20120229.2.fc17
Comment 18 Fedora Update System 2012-02-29 13:53:26 EST
Package dracut-017-17.git20120229.2.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dracut-017-17.git20120229.2.fc17'
as soon as you are able to, then reboot.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-2699/dracut-017-17.git20120229.2.fc17
then log in and leave karma (feedback).
Comment 19 Fedora Update System 2012-03-01 12:09:24 EST
dracut-017-19.git20120301.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-017-19.git20120301.fc17
Comment 20 Fedora Update System 2012-03-02 08:57:32 EST
dracut-017-22.git20120302.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-017-22.git20120302.fc17
Comment 21 Fedora Update System 2012-03-08 08:04:25 EST
dracut-017-39.git20120308.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-017-39.git20120308.fc17
Comment 22 Fedora Update System 2012-03-08 08:57:02 EST
dracut-017-40.git20120308.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-017-40.git20120308.fc17
Comment 23 Fedora Update System 2012-03-08 08:57:14 EST
dracut-017-40.git20120308.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/dracut-017-40.git20120308.fc17
Comment 24 Fedora Update System 2012-03-13 14:34:30 EDT
dracut-017-40.git20120308.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 25 Kapetanakis Giannis 2012-05-29 09:03:57 EDT
(In reply to comment #8)
> May I ask, why this is flagged as a blocker? I thought yum updates are not
> officially supported?
> 
> We can also state, that if you have /usr on a seperate partition it is only
> possible to upgrade with anaconda.

Well, it cannot upgrade from anaconda either...

I'm talking about clean new install of Fedora 16 with seperate /usr /var
filesystems on LVM -> upgrading to 17
Comment 26 Kapetanakis Giannis 2012-05-29 09:05:08 EDT
Created attachment 587400 [details]
upgrade failure

upgrade failure (16->17) from DVD
with seperate /usr /var filesystems
Comment 27 Harald Hoyer 2012-05-29 11:25:40 EDT
(In reply to comment #26)
> Created attachment 587400 [details]
> upgrade failure
> 
> upgrade failure (16->17) from DVD
> with seperate /usr /var filesystems

seems to be a different anaconda error (please open a separate bug):

anaconda 17.29 exception report
Traceback (most recent call first):
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/devices.py", line 702, in _preSetup
    raise DeviceError("device has not been created", self.name)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/devices.py", line 718, in setup
    if not self._preSetup(orig=orig):
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/__init__.py", line 2012, in _parseOneLine
    device.setup()
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/__init__.py", line 2112, in parseFSTab
    device = self._parseOneLine((devspec, mountpoint, fstype, options, dump, passno))
  File "/usr/lib64/python2.7/site-packages/pyanaconda/storage/__init__.py", line 1676, in mountExistingSystem
    fsset.parseFSTab(anaconda=anaconda)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/upgrade.py", line 178, in upgradeMountFilesystems
    mountExistingSystem(anaconda, anaconda.upgradeRoot[0], allowDirty = 0)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/dispatch.py", line 383, in dispatch
    self.dir = self.steps[self.step].target(self.anaconda)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/dispatch.py", line 247, in go_forward
    self.dispatch()
  File "/usr/lib64/python2.7/site-packages/pyanaconda/gui.py", line 1201, in nextClicked
    self.anaconda.dispatch.go_forward()
DeviceError: ('device has not been created', '/dev/disk/by-uuid/c366234f-0aee-4806-9285-020a90609ae6')

Local variables in innermost frame:
self: non-existent 0MB storage /dev/disk/by-uuid/c366234f-0aee-4806-9285-020a90609ae6 (16) with non-existent ext4 filesystem
orig: False

Note You need to log in before you can comment on or make changes to this bug.