Bug 748119

Summary: upgrades not allowed if /var is on a separate volume
Product: [Fedora] Fedora Reporter: Jaroslav Pulchart <jaroslav.pulchart>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: anaconda-maint-list, awilliam, bloch, bperkins, collura, crow, duviolg, edwinh, fedoration, fredericg_99, grgoffe, imc, info, john, jonathan, jpcartal, kagesenshi.87, kellya, k.georgiou, kynalvarus, libregeek, Magnumgr, makowski.fedora, maurochss, mdurkin-fed, michael, michal, Panos.Kavalagios, philipp, prl-bugzilla.redhat.com, quiffman, redhat_bugzilla, Reinhard.Scheck, Shurik, stanmc1, stephendbell, tflink, udovdh, v1dingo, vanmeeuwen+fedora, vonbrand, yann, zdenek
Target Milestone: ---Keywords: CommonBugs
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard: AcceptedNTH https://fedoraproject.org/wiki/Common_F16_bugs#separate-var-anaconda-upgrade-fail
Fixed In Version: anaconda-17.2-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-23 20:06:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 713566, 789114    
Attachments:
Description Flags
ALT + F1 screen
none
ALT + F3 screen
none
ALT + F4 screen
none
fdisk -l
none
fstab
none
grub.conf with preupgrade
none
preupgrade ks.cfg
none
/tmp/X.log
none
/tmp/storage.log
none
/tmp/program.log
none
/tmp/anaconda.log
none
/tmp/syslog
none
grub.conf with preupgrade
none
anaconda.log
none
program.log
none
/tmp/storage.log
none
/tmp/syslog
none
/tmp/X.log
none
fdisk -l
none
flat file containing anaconda.log
none
Another LVM configuration that fails as described here, even with 748119.img none

Description Jaroslav Pulchart 2011-10-22 08:02:20 UTC
Description of problem:
I tested preupgrade from F15 to F16 on my laptop with fix #737731. Upgrade started after reboot but soon Anaconda exited with this error message "The root for the previously installed system system was not found".

I see in logs that anaconda mounted root and all other partitions but after that he exited with error message.

Version-Release number of selected component (if applicable):
F15

How reproducible:
It is reproducible on my Laptop. I tested preupgrade also my desktop and the upgrade was OK.

Steps to Reproduce:
1. run preupgrade to F16 
2. reboot

  
Actual results:
"The root for the previously installed system system was not found"

Expected results:
Upgrade was successful.

Additional info:
I will attach logs and screens later this days. Do you know how I could increase anaconda log level from INFO/ERR to debug level?

Comment 1 Jaroslav Pulchart 2011-10-22 08:48:01 UTC
Created attachment 529585 [details]
ALT + F1 screen

Comment 2 Jaroslav Pulchart 2011-10-22 08:52:02 UTC
Created attachment 529586 [details]
ALT + F3 screen

Comment 3 Jaroslav Pulchart 2011-10-22 08:55:23 UTC
Created attachment 529589 [details]
ALT + F4 screen

Comment 4 Jaroslav Pulchart 2011-10-22 09:05:01 UTC
Created attachment 529590 [details]
fdisk -l

Comment 5 Jaroslav Pulchart 2011-10-22 09:08:44 UTC
Created attachment 529592 [details]
fstab

Comment 6 Jaroslav Pulchart 2011-10-22 09:12:53 UTC
Created attachment 529593 [details]
grub.conf with preupgrade

Comment 7 Jaroslav Pulchart 2011-10-22 09:13:32 UTC
Created attachment 529594 [details]
preupgrade ks.cfg

Comment 8 Jaroslav Pulchart 2011-10-22 09:23:43 UTC
Mhhh, I found the missing logs in /tmp location instead of /var/log.

Comment 9 Jaroslav Pulchart 2011-10-22 09:26:43 UTC
Created attachment 529596 [details]
/tmp/X.log

Comment 10 Jaroslav Pulchart 2011-10-22 09:29:17 UTC
Created attachment 529597 [details]
/tmp/storage.log

Comment 11 Jaroslav Pulchart 2011-10-22 09:31:07 UTC
Created attachment 529598 [details]
/tmp/program.log

Comment 12 Jaroslav Pulchart 2011-10-22 09:31:55 UTC
Created attachment 529599 [details]
/tmp/anaconda.log

Comment 13 Jaroslav Pulchart 2011-10-22 09:32:54 UTC
Created attachment 529600 [details]
/tmp/syslog

Comment 14 Jaroslav Pulchart 2011-10-23 08:16:15 UTC
I checked the code of anaconda and new logs in "getReleaseString" function from v17 could be helpful in v16 like these:

diff anaconda-16.22-1 vs. HEAD of "anaconda/pyanaconda/storage/__init__.py" file:
...
             try:
-                (arch, product, version) = getReleaseString(anaconda.rootPath)
+                (arch, product, version) = getReleaseString()
             except ValueError:
                 # This likely isn't our product, so don't even count it as
                 # notUpgradable.
+                log.info("findExistingRootDevices: no release string.")
                 continue
             finally:
                 device.teardown(recursive=True)
@@ -1547,6 +1550,7 @@
             if arch is None:
                 # we failed to determine the arch (for instance when there is a
                 # corrupted rpm database on the target system)
+                log.info("findExistingRootDevices: no arch.")
                 continue
...

Maybe the problem is in inaccessible RPM db in separated partition than root partition, but I'm without log and cannot check it. I will try to move /var data into root partition later this day.

Comment 15 Alexander Dyadyun 2011-10-23 08:18:24 UTC
    Same problem...

    After preupgrade from F15 => F16 (1.1.10-1.fc15) I got "The root for the
    previously installed system system was not found"

Comment 16 Alexander Dyadyun 2011-10-23 08:19:54 UTC
Created attachment 529657 [details]
grub.conf with preupgrade

grub.conf with preupgrade

Comment 17 Alexander Dyadyun 2011-10-23 08:20:44 UTC
Created attachment 529658 [details]
anaconda.log

Comment 18 Alexander Dyadyun 2011-10-23 08:21:40 UTC
Created attachment 529659 [details]
program.log

Comment 19 Alexander Dyadyun 2011-10-23 08:22:17 UTC
Created attachment 529660 [details]
/tmp/storage.log

Comment 20 Alexander Dyadyun 2011-10-23 08:23:08 UTC
Created attachment 529661 [details]
/tmp/syslog

Comment 21 Alexander Dyadyun 2011-10-23 08:23:45 UTC
Created attachment 529662 [details]
/tmp/X.log

Comment 22 Alexander Dyadyun 2011-10-23 08:27:31 UTC
Created attachment 529663 [details]
fdisk -l

/sda1 - Win7
/sda2 - Win7
/sda3 - /boot (ext4)

/sda5 - LVM with Fedora15
/sda6 - Win7

/sdb1 - Win7
/sdc1 - flash

Comment 23 Jaroslav Pulchart 2011-10-23 15:02:21 UTC
As I expected, the architecture search is provided by RPM db:

findExistingRootDevices function call getReleaseString function with (./pyanaconda/storage/__init__.py):

...
    try:
        # pylint: disable-msg=E1101
        mi = ts.dbMatch('provides', 'initscripts')
    except Exception:
        # This could happen in a variety of cases, but the biggest one is we're
        # examining an installed system that doesn't use RPM.  Raise an
        # exception for the caller to handle.
        raise ValueError
...

the problem is that RPM db is not in same partition.

When I copy whole "lib/rpm*" from "/var" (sda5) partition into "/" partition (sda1), anaconda find the system and run upgrade.

Comment 24 Jaroslav Pulchart 2011-10-23 15:42:15 UTC
New How reproducible:
F15 is installed with separated partitions for "/var" and "/".

Steps to Reproduce:
1. run preupgrade from F15 to F16,
2. reboot,
3. wait.

Comment 25 Alexander Dyadyun 2011-10-23 17:41:45 UTC
You are right! My "/var" and "/" on separated LVM volumes too. :)

Comment 26 Chris Lumens 2011-10-24 14:44:13 UTC
Ugh, good catch.

Comment 27 Jaroslav Pulchart 2011-10-25 06:15:05 UTC
Isn't this issue relevant to this anaconda commit http://git.fedorahosted.org/git/?p=anaconda.git;a=commitdiff;h=6fd9e3e1b602c701138fb1509cb7ef632c1886d2;hp=106f7d44099f2a11c156ed30d08f14d860c5d4be ?


from this commit relArch is checked in getReleaseString function:
--- a/pyanaconda/storage/__init__.py
+++ b/pyanaconda/storage/__init__.py
@@ -1387,9 +1387,21 @@ class Storage(object):
         return 0
 
 def getReleaseString(mountpoint):
+    relArch = None
     relName = None
     relVer = None
 
+    import rpm
+    iutil.resetRpmDb(mountpoint)
+    ts = rpm.TransactionSet(mountpoint)
+
+    # We get the arch from the initscripts package, but the version and name
+    # must come from reading the release file.
+    mi = ts.dbMatch('provides', 'initscripts')
+    for h in mi:
+        relArch = h['arch']
+        break
+
     filename = "%s/etc/redhat-release" % mountpoint
     if os.access(filename, os.R_OK):
         with open(filename) as f:
@@ -1406,7 +1418,7 @@ def getReleaseString(mountpoint):
             relName = product
             relVer = version.split()[0]
 
-    return (relName, relVer)
+    return (relArch, relName, relVer)
 
 def findExistingRootDevices(anaconda, upgradeany=False):
     """ Return a tuple of:
@@ -1444,9 +1456,10 @@ def findExistingRootDevices(anaconda, upgradeany=False):
             continue
 
         if os.access(anaconda.rootPath + "/etc/fstab", os.R_OK):
-            (product, version) = getReleaseString(anaconda.rootPath)
+            (arch, product, version) = getReleaseString(anaconda.rootPath)
+
             if upgradeany or \
-               anaconda.instClass.productUpgradable(product, version):
+               anaconda.instClass.productUpgradable(arch, product, version):
                 rootDevs.append((device, "%s %s" % (product, version)))
             else:
                 notUpgradable.append((product, version, device.name))

Comment 28 Adam Williamson 2011-10-27 00:38:34 UTC
gonna propose this as NTH, I believe the impact is that you can't upgrade systems with /var partitions via anaconda.

Comment 29 Dennis Gilmore 2011-10-27 00:40:14 UTC
im +1 for nth

Comment 30 Tim Flink 2011-10-27 00:43:06 UTC
I'm +1 on NTH, too

Comment 31 Chris Lumens 2011-10-27 02:09:18 UTC
*** Bug 749407 has been marked as a duplicate of this bug. ***

Comment 32 Chris Lumens 2011-10-27 02:20:41 UTC
Note that as a workaround, the "upgradeany" boot parameter ought to work.  It's worth trying, at least.

Comment 33 Alex Kelly 2011-10-27 14:14:37 UTC
I just tried "upgradeany" and got the same error.  I might be hitting a slightly different issue though, because all of my mounts (excluding /boot) are within an LVM.

Comment 34 Jaroslav Pulchart 2011-10-27 16:18:11 UTC
It doesn't depend on LVM or non-LVM partitioning but depend on separated volumes for "/" and "/var". 

The "upgradeany" workaround parameter doesn't work for me too.

Comment 35 John Chivall 2011-10-27 17:59:55 UTC
"upgradeany" does not work for me. I was able to upgrade after copying /var/lib/rpm/* to the root volume so that anaconda could find it, but this is a tedious workaround!

Comment 36 Adam Williamson 2011-10-28 19:06:47 UTC
Discussed at 2011-10-28 NTH review meeting, accepted as NTH as this is an upgrad issue and cannot be fixed with an update.

Comment 37 Izhar Firdaus 2011-11-07 03:12:01 UTC
*** Bug 751611 has been marked as a duplicate of this bug. ***

Comment 38 Izhar Firdaus 2011-11-08 22:55:31 UTC
my workaround to upgrade this was to boot preupgrade to init 1, modified the code below, and switch to init 5 .. 

-    import rpm
-    iutil.resetRpmDb(mountpoint)
-    ts = rpm.TransactionSet(mountpoint)
-
-    # We get the arch from the initscripts package, but the version and name
-    # must come from reading the release file.
-    mi = ts.dbMatch('provides', 'initscripts')
-    for h in mi:
-        relArch = h['arch']
-        break
+    relArch = 'x86_64'


probably use another way to check for relArch instead of through reading the initscripts rpm ? ..

Comment 39 Chris Lumens 2011-11-09 03:35:51 UTC
*** Bug 752241 has been marked as a duplicate of this bug. ***

Comment 40 Horst H. von Brand 2011-11-09 12:31:50 UTC
(In reply to comment #36)
> Discussed at 2011-10-28 NTH review meeting, accepted as NTH as this is an
> upgrad issue and cannot be fixed with an update.

How so? If F16 anaconda is updated, it should then work (AFAIU, preupgrade gets latest packages)?

Comment 41 Richard Guest 2011-11-10 00:32:15 UTC
(In reply to comment #38)
> my workaround to upgrade this was to boot preupgrade to init 1, modified the
> code below, and switch to init 5 .. 
> 
> -    import rpm
> -    iutil.resetRpmDb(mountpoint)
> -    ts = rpm.TransactionSet(mountpoint)
> -
> -    # We get the arch from the initscripts package, but the version and name
> -    # must come from reading the release file.
> -    mi = ts.dbMatch('provides', 'initscripts')
> -    for h in mi:
> -        relArch = h['arch']
> -        break
> +    relArch = 'x86_64'

I had to do this as well.
Such a stupidly annoying bug!

Comment 42 libregeek 2011-11-10 06:29:18 UTC
I have the same issue with my F15(x86_64) desktop with a separate partition for var. I copied the rpm files(as suggested in comment #35) using the following command:

# cp /var/lib/rpm/* / -rf

But it didn't worked and got the following error message:

"The root for the previously installed system was not found"

On exiting the installer, I got the following message in the console:

"error: cannot open Packages database in /mnt/sysimage/var/lib/rpm"

Comment 43 Jaroslav Pulchart 2011-11-10 08:49:01 UTC
You must keep the rpm dir. location:
1/ Boot to livecd
2/ mount your "/" and "/var" volumes into two different mount points 
3/ 
   mkdir -p </ mount point>/var/lib/
   cp -rpvaf <your /var mount point>/lib/rpm  <your / mount point>/var/lib/

4/ umount it
5/ reboot

Comment 44 Georgi Gonev 2011-11-10 08:53:35 UTC
Hi guys, and I have the same problem with my F15(x86) with separate LVM volume for the /var. First try to copy /var/lib/rpm/* to /

# cp -r /var/lib/rpm/* /

After that start preupgrade and got the following error message:

"The root for the previously installed system was not found"

Comment 45 Philippe Makowski 2011-11-10 10:42:30 UTC
this is very problematic
I have also /var in another volume and I can't upgrade
when and how can it be solved ?

Comment 46 Jaroslav Pulchart 2011-11-10 11:15:54 UTC
This bug is stupid and workaround is not easy. You need Linux administration experience.

You have to recreate full "/var/lib/rpm" location in "/" partition/volume. You could do it only when separated "/var" is unmounted. 

One again:

0/ check your UUID of "/" and "/var" volumes
   $ blkid

1/ Create LiveCD and boot into it

2/ open terminal 

3/ be root
   $ su -

4/ create mount points
   $ mkdir /mnt/root
   $ mkdir /mnt/var

5/ mount your "/" volume into /mnt/root
   $ mount UUID=<UUID of /> /mnt/root

   in my case (see attached grub):
   $ mount UUID=0bde3f70-2476-4f39-a3a9-f28d9fb50daf /mnt/root

6/ mount your "/var" volume into /mnt/root
   $ mount UUID=<UUID of /var> /mnt/var

   in my case (see attached grub):
   $ mount UUID=13f4d0c5-e001-4ca6-98ba-3776549f5621 /mnt/var

7/ copy your /mnt/var/lib/rpm into /mnt/root/
   $ mkdir -p /mnt/root/var/lib/
   $ cp -rdpva /mnt/var/lib/rpm /mnt/root/var/lib/

8/ umount 
   $ umount /mnt/var
   $ umount /mnt/root

Comment 47 Philippe Makowski 2011-11-10 11:21:53 UTC
thanks, I can do this, but really an anaconda update would be a better solution IMHO

Comment 48 Jaroslav Pulchart 2011-11-10 11:24:06 UTC
I agree.

Comment 49 libregeek 2011-11-10 11:28:44 UTC
(In reply to comment #43)
> You must keep the rpm dir. location:
> 1/ Boot to livecd
> 2/ mount your "/" and "/var" volumes into two different mount points 
> 3/ 
>    mkdir -p </ mount point>/var/lib/
>    cp -rpvaf <your /var mount point>/lib/rpm  <your / mount point>/var/lib/
> 
> 4/ umount it
> 5/ reboot

I didn't had a livecd or dvd drive. So I disabled /var in fstab and switched to "init 1", copied /var/lib/rpm and proceeded with preupgrade. 

But then another error caused the process to exit which said repository cannot be found. So I went back to init 1 and then copied /var/cache/yum to /. Now everything is fine and this comment is being posted from Fedora 16 (x86_64). 

This is really an annoying bug, but the wonderful people here are quick to respond and I was able to solve it without much hassle. 

Thanks everybody. :)

Hope to have a clean fix ASAP.

Comment 50 Jaroslav Pulchart 2011-11-10 11:37:16 UTC
Manilal: use livecd on USB key

Comment 51 Izhar Firdaus 2011-11-10 15:46:30 UTC
is theres other reliable way to determine the arch instead of querying RPM? 

why not the /bin/arch command? .. I believe the arch user want to upgrade to is always the same as the arch of preupgrade / DVD ..

Comment 52 Chris Lumens 2011-11-10 16:26:59 UTC
Izhar - that's a really good idea.  We can still run /bin/arch on the target system (provided you haven't done anything really crazy like put /bin or /lib on a separate partition) and get the same information.

Anyone affected by this bug who wants to give a fix a try, add the following to your boot arguments:

updates=http://clumens.fedorapeople.org/748119.img

Comment 53 Horst H. von Brand 2011-11-10 18:28:30 UTC
(In reply to comment #52)
> Izhar - that's a really good idea.  We can still run /bin/arch on the target
> system (provided you haven't done anything really crazy like put /bin or /lib
> on a separate partition) and get the same information.

Placing /, /bin, /lib on separate partitions has _never_ worked for me, on any unixy system, ever. The {/,/usr/}{bin,sbin,lib} separation was essential when disks where small, and you had to distribute your system on whatever places you had some space.

Comment 54 Guillaume DUVIOL 2011-11-10 18:39:14 UTC
(In reply to comment #52)
> Izhar - that's a really good idea.  We can still run /bin/arch on the target
> system (provided you haven't done anything really crazy like put /bin or /lib
> on a separate partition) and get the same information.
> 
> Anyone affected by this bug who wants to give a fix a try, add the following to
> your boot arguments:
> 
> updates=http://clumens.fedorapeople.org/748119.img

Thanks, I got / and /var on separate partitions, arch x86_64.
Your fix worked for me.

Comment 55 Adam Williamson 2011-11-11 00:38:23 UTC

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 56 Chris Lumens 2011-11-11 15:23:59 UTC
This will be fixed in the next build of anaconda for rawhide.  For F16, you can continue using the updates image referenced in comment #52.  I also suggest this bug be put on the Common Bugs list and perhaps the updates image put somewhere more official.

Comment 57 Chris Lumens 2011-11-12 15:26:36 UTC
*** Bug 753392 has been marked as a duplicate of this bug. ***

Comment 58 Michael 2011-11-12 16:44:59 UTC
Can someone explain to me, where exactly i must put the : 


updates=http://clumens.fedorapeople.org/748119.img

Comment 59 Michael 2011-11-12 22:02:27 UTC
After this fix, Anaconda find Fedora 15 for Upgrade but when i choose "create new boot loader configuration"  Popup an anaconda window with an Error.

Reported as Bug 743875

Comment 60 Chris Lumens 2011-11-13 00:59:22 UTC
See http://fedoraproject.org/wiki/Anaconda/Updates.

Comment 61 MauroChSS 2011-11-13 08:10:08 UTC
(In reply to comment #52)
> Izhar - that's a really good idea.  We can still run /bin/arch on the target
> system (provided you haven't done anything really crazy like put /bin or /lib
> on a separate partition) and get the same information.
> 
> Anyone affected by this bug who wants to give a fix a try, add the following to
> your boot arguments:
> 
> updates=http://clumens.fedorapeople.org/748119.img


Thanks, I got / and /var and yum cache on separate partitions, arch x86_64.
Your fix worked for me.

For those who are struggling trying to find a place to put "updates=http......"


While in grub menu do as follows:

- Highlight the update item and press (e)dit;
- Highlight the line that says "kernel /upgrade/vmlinuz preupgrade..... press (e)dit again;
- Add " linux options=http://clumens.fedorapeople.org/748119.img", press ENTER and then press (b)oot.

PS: Don't forget the space before linux options.

I hope this helps.

Comment 62 Peter Lister 2011-11-13 23:08:59 UTC
I added "updates=http://clumens.fedorapeople.org/748119.img" to the kernel line of my grub.conf entry for Preupgrade.

Upgrade to F16 proceeded without a hitch (arch=i686).

Comment 63 jean-pierre cartal 2011-11-14 08:41:52 UTC
Works for me too (i686)

Regards.

Comment 64 John Beranek 2011-11-14 12:32:53 UTC
Just tried to preupgrade a Fedora 15 x86_64 using the updates image. It did _not_ work for me.

The only that worked were the suggestions from http://ask.fedoraproject.org/question/133/preupgrade-failed-during-reboot to copy /var/lib/rpm and /var/cache/yum onto the root partition.

Comment 65 Panos Kavalagios 2011-11-14 12:48:40 UTC
Wild guess. Have you attempted to use the updated image through a wireless interface? It didn't work for me either. It seems that NetworkManager fails to set-up correctly the wireless interface. Maybe is doing better in a wired interface.

I've copied the image to /home, but again failed. Copied to /var bug again failed due to errors reading some sockets. My last solution is to use a USB stick to have anaconda image updated, because I wouldn't like to copy anything in / from /var. The idea of having two rpmdb copies in the system is terrifying. I understand that this would be a temporary shadow copy that the mount point of /var will overwrite, but again I would like to be absolutely certain that the upgrade is going to use the correct rpmdb stored in a single path in the system.

Comment 66 Chris Lumens 2011-11-14 14:49:25 UTC
*** Bug 753640 has been marked as a duplicate of this bug. ***

Comment 67 John Beranek 2011-11-14 15:35:06 UTC
I tried the update image on a USB stick, because an http URL to an update image is no good in my corporate environment, where the update would need to use a proxy server.

So, the update image was definitely used, but it did not solve the problem with Anaconda finding the system. One thing I did note was that at the point it failed "lvscan" showed all my LVs as inactive. Is that why the installer was failing?

As for worrying about shadow copies of the RPM database, I might allay your fears by telling you that by the time that the installer was doing anything like installing packages it _had_ mounted the correct /var from the LV I had. I checked this because I too wanted to ensure it wouldn't be updating the RPM DB from the root filesystem.

Comment 68 Gene Snider 2011-11-14 18:17:32 UTC
This bug appears to affect booting the install media into rescue mode.  Can the fix in Comment #52 be applied to the install media without enabling the network?  Or should I reinstall with /var under '/'?

Thanks,
Gene

Comment 69 Ryan Martin 2011-11-14 19:08:29 UTC
(In reply to comment #52)
> Anyone affected by this bug who wants to give a fix a try, add the following to
> your boot arguments:
> 
> updates=http://clumens.fedorapeople.org/748119.img

I had /boot on a normal partition but / and /var on separate LVMs.

I added the updates=... to the end of the preupgrade line in my grub config, using the grub editor. The upgrade is now proceeding normally. Thanks!

Comment 70 Panos Kavalagios 2011-11-15 06:50:25 UTC
(In reply to comment #67)
> I tried the update image on a USB stick, because an http URL to an update image
> is no good in my corporate environment, where the update would need to use a
> proxy server.
> 
> So, the update image was definitely used, but it did not solve the problem with
> Anaconda finding the system.

I also confirm that behaviour on a system having old-school ext4 partitions /, /usr, /var/, /home and /opt, with no LVM. The updated image was loaded from a USB stick and the problem wasn't resolved.

So, it seems that the only workarounds are the copy of the rpmdb and the update URL (proxy can also be configured as kernel options). I would really like to know how those can work on a server machines having no display attached. Maybe editing of grub.conf to provide the update URL and pray that everything will go OK. Finding a screen to plug might be a better idea anyway. Remote upgrades are not possible anymore for F16, unless yum upgrade method is considered.

Comment 71 Peter Lister 2011-11-15 14:05:41 UTC
(In reply to comment #70)

I added the "update=" mantra to the line in grub.conf; it works OK.

Comment 72 Chris Lumens 2011-11-15 15:02:32 UTC
*** Bug 752724 has been marked as a duplicate of this bug. ***

Comment 73 StanMcIntosh 2011-11-16 00:13:43 UTC
Thanks to Chris Lumens my computer running F15 has now preupgraded to F16. I have been structuring my Redhat and Fedora systems since 4.2 with several volumes separate from / (root). I believe I added /var as a separate volume around FC6. This method of structuring Linux was recommended by some Linux guru (I am not one). Adding updates=... as the last entry before the initrd in the title upgrade entry of the grub.conf @ /boot/grub/ caused the preupgrade to start and run to completion. I only wish I had waited for this info before I did a clean install on the computer running F14 that could not preupgrade to F16. It is running properly, but missing a lot of the stuff the preupgrade would have updated. "MY BAD". The arch of my computers running Fedora is x86_64

Comment 74 Adam Williamson 2011-11-18 00:45:05 UTC

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 75 George R. Goffe 2011-11-21 02:47:10 UTC
Hi,

This is certainly an interesting thread.

I'm trying to upgrade from FC14 x86_64 to FC16 x86_64 and am having the same problem of the "root for the previously installed system not found". Unless I added the "updates" incorrectly, adding updates didn't work for me. I suspect that the network has to be up before this will work. Right anyone?

Here's my kernel line for the upgrade:    kernel /upgrade/vmlinuz preupgrade repo=hd::/var/cache/yum/preupgrade ks=hd:UUID=5fd431e6-2892-4a1d-b8e3-c18d384edbce:/upgrade/ks.cfg linux options=http://clumens.fedorapeople.org/748119.img. The UUID is for /dev/sda1 which is my /boot partition.

I too have multiple partitions for various system file systems (/boot, /, /var, /usr, /home). Just getting the rpms downloaded to /var/cache caused me to run out of space in a 2GB /var partition. I moved cache to /home and made it a link in /var. I bet this will cause preupgrade some consternation too?

It seems that the rescue boot processing can work out correctly where the partitions should be mounted. Is the upgrade process using their code? Maybe it should?

George...

Comment 76 Wolfram Wagner 2011-11-21 08:25:18 UTC
I leave this thread, as I have finally upgraded via yum... (difficult enough)

This is one of those bugs, that make normal users bail out. If we need to improve something then the release/update quality. Updates render systems much too often. The last (sporadic but frequent freezes) and this update busied me for too many days... Next time I upgrade a month after the release...

Comment 77 StanMcIntosh 2011-11-21 17:09:26 UTC
(In reply to comment #75)
> Hi,
> 
> This is certainly an interesting thread.
> 
> I'm trying to upgrade from FC14 x86_64 to FC16 x86_64 and am having the same
> problem of the "root for the previously installed system not found". Unless I
> added the "updates" incorrectly, adding updates didn't work for me. I suspect
> that the network has to be up before this will work. Right anyone?
> 
> Here's my kernel line for the upgrade:    kernel /upgrade/vmlinuz preupgrade
> repo=hd::/var/cache/yum/preupgrade
> ks=hd:UUID=5fd431e6-2892-4a1d-b8e3-c18d384edbce:/upgrade/ks.cfg linux
> options=http://clumens.fedorapeople.org/748119.img. The UUID is for /dev/sda1
> which is my /boot partition.
> 
> I too have multiple partitions for various system file systems (/boot, /, /var,
> /usr, /home). Just getting the rpms downloaded to /var/cache caused me to run
> out of space in a 2GB /var partition. I moved cache to /home and made it a link
> in /var. I bet this will cause preupgrade some consternation too?
> 
> It seems that the rescue boot processing can work out correctly where the
> partitions should be mounted. Is the upgrade process using their code? Maybe it
> should?
> 
> George...

Instead of linux options= use updates= That worked for me see my comments in #73 above. Looks like you have it in the right place, but try the substitution of updates=.

Comment 78 George R. Goffe 2011-11-21 19:40:37 UTC
Stan,

Thanks for your response. I'll try this when I get home from work tonight.

I also copied the image to /upgrade... We'll see how that goes.

Thanks,

George...

Comment 79 Adam Williamson 2011-11-21 23:43:37 UTC
wolfram: "This is one of those bugs, that make normal users bail out."

"Normal users" do not have separate /var partitions. The only way to achieve that is to do manual partitioning and decide you want a /var partition, which is not something a "normal user" is going to do.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 80 Adam Williamson 2011-11-21 23:44:41 UTC
george: stan is correct, your line was wrong. the stanza is 'updates=http://blahblah.img'.

yes, it requires a network connection, but when you specify an updates.img on a remote server, anaconda will notice this and prompt you to configure a network connection before trying to retrieve it.

Comment 81 Jaroslav Pulchart 2011-11-22 08:07:40 UTC
Adam Williamson: 

What is normal? I'm normal, my colleges are normal and each of them have separated /var :). 

You mean "normal" user as non IT guy? If so:
*/ Normal user doesn't install Linux :( because the GUI is non-copleted, GUI applications are non-integrated (Linux looks more like some simple tool for geeks for them)
*/ if they use Linux someone like I will install it and maintain it. 
Therefore normal user have separated /var.

Moreover this issue was introduced in anaconda F16 (!), it works before this anaconda commit 6fd9e3e1b602c701138fb1509cb7ef632c1886d2. Where the arch was checked in place where /var is already mounted. Anaconda should have some workflow that nobody can move code from one place to another without thinking about it.

Comment 82 Izhar Firdaus 2011-11-22 09:04:47 UTC
normal people is a myth, everyone talks about it, but nobody seems to know one.

Comment 83 Panos Kavalagios 2011-11-22 12:38:05 UTC
I would suggest to stay on topic. We should not neglect that this is a bug report.

May I ask if we can remove, after the upgrade to F16, the temporary rpmdb copy in / filesystem or we have to keep it for system bootstrap or for future upgrades? 

I would like to verify it, as it seems that there is no love for hard disk partitions among Fedora experts. I've also noticed that /usr partition is not proposed anymore by F16 anaconda custom hard disk layout for example. The /var is still there for the time being.

Comment 84 Chris Lumens 2011-11-22 14:44:04 UTC
> Moreover this issue was introduced in anaconda F16 (!), it works before this
> anaconda commit 6fd9e3e1b602c701138fb1509cb7ef632c1886d2. Where the arch was
> checked in place where /var is already mounted. Anaconda should have some
> workflow that nobody can move code from one place to another without thinking
> about it.

All anaconda patches are discussed on the anaconda-maint-list.  If you feel you have something to add, you are welcome to follow along and comment on patches, where you can raise issues like this.  The fact is that anaconda is an extremely large and complex code base, and it is difficult to see all consequences of all changes.

Comment 85 Adam Williamson 2011-11-22 15:50:11 UTC
jaroslav: it was a quotation from wolfram.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 86 George R. Goffe 2011-11-24 00:37:44 UTC
Hi all,

WOW! The concept of "normal" user hit one of my hot buttons.

I apologize in advance for the off topic comments but just can't help myself.

I am CERTAINLY NOT a "normal" user. I have separate partitions for boot, root, var, opt and home. I started doing Unix like operating systems some time ago but WILL adamantly NOT claim to be an expert, there's just too much that I don't know.

When I started learning Unix "we" had separate partitions as above because having just one big partition had several drawbacks. I know there are workarounds for these drawbacks now and that a "normal" user won't even see their impact, but permit me to list them anyway (I'm sure there are more):

1) A system with one big partition is more likely to sustain damage to the whole operating system + user data if any (i.e., I/O errors or filesystem corruption) whereas separate filesystems can isolate the problem(s).

2) Admittedly, ext4 AND journaling makes FSCK much quicker but FSCK used to take a LONG time where the bigger the partition, the longer it took. I have 2TB drives at home and running FSCK with ext3 is PAINFUL whereas ext4 is VERY fast.

3) Backing up the whole partition took a long time as well as restoring it after a drive crash. Additionally, just restoring a single file would take longer too.

4) Upgrading or replacing an OS is more difficult to do with one big partition whereas having a different partition for user data makes it simpler.

5) With computers becoming faster and memory cheaper, performance is not as much of an issue but filesystem access times ARE affected by the number of inodes to be traversed.

I purposely did not upgrade to FC15 because of the partitioning issues. Pre-defined partitons seemed to be ignored and the seeming inability to define partitions as above ARE big turnoff's for me. I find myself horrified to find that FC16 and beyond is heading the same direction. 

Just who is it that defines "normal"? What's good for the "normals" among us just makes it harder for those of us who are not. Sigh.

Again, I apologize for the off topic comment... My bad. I'll go to my room now.

Flames to /dev/null.

George...

Comment 87 Jaroslav Pulchart 2011-11-24 06:33:23 UTC
George: Yes, there are many reasons why to have separated /var volume (better I/O for it on faster/second/bigger hdd, ...)

Chris: I did not want to attack any anaconda developer, sorry. I wanted to highlight the missing anaconda low level workflow (diagram, tasks, inputs/outputs, requirements in each step), so that the same issue doesn't come up in the future anymore.

Comment 88 George R. Goffe 2011-11-24 09:55:27 UTC
Jaroslav,

Thank you for not jumping on me about this and the off topic.

George...

Comment 89 George R. Goffe 2011-11-24 10:15:05 UTC
Created attachment 535839 [details]
flat file containing anaconda.log

Meanwhile... Another gotcha.

The updates=http... worked perfectly. It looks like the upgrade process assumes DHCP is available and didn't prompt me for other than the network interface to use. Once I specified the wired interface, the file was downloaded... successfully. Apparently the updates= parameter will ONLY allow you to have the updates.img on a different system via the network. It would be nice to allow a local file specification here as well. 

Because my /var partition is too small to hold the preupgrade materials, I had to make /var/cache a link to my home directory. 

The upgrade appears to be done in a chrooted environment since /mnt/sysimage partitions were mounted (the apparent result of a "chroot /mnt/sysimage"). The message below does not appear to be from the chrooted environment though.

I was able to capture the anaconda.log (enclosed below) which shows this message:

01:30:31,750 WARN anaconda: Try 1/10 for file:///mnt/sysimage//var/cache/yum/preupgrade/repodata/repomd.xml failed: [Errno 16] error opening local file from file:///mnt/sysimage//var/cache/yum/preupgrade/repodata/repomd.xml, IOError: [Errno 2] No such file or directory: '/mnt/sysimage/var/cache/yum/x86_64/16/anaconda-0/repomdUvBbEqtmp.xml'

If you try to cd to this directory you will only get as far as /var since /var/cache is a link. It looks like the anaconda(?) upgrade process doesn't handle the link correctly. It should prepend /mnt/sysimage to the link.

Am I missing something here?

George...

Comment 90 Ian Collier 2011-11-24 10:35:33 UTC
George: use a relative symlink (e.g. "ln -s ../home/cache /var").  You can't rely on the environment being chrooted and it's certainly unreasonable to expect anaconda to munge the target of a symlink.

Comment 91 Adam Williamson 2011-11-24 22:20:56 UTC
George: not only off-topic, but flailing at a straw man. No-one said 'Fedora should cater only to normal users'. Someone suggested this bug would be a big problem for normal users, and I suggested it wouldn't, as 'normal users' generally don't have a /var partition. That's the only context in which the term appeared.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 92 Adam Williamson 2011-11-24 22:23:59 UTC
"Apparently the updates= parameter will ONLY allow you to have the updates.img on a different system via the network."


No, it doesn't. This is just the most commonly-used and usually the easiest way of doing it.

"It would be nice to allow a local file specification here as well."

This is allowed. See https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_installation_source and https://fedoraproject.org/wiki/QA:Testcase_Anaconda_updates.img_via_local_media .



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 93 George R. Goffe 2011-11-25 20:08:12 UTC
Adam,

I tried pointing the updates to the same partition where the upgrade directory is but the Anaconda process (kickstart?) went to the network prompts which usually indicates that a parameter is not specified correctly.

How would you code the "updates=" parameter to do what I tried to do?

George...

Comment 95 nada 2011-11-28 18:44:19 UTC
Tnx Chris, your image worked for me.

That's a really nasty bug BTW - I upgraded 3 systems with /var-partition on a separate logical-volume without any troubles and then ran quite unexpected into the issue with the 4th system (where /var was on a separate physical partition...)

Comment 96 George R. Goffe 2011-11-28 19:05:53 UTC
Howdy,

I'm HAPPY to finally report a successful upgrade. Thanks to Ian Collier for his suggestion of using a relative symlink for my /var/crash. My /var IS on a separate partition AND /var/crash had to be moved as well. The upgrade does seem to happen in a chrooted environment as far as I can tell. I ended up having to shuffle various directories around so that I would have enough space to upgrade. This took 4 boots but finally worked!

The upgrade process itself almost went without a hitch. There was a failure which I'll report here when I get home from work tonight. glibc cleanup and grub had some sort of problem. I saved all the logs.

I thought I was a goner when I saw this message and wondered what the condition of my system was in. I copied logs and then rebooted. Grub2 (I think) listed several kernels so I chose the default. The system booted cleanly and seemed to pick right up where it left off (just after the glibc/grub problem). I AM VERY PLEASED AND AMAZED AND APPRECIATIVE AND ECSTATIC for all the hard work that ALL you people have done for the upgrade process! Kudos++ to you all!

George...

Comment 97 George R. Goffe 2011-11-28 23:24:47 UTC
Oops... /var/cache not /var/crash.

This was a FC 14 to FC 16 upgrade.

George...

Comment 98 George R. Goffe 2011-11-29 06:42:16 UTC
Here's the failure I mentioned during the upgrade described above.

George...

06:58:17,214 ERR anaconda: AnacondaYum._run: PackageSackError: Rpmdb checksum is invalid: dCDPT(pkg checksums): grub.x86_64 1:0.97-84.fc16 - u

Comment 100 info@kobaltwit.be 2011-11-30 22:54:41 UTC
Adding the updates=... option on boot helped me upgrade my system to F16. Thanks !

Comment 101 George R. Goffe 2011-12-02 00:22:29 UTC
Hi,

Well, the honeymoon is over now.

I can not seem to be able to compile c and c++ programs on this newly upgraded system.

I suspect it has something to do with the failure in "glibc cleanup" that I experienced. I rebooted and the upgrade appeared to pick up where it left off. This may be the cause of the problem.

I have "yum reinstalled" gcc\* and glib\* and binutils\* without any change in the behavior. See "Red Hat Bugzilla – Bug 758528" for more details. I get this message from all attempts to compile: /"usr/bin/ld: cannot find crt1.o: No such file or directory"

I saved the logs from the upgrade if those will help.

Does anyone have any suggestions please?

Thanks,

George...

Comment 102 Adam Williamson 2011-12-02 00:46:11 UTC
[adamw@adam ~]$ rpm -qf /usr/lib64/crt1.o
glibc-devel-2.14.90-20.fc17.x86_64

try (re)installing glibc-devel.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 103 George R. Goffe 2011-12-02 05:16:51 UTC
Adam,

Thanks for the suggestion.

It didn't work as far as I can tell. I get the same error msg..

clipper bash-4.2 ~# rpm -qf /usr/lib64/crt1.o
glibc-devel-2.14.90-19.x86_64
clipper bash-4.2 ~# ls -al !$
ls -al /usr/lib64/crt1.o
-rw-r--r-- 1 root root 1648 Nov 20 20:44 /usr/lib64/crt1.o

I tried strace on "gcc tst.c". As far as I can tell, nothing (ld) looked in /usr/lib64 for the crt*.o files. ld did appear to look in /usr/lib though. Is ld broken or are both versions of glibc-devel required?

Any ideas please?

Regards,

George...

Comment 104 George R. Goffe 2011-12-02 08:15:39 UTC
Adam,

Just for fun, I installed glibc-devel.i686 and tried "gcc tst.c"... Here's what I got.

clipper bash-4.2 ~# gcc tst.c
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../../libc.so when searching for -lc
/usr/bin/ld: i386 architecture of input file `/usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../../crt1.o' is incompatible with i386:x86-64 output
/usr/bin/ld: i386 architecture of input file `/usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../../crti.o' is incompatible with i386:x86-64 output
/usr/bin/ld: i386 architecture of input file `/usr/lib/gcc/x86_64-redhat-linux/4.6.2/../../../crtn.o' is incompatible with i386:x86-64 output
collect2: ld returned 1 exit status

George...

Comment 105 Michal Jaegermann 2011-12-15 23:49:17 UTC
Hm, interesting.  I was under an impression that if anaconda will find an update in 'RHupdates' subdirectory of a repository then such update will be applied "automagically".  I do have here a small "portable" server with a suitable repo and I did put 748119.img in 'RHupdates' there. Anacoda did find that image and copied it to /tmp/updates/ on a target machine but then repeatedly failed. None of existing file systems from a machine with a separate /var was found. OTOH when I explicitely added 'updates=http://....', using a copy on my server, then this immediately helped and I am updating my "target" right now.

Do I miss something here in how Anaconda is supposed to work?  I did save logs from my "rescue" attempts if anybody would like to see them.

Comment 106 Adam Williamson 2012-01-23 20:06:09 UTC
I'm going to bump this bug to Rawhide and close it, since the fix is in F17's anaconda, and we have provided an updates.img for F16; I think that's as good as it's going to get.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 107 Popescu 2012-02-02 10:21:07 UTC
I have /var on separate partition and when I try upgrade Fedora 15 to F16 after download packets and reboot tell me: Upgrade root not found
I add the following to my boot arguments:
updates=http://clumens.fedorapeople.org/748119.img

and after chose my eth card, tell me "Waiting for Network Manager to configure p3p1" and after that don't can access http://clumens.fedorapeople.org/748119.img
I try and with: http://clumens.fedorapeople.org/updates.img

My network IP is set manualy and I think this may be the cause.
Can I manually set network card options for the upgrade?

Comment 108 Miked 2012-02-04 16:30:04 UTC
The URL http://clumens.fedorapeople.org/748119.img gives me a 404

Where is the correct link please.

I have been working all day to update my F15 to F16 and it looks like I have this problem as my /var is on a separate logical volume.

I have added the URL to my boot parameters but fails as link is not valid.

Comment 109 Ryan Martin 2012-02-09 15:57:01 UTC
@Miked, the img file appears to have been moved to http://clumens.fedorapeople.org/updates.img

Comment 110 Chris Lumens 2012-02-09 16:03:08 UTC
No, it's not, and you should never rely on the contents of updates.img.  That's my own private testing stuff.

Comment 111 Miked 2012-02-09 16:33:52 UTC
Many thanks Chris for sorting out the link.

Comment 112 Ryan Martin 2012-02-09 16:41:37 UTC
(In reply to comment #110)
> No, it's not, and you should never rely on the contents of updates.img.  That's
> my own private testing stuff.

Okay, sorry to presume. Since 748119.img is clearly gone from your fedorapeople directory, where did it go?

Comment 113 Chris Lumens 2012-02-09 19:00:47 UTC
It should be back up now.  I got a little overzealous in my old updates image deleting.

Comment 114 Jonathan Wakely 2012-02-23 21:26:21 UTC
Sorry to comment on a closed bug...

When should I be offered the choice to update my existing system, presumably before the disk partitioning menu?  If so, the updates= parameter didn't work for me when attempting to upgrade F15 using the netinst cd media on x86_64, with / on a physical partition and /var on a logical partition (no LVM involved.)

All I should need to do is hit TAB on the menu and add the updates= after the existing args, right?

vmlinuz initrd=initrd.img quiet updates=http://clumens.fedorapeople.org/748119.img

The only effect that has is it causes anaconda to configure the network card, it doesn't offer me the upgrade dialog, so apparently doesn't find my existing system.

I guess I'll have to copy the rpm files to the root partition.

Comment 115 Jonathan Wakely 2012-02-23 21:38:56 UTC
rescue mode also fails to find the existing installation (I get a dialog saying "You don't have any Linux partition" which is nonsense, I have four!) even when adding the updates URL to the rescue mode parameters

Comment 116 Reinhard 2012-03-23 23:34:08 UTC
comment#115 matched my situation.
The 748119.img alone did not solve the issue.
I'm running /var as an LVM partition.
Booting into rescue mode did not find any LVM partition.

lvm vgchange <my vg> -ay

activated all partitions. Restarting anaconda resolved the issue, then.

Comment 117 Jonathan Wakely 2012-04-17 13:24:13 UTC
Reinhard, were you using the netinst CD?

Is this bug actually fixed properly, or will it recur when installing F17 from netinst CD media?

Comment 118 Brendan LeFebvre 2012-04-17 15:50:57 UTC
Created attachment 578082 [details]
Another LVM configuration that fails as described here, even with 748119.img

The 748119.img didn't fix my issue either. I have /var on a separate LVM partition but also several others. 

I honestly wouldn't expect preupgrade to work at all except that my current partition scheme was established in a Fedora 8 install and has successfully been preupgrade'd several times to the current Fedora 15 (I think I had to skip over 12, also due to Anaconda issues). 

Anyway, I am posting my config here in hopes that someone with Anaconda expertise will have a simple fix in mind, otherwise I will have to delve into rolling my own .img updates.