Bug 972358

Summary: update from 17 to 18 process stops at 67%
Product: [Fedora] Fedora Reporter: hw
Component: selinux-policyAssignee: Miroslav Grepl <mgrepl>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: collura, dominick.grift, dwalsh, hw, mgrepl, russ+bugzilla-redhat, tflink, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-17 15:30:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
/var/log/upgrade.log none

Description hw 2013-06-08 20:36:46 UTC
Description of problem:

+ following the instructions on http://fedoraproject.org/wiki/FedUp#How_Can_I_Upgrade_My_System_with_FedUp.3F to update from Fedora 17 to 18

+ running fedup-cli --network 18

+ rebooting into the update process after packages were downloaded

+ the update process stops at 67% with a message some selinux entry for gnomeclock has become invalid --- nothing further happens, so after a while I reboot by pressing Ctrl+Alt+Del, hoping I could start over

+ grub doesn't have the update option in its menu anymore

+ running fedup-cli --network 18 produces an error message:

[root@yun ~]# fedup-cli --network 18
setting up repos...
getting boot images...
.treeinfo                                                                                                                                                                                                               | 1.1 kB  00:00:00     
setting up update...
finding updates  6% [=============                                                                                                                                                                                                            ]Traceback (most recent call last):
  File "/bin/fedup-cli", line 330, in <module>
    main(args)
  File "/bin/fedup-cli", line 275, in main
    pkgs = download_packages(f)
  File "/bin/fedup-cli", line 57, in download_packages
    updates = f.build_update_transaction(callback=output.DepsolveCallback(f))
  File "/usr/lib/python2.7/site-packages/fedup/download.py", line 185, in build_update_transaction
    (rv, msgs) = self.buildTransaction(unfinished_transactions_check=False)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1125, in buildTransaction
    (rescode, restring) = self.resolveDeps()
  File "/usr/lib/python2.7/site-packages/yum/depsolve.py", line 897, in resolveDeps
    (checkdep, errormsgs) = self._processConflict(*conflict)
  File "/usr/lib/python2.7/site-packages/yum/depsolve.py", line 759, in _processConflict
    self._dscb_procConflict(po, niceformatneed)
  File "/usr/lib/python2.7/site-packages/yum/depsolve.py", line 745, in _dscb_procConflict
    self.dsCallback.procConflictPo(po, niceformatneed)
  File "/usr/lib/python2.7/site-packages/fedup/callback.py", line 142, in procConflictPo
    self.log.debug('CONFLICT: %s → %s', po, formatted_conflict)
NameError: global name 'po' is not defined
[root@yun ~]# 


Version-Release number of selected component (if applicable):

Version information is not available:

[root@yun ~]# man fedup
No manual entry for fedup
[root@yun ~]# fedup --version
usage: fedup SOURCE [options]
fedup: error: unrecognized arguments: --version
[root@yun ~]# fedup -version
usage: fedup SOURCE [options]
fedup: error: argument -v/--verbose: ignored explicit argument 'ersion'
[root@yun ~]# fedup --help
usage: fedup SOURCE [options]

Prepare system for upgrade.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         print more info
  -d, --debug           print lots of debugging info
  --debuglog DEBUGLOG   write lots of debugging output to the given file
  --reboot              automatically reboot to start the upgrade when ready

SOURCE:
  Location to search for upgrade data.

  --device [DEV]        device or mountpoint. default: check mounted devices
  --iso ISO             installation image file
  --network VERSION     online repos matching VERSION (a number or "rawhide")

additional options for --network:
  --enablerepo REPOID   enable one or more repos (wildcards allowed)
  --disablerepo REPOID  disable one or more repos (wildcards allowed)
  --addrepo REPOID=[@]URL
                        add the repo at URL (@URL for mirrorlist)
  --instrepo REPOID     get upgrader boot images from REPOID (default: auto)

cleanup commands:
  --resetbootloader     remove any modifications made to bootloader
  --clean               clean up everything written by fedup
[root@yun ~]# 


How reproducible:


Steps to Reproduce:

see above ...

Actual results:


Expected results:

Upgrading should work, shouldn't it? :)

Additional info:

What am I supposed to do now? I'm probably stuck at a mixture of Fedora 17 and 18 after the update process went only 67% through. Is there a way to do the update again? When running yum update, it says no packages are marked for update.

At least the system is working so far that I can write this report. Problems will probably show up later ...

Comment 1 hw 2013-06-08 20:42:23 UTC
Just after posting this, the screen went black and the computer rebooted. The previous kernel is still booted rather than the new one that was downloaded for the update; there's no entry for the new one.

Comment 2 hw 2013-06-08 20:57:18 UTC
And there we go:

pulseaudio: relocation error: /lib64/libpulsecore-1.1.so: symbol pa_format_info_to_sample_spec_fake, version PULSE_0 not defined in file libpulse.so.0 with link time reference

So now the libraries have been updated only partly ...

Comment 3 hw 2013-06-09 14:12:27 UTC
/usr/lib/python2.7/site-packages/fedup/callback.py

    def procConflictPo(self, name, formatted_conflict):
        self.log.debug('CONFLICT: %s → %s', po, formatted_conflict)


I don't know python, but shouldn't it either be:

    def procConflictPo(self, name, formatted_conflict):
        self.log.debug('CONFLICT: %s → %s', name, formatted_conflict)


or

    def procConflictPo(self, po, formatted_conflict):
        self.log.debug('CONFLICT: %s → %s', po, formatted_conflict)


In the meantime, I tried the suggestions given in https://lists.fedoraproject.org/pipermail/users/2013-June/436181.html without success. Yum doesn't have a transaction to complete --- actually it still seems to think it's Fedora 17 while /etc/fedora-release contains "Fedora release 18 (Spherical Cow)".

yum distro-sync --releasever=18 --skip-broken doesn't work, either:


[...]
--> Finished Dependency Resolution

Packages skipped because of dependency problems:
    clamav-0.97.8-1.fc18.x86_64 from updates
    libsidplayfp-1.0.1-4.fc18.x86_64 from updates
    m17n-lib-anthy-1.6.4-7.fc18.x86_64 from updates
    python-devel-2.7.3-13.fc18.x86_64 from fedora
    python3-devel-3.3.0-1.fc18.x86_64 from fedora
Error:  Multilib version problems found. This often means that the root
       cause is something else and multilib version checking is just
       pointing out that there is a problem. Eg.:
       
         1. You have an upgrade for libgcc which is missing some
            dependency that another package requires. Yum is trying to
            solve this by installing an older version of libgcc of the
            different architecture. If you exclude the bad architecture
            yum will tell you what the root cause is (which package
            requires what). You can try redoing the upgrade with
            --exclude libgcc.otherarch ... this should give you an error
            message showing the root cause of the problem.
       
         2. You have multiple architectures of libgcc installed, but
            yum can only see an upgrade for one of those arcitectures.
            If you don't want/need both architectures anymore then you
            can remove the one with the missing update and everything
            will work.
       
         3. You have duplicate versions of libgcc installed already.
            You can use "yum check" to get yum show these errors.
       
       ...you can also use --setopt=protected_multilib=false to remove
       this checking, however this is almost never the correct thing to
       do as something else is very likely to go wrong (often causing
       much more problems).
       
       Protected multilib versions: libgcc-4.7.2-8.fc18.i686 != libgcc-4.7.2-2.fc17.x86_64
Error: Protected multilib versions: glibc-2.16-31.fc18.i686 != glibc-2.15-59.fc17.x86_64
Error: Protected multilib versions: glib2-2.34.2-2.fc18.i686 != glib2-2.32.4-2.fc17.x86_64
Error: Protected multilib versions: libstdc++-4.7.2-8.fc18.i686 != libstdc++-4.7.2-2.fc17.x86_64
Error: Protected multilib versions: pcre-8.31-5.fc18.i686 != pcre-8.21-7.fc17.x86_64
Error: Protected multilib versions: zlib-1.2.7-9.fc18.i686 != zlib-1.2.5-7.fc17.x86_64
Error: Protected multilib versions: libselinux-2.1.12-7.3.fc18.i686 != libselinux-2.1.10-3.fc17.x86_64
Error: Protected multilib versions: sqlite-3.7.13-2.fc18.i686 != sqlite-3.7.11-3.fc17.x86_64
Error: Protected multilib versions: libuuid-2.22.2-6.fc18.i686 != libuuid-2.21.2-4.fc17.x86_64
Error: Protected multilib versions: nss-softokn-freebl-3.14.3-1.fc18.i686 != nss-softokn-freebl-3.14.3-1.fc17.x86_64
Error: Protected multilib versions: ncurses-libs-5.9-11.20130511.fc18.i686 != ncurses-libs-5.9-11.20130511.fc17.x86_64
Error: Protected multilib versions: libffi-3.0.10-3.fc18.i686 != libffi-3.0.10-2.fc17.x86_64
Error: Protected multilib versions: readline-6.2-5.fc18.i686 != readline-6.2-4.fc17.x86_64
Error: Protected multilib versions: gamin-0.1.10-13.fc18.i686 != gamin-0.1.10-12.fc17.x86_64
[root@yun ~]# 


Why do I even have 32bit libs installed, or why are they going to be installed?

Comment 4 hw 2013-06-09 15:22:50 UTC
changed /usr/lib/python2.7/site-packages/fedup/callback.py, line 141:

    def procConflictPo(self, po, formatted_conflict):
        self.log.debug('CONFLICT: %s → %s', po, formatted_conflict)


running fedup again, goes through now until it wants to reboot --- maybe it works this time ...

Comment 5 hw 2013-06-09 17:05:58 UTC
Ok, it went through this time and running package-cleanup --dupes |wc -l says 2710, listing packages from 17 and 18.

yum --releasever=18 --disableplugin=presto distro-sync says: No Packages marked for Distribution Synchronization

yum --releasever=18 --disableplugin=presto distro-sync full: Error: Problem in reinstall: no package 1:emacs-filesystem-24.2-18.fc18.noarch matched to install

Removing all packages from 17 will probably lead to a non-working system even before they're all removed. How is even possible to have the same packages from different releases installed at the same time? Package management is there to prevent things like this, and it's obviously totally broken in Fedora.

Comment 6 Will Woods 2013-06-12 18:00:48 UTC
(In reply to lee from comment #4)
> changed /usr/lib/python2.7/site-packages/fedup/callback.py, line 141:
> 
>     def procConflictPo(self, po, formatted_conflict):
>         self.log.debug('CONFLICT: %s → %s', po, formatted_conflict)

This was fixed in fedup-0.7.3 - see bug 895576.
fedup-0.7.3-4 is in updates, and fedup-0.7.3-5 is in updates-testing.

This is unrelated to whatever made your upgrade "stop" at 67%.

(In reply to lee from comment #0)
> What am I supposed to do now? I'm probably stuck at a mixture of Fedora 17
> and 18 after the update process went only 67% through. Is there a way to do
> the update again? When running yum update, it says no packages are marked
> for update.

If you do have an upgrade crash in the middle, you should normally just run the same 'fedup' command you ran before, and it should set up the system just like before. The reason it's not working is that you have an old version of fedup - see above.

(Oh, also - try "rpm -q fedup" to get the version.)

As for the upgrade hang itself - do you have a /var/log/upgrade.log you can attach?

Comment 7 hw 2013-06-13 18:55:19 UTC
Hm I was running "yum --enablerepo=updates-testing install fedup" to install fedup, as described in http://fedoraproject.org/wiki/FedUp#How_Can_I_Upgrade_My_System_with_FedUp.3F

It's unrelated to the stop at 67%, but the first thing I tried was running fedup again, hoping that it would pick up where it stopped and finish the upgrade.  I got the error message instead and didn't really dare to modify /usr/lib/python2.7/site-packages/fedup/callback.py at first.  But then, what could I do?

After that, it kinda finished the upgrade but left me with lots of dupes installed which I removed.  So the packages from 17 are gone now, and I still need to take a look at the 353 packages listed with 'package-cleanup --leaves --all'.

'rpm -q fedup' says "fedup-0.7.3-4.fc18.noarch". 'yum list available fedup' says "Error: No matching Packages to list".  Shouldn't fedup be available?

There is a logfile '/var/log/upgrade.log'.  It must have been overwritten on the second run of fedup, if the first run wrote one.  I'll attach the logifle once I figure out how to do that :)

Comment 8 hw 2013-06-13 18:56:54 UTC
Created attachment 760916 [details]
/var/log/upgrade.log

/var/log/upgrade.log, compressed with bzip2

Comment 9 hw 2013-07-16 11:19:03 UTC
When upgrading with fedup from F18 to F19, the upgrade seemed to be hanging at 68% when a new SELinux policy package was installed.  I waited for quite a while and after some time, the computer rebooted by itself.

The screen saver had kicked in while was waiting, so I didn't see what actually happened.  I suspect it merely takes quite a while for this package to install.

The upgrade worked flawless, the only issue is that the keymap for the console doesn't seem to get loaded anymore.  It is very well possible that the upgrade from F17 to F18 might have worked fine if I had waited longer before rebooting.


It would be nice if there was some progress indication or at least a warning message printed for instances when upgrading or installing a package can take unusually long, to reassure the users that their system doesn't hang.

Comment 10 Will Woods 2013-07-16 16:00:21 UTC
(In reply to lee from comment #9)

> It would be nice if there was some progress indication or at least a warning
> message printed for instances when upgrading or installing a package can
> take unusually long, to reassure the users that their system doesn't hang.

It would be nice, wouldn't it? Unfortunately it's just not possible because of the way that we use RPM %post scripts.

fedup has no control over the contents of those scripts - they're written by the packagers. And neither RPM itself nor the Fedora project have defined any way of returning progress information from those scripts.

So: we have no idea if any given script is going to take 30 milliseconds or 30 minutes, nor do we have any way for the package maintainer to tell us how long it's going to take, or how much progress they've made (so we could guess how long it would take).

The best we can really do with the current state of affairs is keep the Fedora logo blinking to assure you that the system itself hasn't hung and tell you to be patient.

That being said:

(In reply to lee from comment #9)
> When upgrading with fedup from F18 to F19, the upgrade seemed to be hanging
> at 68% when a new SELinux policy package was installed.  I waited for quite
> a while and after some time, the computer rebooted by itself.
> 
> The screen saver had kicked in while was waiting, so I didn't see what
> actually happened.  I suspect it merely takes quite a while for this package
> to install.

We could probably turn off the screen saver during the upgrade, which might help.

If it did turn out that something's going wrong and we aren't updating the progress bar after 67%, that *would* be a bug we could fix.. but so far nobody has confirmed that being the case.

Comment 11 hw 2013-07-18 06:54:26 UTC
It seems that replies sent by email don't show up here, so I'll paste it in:



bugzilla writes:

https://bugzilla.redhat.com/show_bug.cgi?id=972358

> --- Comment #10 from Will Woods <wwoods> ---
> (In reply to lee from comment #9)
>
>> It would be nice if there was some progress indication or at least a warning
>> message printed for instances when upgrading or installing a package can
>> take unusually long, to reassure the users that their system doesn't hang.
>
> [...]
>
> The best we can really do with the current state of affairs is keep the Fedora
> logo blinking to assure you that the system itself hasn't hung and tell you to
> be patient.

There is no Fedora logo on the screen during updates.  The nouveau
driver seems to be used to switch to a higher resolution so more text
can be displayed.  I'm seeing a numbered list of packages that are
upgraded/installed while fedup is working, and a progress indicator
printed, saying xx%.

That's nice because it says something like "[1623/1869] package-foobar
...", so I can watch how it progresses.  When it's working on the
selinux-policy package, there is no indication of progress at all.  (I'm
"normally" booting to the console and use 'startx' from there; I guess
that's why I'm not seeing a logo during upgrades.)


Now I got curious and have downloaded
'selinux-policy-3.12.1-63.fc19.src.rpm' and am looking at the
selinux-policy.spec file contained therein.  I'm not familiar with how
rpm packages are built, but it seems to me that 'restorecon' is being
run during the installation of the package.  There are several occasions
where it's being run, particularly:


,---- [ selinux-policy.spec:442--444 ]
| %triggerpostun targeted -- selinux-policy-targeted < 3.12.1-7.fc19
| restorecon -R -p /home
| exit 0
`----


It also seems to be running on some directories in /var (and some
others).  In my case, it has to go over about a million files on /home,
give or take a few 100k.  That can take a while.

The '-p' option to show the progress is only given to 'restorecon' in
line 443.  I don't know if the progress output from 'restorecon' would
show up on the screen because by the time it was running on /home, the
screen might have already been saving energy.

It would sure be nice if the progress output of 'restorecon' would be
shown on the screen (for every time it runs during the installation).

Is that really impossible?

If it is impossible to show the progress, would it be possible to do
these runs of 'restorecon' after the upgrade has finished?  That would
save a lot of time for people who have many files and could prevent them
from thinking that their computer hangs.

A blinking logo doesn't really do that.  Computers apparently doing
nothing for a while usually means that they are stuck, and it doesn't
matter if something is blinking or not.  People are used to that, and
sooner or later they will pull the plug.

BTW, what's the point of displaying a logo --- blinking or not ---
instead of displaying the progress like the list of packages I'm seeing?
Such a list is far more interesting than a blinking logo.


At last, if anything fails, there could be a warning in the release
notes explaining that during the upgrade of the selinux-policy package,
'restorecon' goes over some directories, including /home, and that it can
take the longer to upgrade this package the more files are stored in such
directories.  (Users could even run 'restorecon' before upgrading to
figure out how long it might take to complete.)


> (In reply to lee from comment #9)
>> The screen saver had kicked in while was waiting, so I didn't see what
>> actually happened.
>
> We could probably turn off the screen saver during the upgrade, which might
> help.

I think that won't be necessary.  I could have pressed the Ctrl key or
the like any time to see what's on the screen.  I only didn't do it
because I thought I'd give it quite some time and was reading a book.  I
might have gone to sleep after another while ...

> If it did turn out that something's going wrong and we aren't updating the
> progress bar after 67%,

Progress bar? :)  Only showing a bar and saying xx%?  Seriously?  I
wouldn't want a progress bar; the list of packages I get to see is way
better.

> that *would* be a bug we could fix.. but so far nobody
> has confirmed that being the case.

After the upgrade, I checked /var/log/fedup.log and didn't see any
problems reported.  All packages from F18 are gone, except for a very
few that seem to be installed because kernel packages from F18 have
remained which need them.  Running 'rpm -V -a' didn't show any problems,
either.  Everything seems to be working fine, except that the keymap for
the console isn't loaded (giving me something to think about when I
tried to log in ...).

That makes me think that the upgrade worked flawless.

Comment 12 Will Woods 2013-07-24 21:27:56 UTC
(In reply to lee from comment #11)
> 
> It would sure be nice if the progress output of 'restorecon' would be
> shown on the screen (for every time it runs during the installation).
> 
> Is that really impossible?

There's a subtle difference you seem to have missed here:

What I was saying is that it's currently not possible for fedup to display details about the progress of RPM scriptlets (e.g. selinux-policy), because *the scriptlets don't provide any information*. 

On the other hand, it's absolutely possible for *packages* to *emit* progress info, if they want. They usually don't, but (obviously) sometimes they should. 

So: I'm reassigning this bug to selinux-policy, because there's a few calls to restorecon/fixfiles in the .spec that should be printing progress information so people don't think the upgrade has hung.

Comment 13 Daniel Walsh 2013-07-24 21:30:28 UTC
We have done this type of output in the past but were asked to turn it off.  We can not give an accurate % so all we could do is print * or Some rising counter.

Comment 14 hw 2013-07-25 10:04:53 UTC
All that is needed is a way to distinguish between "computer hangs" and "computer doesn't hang".  If you can print a message like "Checking all files, this may take a while ..." and a counter, users would see the numbers changing and know that something is happening.

I think a rising counter is good because printing "*" might lead users to think that something is wrong after a while and that an indefinite amout of "*" will be printed.  A counter looks more like things are under control :)

Comment 15 Daniel Walsh 2013-07-25 14:32:25 UTC
# restorecon -Rp /var
71k

Where it counts up from 0 to 71k

If you run it on / it will show percentages.

restorecon -Rp /
11.5%

Comment 16 Fedora End Of Life 2015-01-09 18:22:10 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 17 Fedora End Of Life 2015-02-17 15:30:30 UTC
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.