Bug 1480560

Created attachment 1312103 [details]
foreman-installer/satellite.log

Installer log

Imho this should be the fix:


--- /usr/share/katello-installer-base/hooks/pre/30-upgrade.rb	2017-08-07 13:49:13.000000000 +0000
+++ /usr/share/katello-installer-base/hooks/pre/30-upgrade.rb.new	2017-08-11 12:38:29.708168909 +0000
@@ -105,6 +105,7 @@ def upgrade_qpid_paths
       next if queue_name == '.' || queue_name == '..'
       puts "Moving #{queue_name}"
       Dir.foreach("#{qpid_linearstore}/jrnl/#{queue_name}") do |jrnlfile|
+        next if jrnlfile == '.' || jrnlfile == '..'
         Kafo::Helpers.execute("mv #{qpid_linearstore}/jrnl/#{queue_name}/#{jrnlfile} #{qpid_linearstore}/p001/efp/2048k/in_use/#{jrnlfile}")
         Kafo::Helpers.execute("ln -s #{qpid_linearstore}/p001/efp/2048k/in_use/#{jrnlfile} #{qpid_linearstore}/jrnl2/#{queue_name}/#{jrnlfile}")
         unless $?.success?


rationale: moving journal files from /var/lib/qpidd/.qpidd/qls/jrnl/resource_manager/ (or any other queue name instead of res.manager) should skip moving "files" '.' and '..' - see the same "next if queue_name .." few lines above.


This fix should work well on not-yet-upgraded systems.


For upgrades hitting this bug:

1) unpack /var/cache/qpid_queue_backup.tar.gz to restore /var/lib/qpidd directory

2) apply the patch above

3) re-run the upgrade

Basically *any* upgrade to 6.2.11 is affected.

The upgrade won't re-run from the backup:
ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/b5588375-0a01-474b-9edf-ddedc7593275.jrnl’: No such file or directory


That's because the backup already contains the folders the upgrade uses to check if the upgrade was done.
You also have to run:
# rm -rf /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/
# rm -rf /var/lib/qpidd/.qpidd/qls/jrnl2/

Scratch the last comment, I was reading the code wrong.

The patch posted by Pavel is sadly not enough, there is a mkdir missing.

Ivan and I just came up with the following patch:
https://gitlab.sat.lab.tlv.redhat.com/egolov/katello-installer/commit/e360ac29bd2340f7c9b971aae80ad4bb216ccc25

It seems to fix the upgrade in my  VM.

@mithin, did you not have this error? "ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/b5588375-0a01-474b-9edf-ddedc7593275.jrnl’: No such file or directory"

Are we sure that's enough?

I had a freshly installed sat6 (6.2.10):

1) I 'yum update' it to 6.2.11
2) I edited /usr/share/katello-installer-base/hooks/pre/30-upgrade.rb and added the relevant line.
3) I ran: satellite-installer --scenario satellite --upgrade

and I got:

Upgrade Step: upgrade_qpid_paths...



Backing up /var/lib/qpidd in case of migration failure
tar: Removing leading `/' from member names


Moving resource_manager

ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/c2229ad4-58d0-41a2-90f9-adfc2e146866.jrnl’: No such file or directory

[root@sat6 ~]#

Then I ran:
[root@sat6 ~]# rm -rf /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/
[root@sat6 ~]# rm -rf /var/lib/qpidd/.qpidd/qls/jrnl2/
[root@sat6 ~]# satellite-installer --scenario satellite --upgrade
Upgrading...


and got this output:
Backing up /var/lib/qpidd in case of migration failure
tar: Removing leading `/' from member names

mv: cannot stat ‘/var/lib/qpidd/.qpidd/qls/dat’: No such file or directory

Moving resource_manager
Moving celery  

ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/celery/24ba7b39-e4e4-4ec7-8d4c-42ffa162a2b6.jrnl’: No such file or directory

please see the additional comments from ivan in https://gitlab.sat.lab.tlv.redhat.com/satellite6/katello-installer/merge_requests/103

Can you provide sosreport for those errors?

@vincent: Can you provide sosreport for those errors?

@Ivan: Sure can but I think this was due to me removing /var/lib/qpidd and forgetting to restore the backup (
I'm currently re-running the upgrade and it seems to get further...

After some attempts, I gave up, restored the previous snapshot of the VM (Sat 6.2.10).
I then;
1) ran yum update
2) applied patch (see below
3) ran the update again.

This time everything worked fine and my sat6 was updated to 6.2.11.

Patch attached.

Created attachment 1312233 [details]
patch

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2468