Bug 1480560 - Satellite upgrade to 6.2.11 fails : mv: cannot move ‘/var/lib/qpidd/.qpidd/qls/jrnl/resource_manager/.’ to ‘/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/./.’: Device or resource busy
Satellite upgrade to 6.2.11 fails : mv: cannot move ‘/var/lib/qpidd/.qpidd/ql...
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades (Show other bugs)
6.2.11
Unspecified Unspecified
urgent Severity urgent (vote)
: 6.2.11
: --
Assigned To: Evgeni Golov
Katello QA List
: Triaged
Depends On:
Blocks: 1483411
  Show dependency treegraph
 
Reported: 2017-08-11 07:50 EDT by Francisco Garcia
Modified: 2017-09-12 04:42 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1483411 (view as bug list)
Environment:
Last Closed: 2017-08-11 18:12:56 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
foreman-installer/satellite.log (46.10 KB, text/plain)
2017-08-11 07:52 EDT, Francisco Garcia
no flags Details
patch (2.67 KB, patch)
2017-08-11 15:12 EDT, Vincent S. Cojot
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3148641 None None None 2017-08-11 08:53 EDT

  None (edit)
Description Francisco Garcia 2017-08-11 07:50:24 EDT
Description of problem:

When updating a Satellite 6.2.10 to 6.2.11 the following error appears :

----8<------
Upgrade Step: upgrade_qpid_paths...



Backing up /var/lib/qpidd in case of migration failure
tar: Removing leading `/' from member names


Moving resource_manager
mv: cannot move ‘/var/lib/qpidd/.qpidd/qls/jrnl/resource_manager/.’ to ‘/var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/./.’: Device or resource busy

ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/.’: No such file or directory
----8<------

The qpidd backup contains the following content prior to upgrade :


# tar -tvzf /var/cache/qpid_queue_backup.tar.gz 
drwxr-xr-x qpidd/qpidd       0 2017-08-11 13:23 var/lib/qpidd/
-rw------- qpidd/qpidd   12288 2017-05-25 19:37 var/lib/qpidd/qpidd.sasldb
drwxr-xr-x qpidd/qpidd       0 2016-07-01 22:37 var/lib/qpidd/.qpidd/
-rw-r--r-- qpidd/qpidd       0 2016-07-01 22:37 var/lib/qpidd/.qpidd/lock
drwxr-xr-x qpidd/qpidd       0 2016-07-01 22:37 var/lib/qpidd/.qpidd/pq/
-rw-r--r-- qpidd/qpidd       0 2016-07-01 22:37 var/lib/qpidd/.qpidd/pq/lock
-rw-r--r-- qpidd/qpidd      44 2017-08-11 13:09 var/lib/qpidd/.qpidd/.mbrokerdata
-rw-r--r-- qpidd/qpidd      37 2016-07-01 22:37 var/lib/qpidd/.qpidd/systemId
drwxr-xr-x qpidd/qpidd       0 2017-08-11 13:30 var/lib/qpidd/.qpidd/qls/
drwxr-xr-x qpidd/qpidd       0 2017-08-11 13:09 var/lib/qpidd/.qpidd/qls/dat/
-rw-r----- qpidd/qpidd 10485760 2017-08-11 13:15 var/lib/qpidd/.qpidd/qls/dat/log.0000000001
-rw-r----- qpidd/qpidd     8192 2017-08-11 13:15 var/lib/qpidd/.qpidd/qls/dat/queues.db
-rw-r----- qpidd/qpidd     8192 2016-07-01 22:37 var/lib/qpidd/.qpidd/qls/dat/config.db
-rw-r----- qpidd/qpidd     8192 2016-07-03 21:45 var/lib/qpidd/.qpidd/qls/dat/exchanges.db
-rw-r----- qpidd/qpidd     8192 2016-07-01 22:37 var/lib/qpidd/.qpidd/qls/dat/mappings.db
-rw-r----- qpidd/qpidd     8192 2017-08-11 13:15 var/lib/qpidd/.qpidd/qls/dat/bindings.db
-rw-r----- qpidd/qpidd     8192 2016-07-01 22:37 var/lib/qpidd/.qpidd/qls/dat/general.db
-rw-r----- qpidd/qpidd   303104 2017-08-11 13:15 var/lib/qpidd/.qpidd/qls/dat/__db.001
-rw-r----- qpidd/qpidd    40960 2017-08-11 13:15 var/lib/qpidd/.qpidd/qls/dat/__db.002
-rw-r----- qpidd/qpidd    93592 2017-08-11 13:15 var/lib/qpidd/.qpidd/qls/dat/__db.003
drwxr-xr-x qpidd/qpidd        0 2016-07-01 22:37 var/lib/qpidd/.qpidd/qls/p001/
drwxr-xr-x qpidd/qpidd        0 2016-07-01 22:37 var/lib/qpidd/.qpidd/qls/p001/efp/
drwxr-xr-x qpidd/qpidd        0 2017-08-11 13:30 var/lib/qpidd/.qpidd/qls/p001/efp/2048k/
drwxr-xr-x root/root          0 2017-08-11 13:30 var/lib/qpidd/.qpidd/qls/p001/efp/2048k/in_use/
drwxr-xr-x root/root          0 2017-08-11 13:30 var/lib/qpidd/.qpidd/qls/p001/efp/2048k/returned/
drwxr-xr-x qpidd/qpidd        0 2017-08-11 13:15 var/lib/qpidd/.qpidd/qls/jrnl/
drwxr-xr-x qpidd/qpidd        0 2016-07-01 22:38 var/lib/qpidd/.qpidd/qls/jrnl/resource_manager/
-rw-r--r-- qpidd/qpidd  2101248 2016-07-01 22:38 var/lib/qpidd/.qpidd/qls/jrnl/resource_manager/4a9b4efa-82cb-4494-888e-0af6ce70e857.jrnl
drwxr-xr-x qpidd/qpidd        0 2016-07-01 22:38 var/lib/qpidd/.qpidd/qls/jrnl/celery/
-rw-r--r-- qpidd/qpidd  2101248 2016-07-01 22:38 var/lib/qpidd/.qpidd/qls/jrnl/celery/26deb06b-d74e-45cc-9683-78da2b5cff7d.jrnl
drwxr-xr-x qpidd/qpidd        0 2016-07-01 22:38 var/lib/qpidd/.qpidd/qls/jrnl/pulp.task/
-rw-r--r-- qpidd/qpidd  2101248 2016-07-12 18:09 var/lib/qpidd/.qpidd/qls/jrnl/pulp.task/f1a76267-e4f1-4943-922c-4d1cb63e6aae.jrnl
drwxr-xr-x qpidd/qpidd        0 2017-07-02 17:17 var/lib/qpidd/.qpidd/qls/jrnl/katello_event_queue/
-rw-r--r-- qpidd/qpidd  2101248 2017-08-11 13:10 var/lib/qpidd/.qpidd/qls/jrnl/katello_event_queue/41f67b54-cd7c-4f5f-b21c-e20173643bef.jrnl
drwxr-xr-x root/root          0 2017-08-11 13:30 var/lib/qpidd/.qpidd/qls/jrnl2/




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. katello-service stop
2. yum upgrade -y
3. satellite-installer --scenario satellite --upgrade
Comment 1 Francisco Garcia 2017-08-11 07:52 EDT
Created attachment 1312103 [details]
foreman-installer/satellite.log

Installer log
Comment 2 Pavel Moravec 2017-08-11 08:42:03 EDT
Imho this should be the fix:


--- /usr/share/katello-installer-base/hooks/pre/30-upgrade.rb	2017-08-07 13:49:13.000000000 +0000
+++ /usr/share/katello-installer-base/hooks/pre/30-upgrade.rb.new	2017-08-11 12:38:29.708168909 +0000
@@ -105,6 +105,7 @@ def upgrade_qpid_paths
       next if queue_name == '.' || queue_name == '..'
       puts "Moving #{queue_name}"
       Dir.foreach("#{qpid_linearstore}/jrnl/#{queue_name}") do |jrnlfile|
+        next if jrnlfile == '.' || jrnlfile == '..'
         Kafo::Helpers.execute("mv #{qpid_linearstore}/jrnl/#{queue_name}/#{jrnlfile} #{qpid_linearstore}/p001/efp/2048k/in_use/#{jrnlfile}")
         Kafo::Helpers.execute("ln -s #{qpid_linearstore}/p001/efp/2048k/in_use/#{jrnlfile} #{qpid_linearstore}/jrnl2/#{queue_name}/#{jrnlfile}")
         unless $?.success?


rationale: moving journal files from /var/lib/qpidd/.qpidd/qls/jrnl/resource_manager/ (or any other queue name instead of res.manager) should skip moving "files" '.' and '..' - see the same "next if queue_name .." few lines above.


This fix should work well on not-yet-upgraded systems.


For upgrades hitting this bug:

1) unpack /var/cache/qpid_queue_backup.tar.gz to restore /var/lib/qpidd directory

2) apply the patch above

3) re-run the upgrade
Comment 3 Pavel Moravec 2017-08-11 08:43:14 EDT
Basically *any* upgrade to 6.2.11 is affected.
Comment 5 Evgeni Golov 2017-08-11 09:51:29 EDT
The upgrade won't re-run from the backup:
ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/b5588375-0a01-474b-9edf-ddedc7593275.jrnl’: No such file or directory


That's because the backup already contains the folders the upgrade uses to check if the upgrade was done.
You also have to run:
# rm -rf /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/
# rm -rf /var/lib/qpidd/.qpidd/qls/jrnl2/
Comment 6 Evgeni Golov 2017-08-11 09:56:11 EDT
Scratch the last comment, I was reading the code wrong.
Comment 8 Evgeni Golov 2017-08-11 10:26:29 EDT
The patch posted by Pavel is sadly not enough, there is a mkdir missing.

Ivan and I just came up with the following patch:
https://gitlab.sat.lab.tlv.redhat.com/egolov/katello-installer/commit/e360ac29bd2340f7c9b971aae80ad4bb216ccc25

It seems to fix the upgrade in my  VM.

@mithin, did you not have this error? "ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/b5588375-0a01-474b-9edf-ddedc7593275.jrnl’: No such file or directory"
Comment 10 Vincent S. Cojot 2017-08-11 10:50:44 EDT
Are we sure that's enough?

I had a freshly installed sat6 (6.2.10):

1) I 'yum update' it to 6.2.11
2) I edited /usr/share/katello-installer-base/hooks/pre/30-upgrade.rb and added the relevant line.
3) I ran: satellite-installer --scenario satellite --upgrade

and I got:

Upgrade Step: upgrade_qpid_paths...



Backing up /var/lib/qpidd in case of migration failure
tar: Removing leading `/' from member names


Moving resource_manager

ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/resource_manager/c2229ad4-58d0-41a2-90f9-adfc2e146866.jrnl’: No such file or directory

[root@sat6 ~]#
Comment 11 Vincent S. Cojot 2017-08-11 10:51:44 EDT
Then I ran:
[root@sat6 ~]# rm -rf /var/lib/qpidd/.qpidd/qls/p001/efp/2048k/
[root@sat6 ~]# rm -rf /var/lib/qpidd/.qpidd/qls/jrnl2/
[root@sat6 ~]# satellite-installer --scenario satellite --upgrade
Upgrading...


and got this output:
Backing up /var/lib/qpidd in case of migration failure
tar: Removing leading `/' from member names

mv: cannot stat ‘/var/lib/qpidd/.qpidd/qls/dat’: No such file or directory

Moving resource_manager
Moving celery  

ln: failed to create symbolic link ‘/var/lib/qpidd/.qpidd/qls/jrnl2/celery/24ba7b39-e4e4-4ec7-8d4c-42ffa162a2b6.jrnl’: No such file or directory
Comment 17 Evgeni Golov 2017-08-11 12:15:57 EDT
please see the additional comments from ivan in https://gitlab.sat.lab.tlv.redhat.com/satellite6/katello-installer/merge_requests/103
Comment 18 Ivan Necas 2017-08-11 12:19:42 EDT
Can you provide sosreport for those errors?
Comment 19 Ivan Necas 2017-08-11 12:20:19 EDT
@vincent: Can you provide sosreport for those errors?
Comment 20 Vincent S. Cojot 2017-08-11 12:31:37 EDT
@Ivan: Sure can but I think this was due to me removing /var/lib/qpidd and forgetting to restore the backup (
I'm currently re-running the upgrade and it seems to get further...
Comment 24 Vincent S. Cojot 2017-08-11 15:11:33 EDT
After some attempts, I gave up, restored the previous snapshot of the VM (Sat 6.2.10).
I then;
1) ran yum update
2) applied patch (see below
3) ran the update again.

This time everything worked fine and my sat6 was updated to 6.2.11.

Patch attached.
Comment 25 Vincent S. Cojot 2017-08-11 15:12 EDT
Created attachment 1312233 [details]
patch
Comment 27 errata-xmlrpc 2017-08-11 18:12:56 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2468

Note You need to log in before you can comment on or make changes to this bug.