Bug 1410783

Summary: [Upgrade] database removal failed: ERROR: database "gutterball" is being accessed by other users
Product: Red Hat Satellite Reporter: Jitendra Yejare <jyejare>
Component: UpgradesAssignee: Chris Roberts <chrobert>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: high    
Version: 6.1.11CC: bbuckingham, bcourt, egolov, inecas, lpramuk, mbacovsk, shihliu, zhunting
Target Milestone: UnspecifiedKeywords: PrioBumpGSS, Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
URL: https://gitlab.sat.lab.tlv.redhat.com/satellite6/katello-installer/merge_requests/81
Whiteboard:
Fixed In Version: katello-installer-base-3.0.0.70-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-26 10:47:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1364980, 1410795    
Attachments:
Description Flags
Satellite Foreman Debug
none
satellite.log on satellite6.2.7-sp2
none
forman-debug-info from satellite6.1.11 to satellite6.2.7.sp2 none

Description Jitendra Yejare 2017-01-06 13:36:08 UTC
Created attachment 1237966 [details]
Satellite Foreman Debug

Description of problem:
**Satellite upgrade from 6.1.11 to 6.2.7 is failed in below step:
Upgrade Step: remove_gutterball...
**And with error:
dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Version-Release number of selected component (if applicable):
  Satellite: 6.2 Snap 7
  OS: redhat
  RELEASE: Red Hat Enterprise Linux Server release 7.3 (Maipo)
  FOREMAN: 1.11.0.64
  RUBY: ruby 2.0.0p648 (2015-12-16) [x86_64-linux]
  PUPPET: 3.8.6
  DENIALS: 471

How reproducible:


Steps to Reproduce:
1. Satellite installed with 6.1.11
2. yum update the OS
3. Enable/set 6.2 snap 7 repo on satellite and other required repos
4. Disable not required repos
5. yum update again with sat repo set in step 3
6. Run upgrade command on satellite

Actual results:
The upgrade command(or whole upgrade) is failing in step 'remove gutterball' and with error 'dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users, DETAIL:  There are 5 other sessions using the database.'

Expected results:
The SAtellite Upgrade command and upgrade should be successful.

Additional info:
OS: RHEL7 (Didnt try on RHEL6)

Comment 6 Liushihui 2017-01-12 14:26:26 UTC
When upgrade from satellite6.1.11 to satellite6.2.7 has the same problem on rhel6.8,  the error info as the following:
"dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Upgrade step remove_gutterball failed. Check logs for more information.
"

Comment 8 Liushihui 2017-01-18 09:26:32 UTC
It still failed to upgrade from satellite6.1.11 to satellite6.2.7-sp2. but the error info is different from it as bug description. please see satellite detail log in attachment satellite.log:

[root@hp-dl2x170g6-01 yum.repos.d]# satellite-installer --scenario satellite --upgrade
Upgrading...
Upgrade Step: stop_services...
.............................
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20150930183738 MigrateContentHosts: migrating ==============================

false

Upgrade Step: remove_nodes_distributors...
MongoDB shell version: 2.6.11
connecting to: pulp_database
WriteResult({ "nRemoved" : 0 })

Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing             Done                                               [100%] [........................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade failed during the installation phase. Fix the error and re-run the upgrade.


[root@hp-dl2x170g6-01 yum.repos.d]# rpm -q satellite
satellite-6.2.7-1.0.el7sat.noarch

[root@hp-dl2x170g6-01 yum.repos.d]# rpm -q katello-installer-base
katello-installer-base-3.0.0.70-1.el7sat.noarch

Comment 9 Liushihui 2017-01-18 09:27:29 UTC
Created attachment 1242096 [details]
satellite.log on satellite6.2.7-sp2

Comment 10 Liushihui 2017-01-18 09:38:29 UTC
Created attachment 1242101 [details]
forman-debug-info from satellite6.1.11 to satellite6.2.7.sp2

Comment 11 Lukas Pramuk 2017-01-18 21:03:38 UTC
VERIFIED.

@satellite-6.2.7-1.0.el7sat.noarch
katello-installer-base-3.0.0.70-1.el7sat.noarch

6.1.11 > 6.2.7

# satellite-installer -S satellite --upgrade
...

Upgrade Step: migrate_candlepin...
Migrating candlepin database
Liquibase Update Successful

Upgrade Step: remove_gutterball...

Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start  tomcat.service

Success!

...

>>> gutterball DB was dropped successfully in preinstall time

Comment 12 Lukas Pramuk 2017-01-19 13:40:03 UTC
Liushihui,

Your issue has different root cause and is tracked in BZ 1414780

Comment 14 errata-xmlrpc 2017-01-26 10:47:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0197