Bug 1364980

Summary: upgrade does not remove gutterball tomcat web application
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: UpgradesAssignee: Chris Roberts <chrobert>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.0CC: bbuckingham, bkearney, chrobert, jcallaha, ktordeur, lpramuk, mmccune, pcfe, xdmoon, zhunting
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
URL: http://projects.theforeman.org/issues/16864
Whiteboard:
Fixed In Version: katello-installer-base-3.0.0.69-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1405535 (view as bug list) Environment:
Last Closed: 2017-01-26 10:41:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1410783    
Bug Blocks: 1405535    

Description Pavel Moravec 2016-08-08 10:06:58 UTC 
Description of problem:
Despite (fresh installation of) Sat6.2 has removed gutterball from its deployment, an upgrade from 6.1/6.0 does not remove it. postgres gets gutterball connections, tomcat deploys it as an application,..


Version-Release number of selected component (if applicable):
satellite-installer-6.2.0.11-1.el7sat.noarch
satellite-6.2.0-21.1.el7sat.noarch
katello-installer-base-3.0.0.56-1.el7sat.noarch
foreman-installer-1.11.0.9-1.el7sat.noarch
foreman-installer-katello-3.0.0.56-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. Install 6.1
2. Upgrade to 6.2
3. Check /var/log/tomcat/catalina.* log
4. Check "ps aux | grep gutter"


Actual results:
3. finds:
Aug 02, 2016 8:11:56 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/gutterball has finished in 6,969 ms

4. finds:
postgres 30890  0.0  0.0 233452  4312 ?        Ss   10:00   0:00 postgres: gutterball gutterball 127.0.0.1(50250) idle


Expected results:
No gutterball after upgrade to 6.2

Additional info:
Comment 3 Chris Roberts 2016-10-11 15:26:09 UTC 
upstream redmine issue created:

http://projects.theforeman.org/issues/16864
Comment 4 Bryan Kearney 2016-10-13 16:07:40 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16864 has been resolved.
Comment 6 Pavel Moravec 2016-10-25 14:10:07 UTC 
I think the fix lacks one important step:

rm -rf /var/lib/tomcat*/webapps/gutterball

(rhel7 has "tomcat" dir, rhel6 has "tomcat6", hence the "tomcat*")

For upgrades from 6.1, that directory remains undeleted, so tomcat still deploys gutterball app despite no gutterball RPM present, no gutterball DB. And that still subscribes to the $(fqdn):event qpid queue and still causes https://access.redhat.com/solutions/2608341 - confirmed in field already.

Bryan, does it make sense to move to NEW? Since the current fix does not fix e.g. the BZ subject :).
Comment 7 Mike McCune 2016-10-25 14:12:13 UTC 
Moving this back to ASSIGNED as we need to remove the webapp as Pavel describes
Comment 8 Pavel Moravec 2016-11-01 06:58:34 UTC 
Hi Toledo,
will the fix contain removal of $(fqdn):event queue from qpid broker as well, such that https://bugzilla.redhat.com/show_bug.cgi?id=1375659 is resolved?

(we know that presence of the queue is the culprit of most problems, thats why I am asking)
Comment 13 Chris Roberts 2016-11-04 15:40:14 UTC 
Upgrade Step: remove_event_queue...
[ INFO 2016-11-02 15:41:03 verbose] Upgrade Step: remove_event_queue...

Upgrade completed!
[ INFO 2016-11-02 15:41:04 verbose] Upgrade completed!
[ INFO 2016-11-02 15:41:04 verbose] All hooks in group post finished

Gutterball DB removed in pre hook:

                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges    
-----------+----------+----------+-------------+-------------+------------------------
 candlepin | postgres | UTF8     | en_US.utf8  | en_US.utf8  | =T/postgres           +
           |          |          |             |             | postgres=CTc/postgres +
           |          |          |             |             | candlepin=CTc/postgres
 foreman   | foreman  | UTF8     | en_US.utf8  | en_US.utf8  | =T/foreman            +
           |          |          |             |             | foreman=CTc/foreman
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
(5 rows)


Webapp directory removed

[root@rhel7 ~]# ls /var/lib/tomcat/webapps/
candlepin

event queue removed

[root@rhel7 ~]# qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b "amqps://localhost:5671" -q
Queues
  queue                                                       dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
  ===========================================================================================================================================
  19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:1.0                         Y        Y        0     8      8       0   4.86k    4.86k        1     2
  19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:2.0                         Y        Y        0     4      4       0   2.50k    2.50k        1     2
  2b791125-ea03-435a-930b-e81bcc242a01:1.0                         Y        Y        0     2      2       0    486      486         1     2
  4b11fbdc-bed7-4424-9e7b-b28bf0e87b6b:1.0                         Y        Y        0     2      2       0    486      486         1     2
  6eb77d4f-4ad9-41c1-9502-b22870e9821a:1.0                         Y        Y        0     5      5       0   2.67k    2.67k        1     2
  9772ea6e-d763-492e-8780-b3e5091ebec0:1.0                         Y        Y        0     2      2       0    486      486         1     2
  ac2e209c-25e2-4654-8ef5-16408831a921:1.0                         Y        Y        0     0      0       0      0        0         1     2
  b3a19f42-9660-442e-b476-11746b679ed0:0.0                         Y        Y        0     0      0       0      0        0         1     2
  celery                                                      Y                      0     1      1       0    827      827         2     2
  celeryev.0db6053f-dad5-4076-9dd3-33fe5ca6949b                    Y                 0    27     27       0   22.0k    22.0k        1     2
  d2ae1e15-0f85-483b-89f5-c7860499e801:1.0                         Y        Y        0     8      8       0   4.88k    4.88k        1     2
  d2ae1e15-0f85-483b-89f5-c7860499e801:2.0                         Y        Y        0     4      4       0   2.48k    2.48k        1     2
  e34154dc-19e7-48e4-bf53-cde1092a0cf4:1.0                         Y        Y        0     8      8       0   4.86k    4.86k        1     2
  e34154dc-19e7-48e4-bf53-cde1092a0cf4:2.0                         Y        Y        0     4      4       0   2.50k    2.50k        1     2
  f7afcfdf-610d-4033-8053-5ae5aaa9d9d9:1.0                         Y        Y        0     4      4       0   2.42k    2.42k        1     2
  katello_event_queue                                         Y                      0     0      0       0      0        0         1     2
  pulp.task                                                   Y                      0     0      0       0      0        0         3     1
  reserved_resource_worker-0.com.celery.pidbox       Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-0.com.dq             Y    Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-1.com.celery.pidbox       Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-1.com.dq             Y    Y                 0     0      0       0      0        0         1     2
  resource_manager                                            Y                      0     0      0       0      0        0         1     2
  resource_manager.com.celery.pidbox                 Y                 0     0      0       0      0        0         1     2
  resource_manager.com.dq                       Y    Y                 0     0      0       0      0        0         1     2

Will fix changes and resubmit PR
Comment 14 Lukas Pramuk 2017-01-16 11:28:27 UTC 
FailedQA.

@satellite-6.2.7-1.0.el7sat.noarch
katello-installer-base-3.0.0.69-1.el7sat.noarch

Doing upgrade 6.1.11 > 6.2.7

# satellite-installer -S satellite --upgrade
...

Upgrade Step: remove_gutterball...
dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Upgrade step remove_gutterball failed. Check logs for more information.


# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event 
  <SAT_FQDN>:event           Y             Y        0     0      0       0      0        0         1     2


It seems that fix from BZ 1382524 (Upgrade step:remove_event_queue) should be (also) run before dropping gutterball db
Comment 15 Lukas Pramuk 2017-01-16 11:34:01 UTC 
Nope. Deleting qpid queue didn't helped. (Not related to remove_event_queue)

# qpid-config --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 del queue $(hostname -f):event --force

# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event 
<empty>

# satellite-installer -S satellite --upgrade
...
Upgrade Step: remove_gutterball...
dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Upgrade step remove_gutterball failed. Check logs for more information.
Comment 16 Lukas Pramuk 2017-01-16 12:02:35 UTC 
There are 5 connections to the database preventing its drop.

# runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball
 17661 | gutterball | 125594 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38194 | 2017-01-16 06:37:35.375769-05 |                               | 2017-01-16 06:37:35.380252-05 | 2017-01-16 06:37:35.380286-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125595 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38197 | 2017-01-16 06:37:35.376306-05 |                               | 2017-01-16 06:37:35.381788-05 | 2017-01-16 06:37:35.381815-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125596 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38196 | 2017-01-16 06:37:35.376557-05 |                               | 2017-01-16 06:37:35.381747-05 | 2017-01-16 06:37:35.381797-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125597 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38200 | 2017-01-16 06:37:35.382417-05 |                               | 2017-01-16 06:37:35.386034-05 | 2017-01-16 06:37:35.386054-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125598 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38202 | 2017-01-16 06:37:35.383537-05 |                               | 2017-01-16 06:37:35.387371-05 | 2017-01-16 06:37:35.387393-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
Comment 17 Lukas Pramuk 2017-01-16 12:22:36 UTC 
These connections are coming out of gutterball webapp, thus:

# rm -rf /var/lib/tomcat/webapps/gutterball

And after a while (5secs or so)

# runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball
<empty>


We better start with removing gutterball webapp and then continue with gutterball DB drop
Comment 19 Pavel Moravec 2017-01-16 12:31:41 UTC 
(In reply to Lukas Pramuk from comment #17)
> These connections are coming out of gutterball webapp, thus:
> 
> # rm -rf /var/lib/tomcat/webapps/gutterball
> 
> And after a while (5secs or so)
> 
> # runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep
> gutterball
> <empty>
> 
> 
> We better start with removing gutterball webapp and then continue with
> gutterball DB drop

I *think* safer variant would be stopping tomcat before removing that dir. Just to ensure the running instance does not cache to memory the bits you delete.

(maybe that is already done in some previous step, then let ignore my comment)
Comment 23 Lukas Pramuk 2017-01-18 21:23:25 UTC 
VERIFIED.

@satellite-6.2.7-1.0.el7sat.noarch
katello-installer-base-3.0.0.70-1.el7sat.noarch

6.1.11 > 6.2.7

# satellite-installer -S satellite --upgrade
...

Upgrade Step: migrate_candlepin...
Migrating candlepin database
Liquibase Update Successful

Upgrade Step: remove_gutterball...

Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start  tomcat.service

Success!

Upgrade Step: migrate_foreman...
true


false

Upgrade Step: Running installer...
...


Upgrade Step: remove_gutterball...
gutterball-1.0.15.0.2-1.el7.noarch

No Match for argument: tfm-rubygem-foreman_gutterball
No Match for argument: tfm-rubygem-hammer_cli_gutterball

================================================================================
 Package        Arch   Version          Repository                         Size
================================================================================
Removing:
 gutterball     noarch 1.0.15.0.2-1.el7 @rhel-7-server-satellite-6.1-rpms 301 k
 gutterball-certs
                noarch 1.0-1            installed                          11 k

Transaction Summary
================================================================================
Remove  2 Packages

Installed size: 312 k
warning: /etc/gutterball/gutterball.conf saved as /etc/gutterball/gutterball.conf.rpmsave

Removed:
  gutterball.noarch 0:1.0.15.0.2-1.el7      gutterball-certs.noarch 0:1.0-1     


removed directory: ‘/var/lib/tomcat/webapps/gutterball’
...

>>> Upgrades finish successfully


# ls /var/lib/tomcat{6,}/webapps/gutterball
ls: cannot access /var/lib/tomcat6/webapps/gutterball: No such file or directory
ls: cannot access /var/lib/tomcat/webapps/gutterball: No such file or directory

>>> gutterball webapp is undeployed


# runuser - postgres -c "psql -l" |grep gutterball
<empty>

>>> gutterball DB is dropped
Comment 25 errata-xmlrpc 2017-01-26 10:41:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0197