Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1364980 - upgrade does not remove gutterball tomcat web application
Summary: upgrade does not remove gutterball tomcat web application
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Upgrades
Version: 6.2.0
Hardware: All
OS: Linux
medium
medium
Target Milestone: Unspecified
Assignee: Chris Roberts
QA Contact: Lukas Pramuk
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On: 1410783
Blocks: 1405535
TreeView+ depends on / blocked
 
Reported: 2016-08-08 10:06 UTC by Pavel Moravec
Modified: 2021-08-30 12:33 UTC (History)
10 users (show)

Fixed In Version: katello-installer-base-3.0.0.69-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1405535 (view as bug list)
Environment:
Last Closed: 2017-01-26 10:41:06 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 16864 0 None None None 2016-10-11 15:26:08 UTC
Red Hat Bugzilla 1375659 0 urgent CLOSED Upgrade to 6.2 does not delete $(fqdn):event queue what can end up in 503 error in WebUI 2021-08-30 12:43:27 UTC
Red Hat Knowledge Base (Solution) 2608341 0 None None None 2016-10-25 19:23:52 UTC
Red Hat Product Errata RHBA-2017:0197 0 normal SHIPPED_LIVE Satellite 6.2.7 Async Bug Release 2017-01-26 15:38:38 UTC

Internal Links: 1375659

Description Pavel Moravec 2016-08-08 10:06:58 UTC 
Description of problem:
Despite (fresh installation of) Sat6.2 has removed gutterball from its deployment, an upgrade from 6.1/6.0 does not remove it. postgres gets gutterball connections, tomcat deploys it as an application,..


Version-Release number of selected component (if applicable):
satellite-installer-6.2.0.11-1.el7sat.noarch
satellite-6.2.0-21.1.el7sat.noarch
katello-installer-base-3.0.0.56-1.el7sat.noarch
foreman-installer-1.11.0.9-1.el7sat.noarch
foreman-installer-katello-3.0.0.56-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. Install 6.1
2. Upgrade to 6.2
3. Check /var/log/tomcat/catalina.* log
4. Check "ps aux | grep gutter"


Actual results:
3. finds:
Aug 02, 2016 8:11:56 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /var/lib/tomcat/webapps/gutterball has finished in 6,969 ms

4. finds:
postgres 30890  0.0  0.0 233452  4312 ?        Ss   10:00   0:00 postgres: gutterball gutterball 127.0.0.1(50250) idle


Expected results:
No gutterball after upgrade to 6.2

Additional info:
Comment 3 Chris Roberts 2016-10-11 15:26:09 UTC 
upstream redmine issue created:

http://projects.theforeman.org/issues/16864
Comment 4 Bryan Kearney 2016-10-13 16:07:40 UTC 
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16864 has been resolved.
Comment 6 Pavel Moravec 2016-10-25 14:10:07 UTC 
I think the fix lacks one important step:

rm -rf /var/lib/tomcat*/webapps/gutterball

(rhel7 has "tomcat" dir, rhel6 has "tomcat6", hence the "tomcat*")

For upgrades from 6.1, that directory remains undeleted, so tomcat still deploys gutterball app despite no gutterball RPM present, no gutterball DB. And that still subscribes to the $(fqdn):event qpid queue and still causes https://access.redhat.com/solutions/2608341 - confirmed in field already.

Bryan, does it make sense to move to NEW? Since the current fix does not fix e.g. the BZ subject :).
Comment 7 Mike McCune 2016-10-25 14:12:13 UTC 
Moving this back to ASSIGNED as we need to remove the webapp as Pavel describes
Comment 8 Pavel Moravec 2016-11-01 06:58:34 UTC 
Hi Toledo,
will the fix contain removal of $(fqdn):event queue from qpid broker as well, such that https://bugzilla.redhat.com/show_bug.cgi?id=1375659 is resolved?

(we know that presence of the queue is the culprit of most problems, thats why I am asking)
Comment 13 Chris Roberts 2016-11-04 15:40:14 UTC 
Upgrade Step: remove_event_queue...
[ INFO 2016-11-02 15:41:03 verbose] Upgrade Step: remove_event_queue...

Upgrade completed!
[ INFO 2016-11-02 15:41:04 verbose] Upgrade completed!
[ INFO 2016-11-02 15:41:04 verbose] All hooks in group post finished

Gutterball DB removed in pre hook:

                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges    
-----------+----------+----------+-------------+-------------+------------------------
 candlepin | postgres | UTF8     | en_US.utf8  | en_US.utf8  | =T/postgres           +
           |          |          |             |             | postgres=CTc/postgres +
           |          |          |             |             | candlepin=CTc/postgres
 foreman   | foreman  | UTF8     | en_US.utf8  | en_US.utf8  | =T/foreman            +
           |          |          |             |             | foreman=CTc/foreman
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | 
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres           +
           |          |          |             |             | postgres=CTc/postgres
(5 rows)


Webapp directory removed

[root@rhel7 ~]# ls /var/lib/tomcat/webapps/
candlepin

event queue removed

[root@rhel7 ~]# qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b "amqps://localhost:5671" -q
Queues
  queue                                                       dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
  ===========================================================================================================================================
  19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:1.0                         Y        Y        0     8      8       0   4.86k    4.86k        1     2
  19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:2.0                         Y        Y        0     4      4       0   2.50k    2.50k        1     2
  2b791125-ea03-435a-930b-e81bcc242a01:1.0                         Y        Y        0     2      2       0    486      486         1     2
  4b11fbdc-bed7-4424-9e7b-b28bf0e87b6b:1.0                         Y        Y        0     2      2       0    486      486         1     2
  6eb77d4f-4ad9-41c1-9502-b22870e9821a:1.0                         Y        Y        0     5      5       0   2.67k    2.67k        1     2
  9772ea6e-d763-492e-8780-b3e5091ebec0:1.0                         Y        Y        0     2      2       0    486      486         1     2
  ac2e209c-25e2-4654-8ef5-16408831a921:1.0                         Y        Y        0     0      0       0      0        0         1     2
  b3a19f42-9660-442e-b476-11746b679ed0:0.0                         Y        Y        0     0      0       0      0        0         1     2
  celery                                                      Y                      0     1      1       0    827      827         2     2
  celeryev.0db6053f-dad5-4076-9dd3-33fe5ca6949b                    Y                 0    27     27       0   22.0k    22.0k        1     2
  d2ae1e15-0f85-483b-89f5-c7860499e801:1.0                         Y        Y        0     8      8       0   4.88k    4.88k        1     2
  d2ae1e15-0f85-483b-89f5-c7860499e801:2.0                         Y        Y        0     4      4       0   2.48k    2.48k        1     2
  e34154dc-19e7-48e4-bf53-cde1092a0cf4:1.0                         Y        Y        0     8      8       0   4.86k    4.86k        1     2
  e34154dc-19e7-48e4-bf53-cde1092a0cf4:2.0                         Y        Y        0     4      4       0   2.50k    2.50k        1     2
  f7afcfdf-610d-4033-8053-5ae5aaa9d9d9:1.0                         Y        Y        0     4      4       0   2.42k    2.42k        1     2
  katello_event_queue                                         Y                      0     0      0       0      0        0         1     2
  pulp.task                                                   Y                      0     0      0       0      0        0         3     1
  reserved_resource_worker-0.com.celery.pidbox       Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-0.com.dq             Y    Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-1.com.celery.pidbox       Y                 0     0      0       0      0        0         1     2
  reserved_resource_worker-1.com.dq             Y    Y                 0     0      0       0      0        0         1     2
  resource_manager                                            Y                      0     0      0       0      0        0         1     2
  resource_manager.com.celery.pidbox                 Y                 0     0      0       0      0        0         1     2
  resource_manager.com.dq                       Y    Y                 0     0      0       0      0        0         1     2

Will fix changes and resubmit PR
Comment 14 Lukas Pramuk 2017-01-16 11:28:27 UTC 
FailedQA.

@satellite-6.2.7-1.0.el7sat.noarch
katello-installer-base-3.0.0.69-1.el7sat.noarch

Doing upgrade 6.1.11 > 6.2.7

# satellite-installer -S satellite --upgrade
...

Upgrade Step: remove_gutterball...
dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Upgrade step remove_gutterball failed. Check logs for more information.


# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event 
  <SAT_FQDN>:event           Y             Y        0     0      0       0      0        0         1     2


It seems that fix from BZ 1382524 (Upgrade step:remove_event_queue) should be (also) run before dropping gutterball db
Comment 15 Lukas Pramuk 2017-01-16 11:34:01 UTC 
Nope. Deleting qpid queue didn't helped. (Not related to remove_event_queue)

# qpid-config --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 del queue $(hostname -f):event --force

# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event 
<empty>

# satellite-installer -S satellite --upgrade
...
Upgrade Step: remove_gutterball...
dropdb: database removal failed: ERROR:  database "gutterball" is being accessed by other users
DETAIL:  There are 5 other sessions using the database.

Upgrade step remove_gutterball failed. Check logs for more information.
Comment 16 Lukas Pramuk 2017-01-16 12:02:35 UTC 
There are 5 connections to the database preventing its drop.

# runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball
 17661 | gutterball | 125594 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38194 | 2017-01-16 06:37:35.375769-05 |                               | 2017-01-16 06:37:35.380252-05 | 2017-01-16 06:37:35.380286-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125595 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38197 | 2017-01-16 06:37:35.376306-05 |                               | 2017-01-16 06:37:35.381788-05 | 2017-01-16 06:37:35.381815-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125596 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38196 | 2017-01-16 06:37:35.376557-05 |                               | 2017-01-16 06:37:35.381747-05 | 2017-01-16 06:37:35.381797-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125597 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38200 | 2017-01-16 06:37:35.382417-05 |                               | 2017-01-16 06:37:35.386034-05 | 2017-01-16 06:37:35.386054-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
 17661 | gutterball | 125598 |    16384 | gutterball |                                       | 127.0.0.1   |                 |       38202 | 2017-01-16 06:37:35.383537-05 |                               | 2017-01-16 06:37:35.387371-05 | 2017-01-16 06:37:35.387393-05 | f       | idle   | SHOW TRANSACTION ISOLATION LEVEL
Comment 17 Lukas Pramuk 2017-01-16 12:22:36 UTC 
These connections are coming out of gutterball webapp, thus:

# rm -rf /var/lib/tomcat/webapps/gutterball

And after a while (5secs or so)

# runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball
<empty>


We better start with removing gutterball webapp and then continue with gutterball DB drop
Comment 19 Pavel Moravec 2017-01-16 12:31:41 UTC 
(In reply to Lukas Pramuk from comment #17)
> These connections are coming out of gutterball webapp, thus:
> 
> # rm -rf /var/lib/tomcat/webapps/gutterball
> 
> And after a while (5secs or so)
> 
> # runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep
> gutterball
> <empty>
> 
> 
> We better start with removing gutterball webapp and then continue with
> gutterball DB drop

I *think* safer variant would be stopping tomcat before removing that dir. Just to ensure the running instance does not cache to memory the bits you delete.

(maybe that is already done in some previous step, then let ignore my comment)
Comment 23 Lukas Pramuk 2017-01-18 21:23:25 UTC 
VERIFIED.

@satellite-6.2.7-1.0.el7sat.noarch
katello-installer-base-3.0.0.70-1.el7sat.noarch

6.1.11 > 6.2.7

# satellite-installer -S satellite --upgrade
...

Upgrade Step: migrate_candlepin...
Migrating candlepin database
Liquibase Update Successful

Upgrade Step: remove_gutterball...

Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start  tomcat.service

Success!

Upgrade Step: migrate_foreman...
true


false

Upgrade Step: Running installer...
...


Upgrade Step: remove_gutterball...
gutterball-1.0.15.0.2-1.el7.noarch

No Match for argument: tfm-rubygem-foreman_gutterball
No Match for argument: tfm-rubygem-hammer_cli_gutterball

================================================================================
 Package        Arch   Version          Repository                         Size
================================================================================
Removing:
 gutterball     noarch 1.0.15.0.2-1.el7 @rhel-7-server-satellite-6.1-rpms 301 k
 gutterball-certs
                noarch 1.0-1            installed                          11 k

Transaction Summary
================================================================================
Remove  2 Packages

Installed size: 312 k
warning: /etc/gutterball/gutterball.conf saved as /etc/gutterball/gutterball.conf.rpmsave

Removed:
  gutterball.noarch 0:1.0.15.0.2-1.el7      gutterball-certs.noarch 0:1.0-1     


removed directory: ‘/var/lib/tomcat/webapps/gutterball’
...

>>> Upgrades finish successfully


# ls /var/lib/tomcat{6,}/webapps/gutterball
ls: cannot access /var/lib/tomcat6/webapps/gutterball: No such file or directory
ls: cannot access /var/lib/tomcat/webapps/gutterball: No such file or directory

>>> gutterball webapp is undeployed


# runuser - postgres -c "psql -l" |grep gutterball
<empty>

>>> gutterball DB is dropped
Comment 25 errata-xmlrpc 2017-01-26 10:41:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0197
Note You need to log in before you can comment on or make changes to this bug.