Red Hat Bugzilla – Bug 1364980
upgrade does not remove gutterball tomcat web application
Last modified: 2018-08-31 11:19 EDT
Description of problem: Despite (fresh installation of) Sat6.2 has removed gutterball from its deployment, an upgrade from 6.1/6.0 does not remove it. postgres gets gutterball connections, tomcat deploys it as an application,.. Version-Release number of selected component (if applicable): satellite-installer-6.2.0.11-1.el7sat.noarch satellite-6.2.0-21.1.el7sat.noarch katello-installer-base-3.0.0.56-1.el7sat.noarch foreman-installer-1.11.0.9-1.el7sat.noarch foreman-installer-katello-3.0.0.56-1.el7sat.noarch How reproducible: 100% Steps to Reproduce: 1. Install 6.1 2. Upgrade to 6.2 3. Check /var/log/tomcat/catalina.* log 4. Check "ps aux | grep gutter" Actual results: 3. finds: Aug 02, 2016 8:11:56 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deployment of web application directory /var/lib/tomcat/webapps/gutterball has finished in 6,969 ms 4. finds: postgres 30890 0.0 0.0 233452 4312 ? Ss 10:00 0:00 postgres: gutterball gutterball 127.0.0.1(50250) idle Expected results: No gutterball after upgrade to 6.2 Additional info:
upstream redmine issue created: http://projects.theforeman.org/issues/16864
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16864 has been resolved.
I think the fix lacks one important step: rm -rf /var/lib/tomcat*/webapps/gutterball (rhel7 has "tomcat" dir, rhel6 has "tomcat6", hence the "tomcat*") For upgrades from 6.1, that directory remains undeleted, so tomcat still deploys gutterball app despite no gutterball RPM present, no gutterball DB. And that still subscribes to the $(fqdn):event qpid queue and still causes https://access.redhat.com/solutions/2608341 - confirmed in field already. Bryan, does it make sense to move to NEW? Since the current fix does not fix e.g. the BZ subject :).
Moving this back to ASSIGNED as we need to remove the webapp as Pavel describes
Hi Toledo, will the fix contain removal of $(fqdn):event queue from qpid broker as well, such that https://bugzilla.redhat.com/show_bug.cgi?id=1375659 is resolved? (we know that presence of the queue is the culprit of most problems, thats why I am asking)
Upgrade Step: remove_event_queue... [ INFO 2016-11-02 15:41:03 verbose] Upgrade Step: remove_event_queue... Upgrade completed! [ INFO 2016-11-02 15:41:04 verbose] Upgrade completed! [ INFO 2016-11-02 15:41:04 verbose] All hooks in group post finished Gutterball DB removed in pre hook: List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+------------------------ candlepin | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =T/postgres + | | | | | postgres=CTc/postgres + | | | | | candlepin=CTc/postgres foreman | foreman | UTF8 | en_US.utf8 | en_US.utf8 | =T/foreman + | | | | | foreman=CTc/foreman postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (5 rows) Webapp directory removed [root@rhel7 ~]# ls /var/lib/tomcat/webapps/ candlepin event queue removed [root@rhel7 ~]# qpid-stat --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b "amqps://localhost:5671" -q Queues queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind =========================================================================================================================================== 19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:1.0 Y Y 0 8 8 0 4.86k 4.86k 1 2 19c6bb96-1fb4-4d32-a06c-6f1cd572b8a1:2.0 Y Y 0 4 4 0 2.50k 2.50k 1 2 2b791125-ea03-435a-930b-e81bcc242a01:1.0 Y Y 0 2 2 0 486 486 1 2 4b11fbdc-bed7-4424-9e7b-b28bf0e87b6b:1.0 Y Y 0 2 2 0 486 486 1 2 6eb77d4f-4ad9-41c1-9502-b22870e9821a:1.0 Y Y 0 5 5 0 2.67k 2.67k 1 2 9772ea6e-d763-492e-8780-b3e5091ebec0:1.0 Y Y 0 2 2 0 486 486 1 2 ac2e209c-25e2-4654-8ef5-16408831a921:1.0 Y Y 0 0 0 0 0 0 1 2 b3a19f42-9660-442e-b476-11746b679ed0:0.0 Y Y 0 0 0 0 0 0 1 2 celery Y 0 1 1 0 827 827 2 2 celeryev.0db6053f-dad5-4076-9dd3-33fe5ca6949b Y 0 27 27 0 22.0k 22.0k 1 2 d2ae1e15-0f85-483b-89f5-c7860499e801:1.0 Y Y 0 8 8 0 4.88k 4.88k 1 2 d2ae1e15-0f85-483b-89f5-c7860499e801:2.0 Y Y 0 4 4 0 2.48k 2.48k 1 2 e34154dc-19e7-48e4-bf53-cde1092a0cf4:1.0 Y Y 0 8 8 0 4.86k 4.86k 1 2 e34154dc-19e7-48e4-bf53-cde1092a0cf4:2.0 Y Y 0 4 4 0 2.50k 2.50k 1 2 f7afcfdf-610d-4033-8053-5ae5aaa9d9d9:1.0 Y Y 0 4 4 0 2.42k 2.42k 1 2 katello_event_queue Y 0 0 0 0 0 0 1 2 pulp.task Y 0 0 0 0 0 0 3 1 reserved_resource_worker-0@rhel7.example.com.celery.pidbox Y 0 0 0 0 0 0 1 2 reserved_resource_worker-0@rhel7.example.com.dq Y Y 0 0 0 0 0 0 1 2 reserved_resource_worker-1@rhel7.example.com.celery.pidbox Y 0 0 0 0 0 0 1 2 reserved_resource_worker-1@rhel7.example.com.dq Y Y 0 0 0 0 0 0 1 2 resource_manager Y 0 0 0 0 0 0 1 2 resource_manager@rhel7.example.com.celery.pidbox Y 0 0 0 0 0 0 1 2 resource_manager@rhel7.example.com.dq Y Y 0 0 0 0 0 0 1 2 Will fix changes and resubmit PR
FailedQA. @satellite-6.2.7-1.0.el7sat.noarch katello-installer-base-3.0.0.69-1.el7sat.noarch Doing upgrade 6.1.11 > 6.2.7 # satellite-installer -S satellite --upgrade ... Upgrade Step: remove_gutterball... dropdb: database removal failed: ERROR: database "gutterball" is being accessed by other users DETAIL: There are 5 other sessions using the database. Upgrade step remove_gutterball failed. Check logs for more information. # qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event <SAT_FQDN>:event Y Y 0 0 0 0 0 0 1 2 It seems that fix from BZ 1382524 (Upgrade step:remove_event_queue) should be (also) run before dropping gutterball db
Nope. Deleting qpid queue didn't helped. (Not related to remove_event_queue) # qpid-config --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 del queue $(hostname -f):event --force # qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep :event <empty> # satellite-installer -S satellite --upgrade ... Upgrade Step: remove_gutterball... dropdb: database removal failed: ERROR: database "gutterball" is being accessed by other users DETAIL: There are 5 other sessions using the database. Upgrade step remove_gutterball failed. Check logs for more information.
There are 5 connections to the database preventing its drop. # runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball 17661 | gutterball | 125594 | 16384 | gutterball | | 127.0.0.1 | | 38194 | 2017-01-16 06:37:35.375769-05 | | 2017-01-16 06:37:35.380252-05 | 2017-01-16 06:37:35.380286-05 | f | idle | SHOW TRANSACTION ISOLATION LEVEL 17661 | gutterball | 125595 | 16384 | gutterball | | 127.0.0.1 | | 38197 | 2017-01-16 06:37:35.376306-05 | | 2017-01-16 06:37:35.381788-05 | 2017-01-16 06:37:35.381815-05 | f | idle | SHOW TRANSACTION ISOLATION LEVEL 17661 | gutterball | 125596 | 16384 | gutterball | | 127.0.0.1 | | 38196 | 2017-01-16 06:37:35.376557-05 | | 2017-01-16 06:37:35.381747-05 | 2017-01-16 06:37:35.381797-05 | f | idle | SHOW TRANSACTION ISOLATION LEVEL 17661 | gutterball | 125597 | 16384 | gutterball | | 127.0.0.1 | | 38200 | 2017-01-16 06:37:35.382417-05 | | 2017-01-16 06:37:35.386034-05 | 2017-01-16 06:37:35.386054-05 | f | idle | SHOW TRANSACTION ISOLATION LEVEL 17661 | gutterball | 125598 | 16384 | gutterball | | 127.0.0.1 | | 38202 | 2017-01-16 06:37:35.383537-05 | | 2017-01-16 06:37:35.387371-05 | 2017-01-16 06:37:35.387393-05 | f | idle | SHOW TRANSACTION ISOLATION LEVEL
These connections are coming out of gutterball webapp, thus: # rm -rf /var/lib/tomcat/webapps/gutterball And after a while (5secs or so) # runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep gutterball <empty> We better start with removing gutterball webapp and then continue with gutterball DB drop
(In reply to Lukas Pramuk from comment #17) > These connections are coming out of gutterball webapp, thus: > > # rm -rf /var/lib/tomcat/webapps/gutterball > > And after a while (5secs or so) > > # runuser - postgres -c "psql -c 'select * from pg_stat_activity'" |grep > gutterball > <empty> > > > We better start with removing gutterball webapp and then continue with > gutterball DB drop I *think* safer variant would be stopping tomcat before removing that dir. Just to ensure the running instance does not cache to memory the bits you delete. (maybe that is already done in some previous step, then let ignore my comment)
VERIFIED. @satellite-6.2.7-1.0.el7sat.noarch katello-installer-base-3.0.0.70-1.el7sat.noarch 6.1.11 > 6.2.7 # satellite-installer -S satellite --upgrade ... Upgrade Step: migrate_candlepin... Migrating candlepin database Liquibase Update Successful Upgrade Step: remove_gutterball... Upgrade Step: start_tomcat... Redirecting to /bin/systemctl start tomcat.service Success! Upgrade Step: migrate_foreman... true false Upgrade Step: Running installer... ... Upgrade Step: remove_gutterball... gutterball-1.0.15.0.2-1.el7.noarch No Match for argument: tfm-rubygem-foreman_gutterball No Match for argument: tfm-rubygem-hammer_cli_gutterball ================================================================================ Package Arch Version Repository Size ================================================================================ Removing: gutterball noarch 1.0.15.0.2-1.el7 @rhel-7-server-satellite-6.1-rpms 301 k gutterball-certs noarch 1.0-1 installed 11 k Transaction Summary ================================================================================ Remove 2 Packages Installed size: 312 k warning: /etc/gutterball/gutterball.conf saved as /etc/gutterball/gutterball.conf.rpmsave Removed: gutterball.noarch 0:1.0.15.0.2-1.el7 gutterball-certs.noarch 0:1.0-1 removed directory: ‘/var/lib/tomcat/webapps/gutterball’ ... >>> Upgrades finish successfully # ls /var/lib/tomcat{6,}/webapps/gutterball ls: cannot access /var/lib/tomcat6/webapps/gutterball: No such file or directory ls: cannot access /var/lib/tomcat/webapps/gutterball: No such file or directory >>> gutterball webapp is undeployed # runuser - postgres -c "psql -l" |grep gutterball <empty> >>> gutterball DB is dropped
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0197