Description of problem: In the customer environment, data area is pressed by pg_largeobject table. EAP 6 application data seems to remain in the pg_largeobject table. Version-Release number of selected component (if applicable): JON 3.3 How reproducible: Deploy EAP 6 application via JON console. Steps to Reproduce: 1. Deploy application. RHQ console -> Inventory -> Servers -> <Select EAP 6 server> -> Create Child(Right click) -> Deployment -> Next -> <Choose file and Upload> -> Next -> Finish ~~~ Data Info~~~~ rhq=> select id,display_name,display_version,file_name,file_size,package_id,package_bits_id from rhq_package_version; id | display_name | display_version | file_name | file_size | package_id | package_bits_id -------+---------------------+-----------------+----------------------+-----------+------------+----------------- 10001 | test.war | | test.war | 457 | 10001 | 10001 rhq=> select * from RHQ_PACKAGE_BITS; id | bits -------+-------- 10001 | 23995 rhq=> SELECT l.loid FROM pg_largeobject l GROUP BY loid HAVING (EXISTS (SELECT 1 FROM RHQ_PACKAGE_BITS t WHERE t.bits = l.loid)); loid -------- 23995 ~~~~~~ 2. Update application. RHQ console -> Inventory -> Servers -> <Select EAP 6 server> -> EAP -> Deployment -> <Deployed application> -> Content -> New -> UPLOAD NEW PACKAGE -> <Upload file> -> Continue -> Continue ~~~ Data Info~~~~ rhq=> select id,display_name,display_version,file_name,file_size,package_id,package_bits_id from rhq_package_version; id | display_name | display_version | file_name | file_size | package_id | package_bits_id -------+---------------------+-----------------+----------------------+-----------+------------+----------------- 10001 | test.war | | test.war | 457 | 10001 | 10001 10011 | | 1.0 | test.war | 457 | 10001 | 10011 rhq=> select * from RHQ_PACKAGE_BITS; id | bits -------+-------- 10001 | 23995 10011 | 23997 rhq=> SELECT l.loid FROM pg_largeobject l GROUP BY loid HAVING (EXISTS (SELECT 1 FROM RHQ_PACKAGE_BITS t WHERE t.bits = l.loid)); loid -------- 23997 23995 ~~~~~~ 3. Delete application RHQ console -> Inventory -> Servers -> <Select EAP 6 server> -> EAP -> Inventory -> <Select deployed application> -> Delete ~~~ Data Info~~~~ Same as the step 2. It means that those application data did not delete. ~~~~~~ Actual results: There is no way to delete EAP 6 application data. Expected results: Need delete function. Additional info: In the customer environment, there are a lot of data in the RHQ_PACKAGE_BITS table which does not be linked with RHQ_PACKAGE_VERSION table because they repeated work of above redeploy step. I have a workaround* for it. * workaround for postgres Delete orphaned data in RHQ_PACKAGE_BITS table by below query. ~~~ SELECT rhq_package_bits.* INTO tmp_todelete FROM rhq_package_bits LEFT JOIN rhq_package_version ON rhq_package_version.package_bits_id = rhq_package_bits.id WHERE rhq_package_version.package_bits_id IS NULL; SELECT lo_unlink(tmp_todelete.bits) FROM tmp_todelete; DELETE FROM rhq_package_bits WHERE rhq_package_bits.id IN (SELECT id FROM tmp_todelete); DROP TABLE tmp_todelete; COMMIT; ~~
The issue described here is not a bug but intended behavior. As per the steps in the reproduction steps, it is not expected that anything would be actually deleted from a JBoss ON inventory perspective. > 3. Delete application > RHQ console -> Inventory -> Servers -> <Select EAP 6 server> -> EAP -> Inventory -> <Select deployed application> -> Delete This will undeploy the application from EAP and then mark the resource as DELETED. But this does not actually remove the resource or any of its associated histories or data. When a resource is in a deleted state, it is eligible for rediscovery. Meaning that if a resource with the same key is discovered in the future, the deleted resource is just re-activated and all of its histories remain intact and once again accessible. The issue that this bug does raise though is that once a resource is in a deleted state, there is no way to uninventory it later. Meaning that the resource and all of its histories will remain in the database forever. > * workaround for postgres > Delete orphaned data in RHQ_PACKAGE_BITS table by below query. > ~~~ > SELECT rhq_package_bits.* INTO tmp_todelete FROM rhq_package_bits LEFT JOIN > rhq_package_version ON rhq_package_version.package_bits_id = > rhq_package_bits.id WHERE rhq_package_version.package_bits_id IS NULL; > SELECT lo_unlink(tmp_todelete.bits) FROM tmp_todelete; > DELETE FROM rhq_package_bits WHERE rhq_package_bits.id IN (SELECT id FROM > tmp_todelete); > DROP TABLE tmp_todelete; > COMMIT; > ~~ This will not workaround this issue. This query has nothing to do with what is described here. It is not clear how this would change anything as it relates to removing/deleting of a WAR or any other resource. However, from the customer ticket, it is clear that one of the suggested workarounds of using uninventory instead of delete is also broken as it truly does leave behind old package versions. This workaround query would help with that issue but again, that is not related to this. In summary: We need to figure out what we should be doing in the delete/remove scenario to prevent resource history from remaining behind when a deleted/undeployed resource is never coming back. A new BZ will be created for the uninventory issue and once captured will be referenced here. @Biljana, can you take care of that?
... > > In summary: > > We need to figure out what we should be doing in the delete/remove scenario > to prevent resource history from remaining behind when a deleted/undeployed > resource is never coming back. > > A new BZ will be created for the uninventory issue and once captured will be > referenced here. @Biljana, can you take care of that? Larry, newly created bugzilla for above is: https://bugzilla.redhat.com/show_bug.cgi?id=1306602
I am updating the title of this bug to properly reflect the issue. First, although this is a bug, it is being considered as an enhancement due to the fact that this issue has always existing and was a flaw/oversight in the original design of the "DELETED" resource state. This issue highlights the fact that if a user deletes a resource, such as what happens when you "undeploy" a WAR using the JBoss ON UI, the resource along with its associated data is not actually removed. Instead, the data is left intact so that if the resource is later re-added, it will get re-associated with its existing data. For example, if you deploy hello-world.war and then decide to undeploy it, the resource that represents hello-world.war along with its content, metrics, availability, events, operations, etc, will remain in the database. They will not not be visible to the user. Later, if you decide to deploy hello-world.war again -- say version 2 -- and its context/path is the same as the original deployment, a new resource IS NOT added. Instead, the original hello-world.war has its inventory status flipped from DELETED to COMMITTED. This means that once again, it is visible to the user along with all the historic metrics and data. The problem is that if you undeploy hello-world.war and really want to get rid of it, there is not way to do this at the moment. This means that all of the data associated with it will continue to use space in the database. In some cases, this could be several gigabytes of space because all the binary content is also being stored for the doomed resource. To address this, a view or special inventory state filter needs to be added to the existing inventory pages to see "DELETED" resources. An inventory state filter would be similar to the ignored, new, and committed states already available on the inventory view pages. By exposing the deleted resources, a user with manage resource permission could then permanently uninventory the resource and all of its associated data. Please be aware that Bug 1168890 attempted to remove the DELETED inventory state from JBoss ON. I do not think this is the correct action as this has been a long standing function/feature of JBoss ON to allow new versions of existing resources to be deployed and keep a single view of historic data.
Moving to ON_QA as available for test with build: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=534002 http://download.eng.bos.redhat.com/brewroot/packages/org.jboss.on-jboss-on-parent/3.3.0.GA/110/maven/org/jboss/on/jon-server-patch/3.3.0.GA/jon-server-patch-3.3.0.GA.zip maps to ER01 build of JON 3.3.8.
Verified on revision ER01 build of JON 3.3.8. To clarify how it was tested and what is the result: When doing Uninventory of Deployment resource, it removes it from database tables. But the next Autodiscovery Run returns that Deployment back into Inventory and into "rhq_package_version" table. So for complete remove of Deployment, it requires to manually delete it from EAP side via Management CLI. When doing Delete of Deployment resource, it does not remove data from database. So it requires manually run of CLI workaround script and it deletes all data from database.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2017-0285.html