Bug 1386913

Isn't this the same problem as BZ 1381720 / BZ 1306602? With 1306602 we remove the rows, but 1381720 should solve the issue of unlinking the blob also?

No, BZ 1381720 does not prevent or address this.

What is happening here is that when a new package version is uploaded, the previous version remains fully intact. Not only its blob but its rhq_package_version row along with rhq_package_bits. This means that the actual blob is not orphaned and is not eligible for removal when executing the new operation being introduced in BZ 1381720.

The fact that the package version remains is by design. The intention appears that a user should be able to see all prior versions of the content and then revert to any prior version. However, the UI does not provide that capability. I think the capability remains in the legacy UI and was never moved into SmartGWT.

The result is that as a deployment evolves, the space required in the database continues to grow unbound and the user has no way of cleaning this up if they choose to remove prior versions. In other words:

- they cannot revert;
- they cannot remove prior versions;

Hope that clears it up.

commit 702b1f634366c49c77ed6aeec27a2de73d371118
Merge: 5002add 6b2f422
Author: Josejulio Martínez <finwemartinez>
Date:   Wed Jan 25 23:48:55 2017 -0600

    Merge pull request #287 from rubenvp8510/purgebitsversions
    
    Bug 1386913 - Remove old version from package_bits when deploys a new version

commit 6b2f422f3acc3cfa0b8acb764f8206808ec44a53
Author: Ruben Vargas <ruben.vp8510>
Date:   Thu Jan 12 17:36:03 2017 -0600

    Remove old version from package_bits when deploys a new version

1)It's not clear to me, Could you please provide me steps to reproduce it? Thank you.

2) I'll review, it supposed to clean the pg_largeobject table, this fix always leaves the first deployment intact (and orphaned) but it should remove other versions if they are present. Are all old versions bits are still in pg_largeobject?

1) This issue was found by CLI automation which is doing following:
- deploy first version and retrieveBackingContent
- deploy second version (updateBackingContent) and retrieveBackingContent
Second step fails with exceptions in comment 7.

It might be just timing issue when updateBackingContent finishes too soon and following retrieveBackingContent fails (later invocation of retrieveBackingContent works). Anyway this automaton failure has never been there in previous JON builds. If it's expected that updateBackingContent is not blocking operation and it's necessary to wait before calling retrieveBackingContent I will just update automation to do that.


2) I noticed that it always leaves two latest versions in package_bits. Older versions are removed from package_bits but not from pg_largeobject. Is it possible that pg_largeobject is not cleaned immediately but some time later? I have not checked this. I will test it more later today and let you know.

I can confirm that rhq_package_bits contains only two latest versions. Older versions are removed from rhq_package_bits but not from pg_largeobject.
I can also confirm that orphaned rows from pg_largeobject are cleaned by the new purge job introduced in BZ 1381720

For the problem with pg_largeobject, I verified that the pg_largeobject are still present, I have a PR that I'm still testing, and others can review.


About the issue with retrieveBackingContent, it seems like it's a kind of race condition, the thing is.. when you run retrieveBackingContent immediately after deploys a new version, it takes the old installed version but at some point, the old version updated his bits to null (which is part of this BZ) so it can't retrieve the content.


The solution to the problem could be put a delay between the calls, I don't know if that is a satisfactory solution. WDYT? @loleary

(In reply to Ruben Vargas Palma from comment #11)
> The solution to the problem could be put a delay between the calls, I don't
> know if that is a satisfactory solution. WDYT? @loleary

This won't work as existing users will not be able to update their scripts or client applications. We need to look into why retreiveBackingContent is getting a reference to the old entry. Or perhaps figure out why updateBackingContent is returning before the proper/final id is created.

commit 9fb1b28d4d55b7bc6df79aea481e7f8060ce256e
Merge: 2b98508 f1eb57f
Author: Josejulio Martínez <finwemartinez>
Date:   Tue Feb 7 11:36:32 2017 -0600

    Merge pull request #293 from rubenvp8510/fix-bits-2.0
    
    Bug 1386913 - Newly uploaded content results in previous content vers…

commit f1eb57f27e1683079783dd7cb20c41427793c4a9
Author: Ruben Vargas <ruben.vp8510>
Date:   Sat Feb 4 20:19:44 2017 -0600

    Bug 1386913 - Newly uploaded content results in previous content version being left in database with no way to clean it up

commit ccb3d2722279c0f98888c73f80d5a3a641f73662
Merge: 1e7e638 f2c7ad8
Author: Josejulio Martínez <finwemartinez>
Date:   Thu Feb 9 16:19:43 2017 -0600

    Merge pull request #294 from rubenvp8510/tx-purge-bits
    
    BZ-1386913 Separate transactions for remove bits.

commit f2c7ad823705cc637de80645b4c54b3d43e18bc4
Author: Ruben Vargas <ruben.vp8510>
Date:   Thu Feb 9 11:16:07 2017 -0600

    BZ-1386913
    - Added validations for unlink only on Postgres
    - Put unlink process as part of other transaction
    - One transaction per package.

In the master:

commit 730168bd3f036a10b6e710af90b79a303971d372
Author: Michael Burman <miburman>
Date:   Mon Feb 13 12:44:39 2017 +0200

    [BZ 1386913] After unlinking, close the connection resources

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2017-0285.html