Bug 1926522 - oc adm catalog does not clean temporary files
Summary: oc adm catalog does not clean temporary files
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.6
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Tyler Slaton
QA Contact: Tom Buskey
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-02-08 23:36 UTC by tonyg
Modified: 2022-03-10 16:03 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:02:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 935 0 None open Bug 1926522: oc adm catalog mirror tmp directory cleanup 2021-09-28 18:53:48 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:03:19 UTC

Description tonyg 2021-02-08 23:36:17 UTC
Description of problem:


oc adm catalog mirror does not clean temporary files after run


Version-Release number of selected component (if applicable):


Tested on versions:
    4.4.33
    4.5.30
    4.6.16


How reproducible:

All the time


Steps to Reproduce:

Once the catalog is built the temporary files are not removed from the system. 

To test with the script below the following values need to be defined to use the local registry and the secrets as needed.

CLIENT_DIR=<PATH_TO_OC_VERSIONS>
LOCAL_SECRET_JSON=<PATH_TO_PULL_SECRET>
LOCAL_REGISTRY=<REGISTRY_AND_PORT_ADDRESS>
LOCAL_REPOSITORY=<PATH_TO_LOCAL_REPOSITORY>

OC_VERSIONS=(
    4.4.33
    4.5.30
    4.6.16
)

for ver in ${OC_VERSIONS[@]};  do
    echo ${ver} ===
    {
      ${CLIENT_DIR}/${ver}/oc adm catalog mirror \
        ${LOCAL_REGISTRY}${LOCAL_REPOSITORY}:v${ver%.*} \
        ${LOCAL_REGISTRY} \
        -a ${LOCAL_SECRET_JSON} \
        --insecure \
        --filter-by-os="linux/amd64" \
        --manifests-only \
        --to-manifests=oc_${ver%.*} &>mirror.log
    }
    
    grep -m1 wrote mirror.log |
        grep -oP '/tmp/\d+' |
        xargs du -sh
done


Actual results:


A single directory is created every time the catalog is built taking considerable amount of disk space.

From the example above the results are:

4.4.33 ===
453M    /tmp/488783710
4.5.30 ===
477M    /tmp/064672583
4.6.16 ===
630M    /tmp/265655726


Expected results:

If the temporary files created are not going to be reused they should be deleted to avoid filling the file system.


Additional info:

Comment 7 tonyg 2021-02-24 17:32:12 UTC
Thanks for the fix, is there anything that's needed on my side?

Comment 8 Robin Cernin 2021-02-25 00:39:12 UTC
Hi @tonyg

Question, the current Target release is set to latest. Once this is merged in the latest, if possible do you want me to backporting this into any other versions or are you fine with latest?

Comment 9 tonyg 2021-02-25 16:01:34 UTC
Hi Robin,

I just realized that the temp files may be used by others, this documentation actually suggest that: https://docs.openshift.com/container-platform/4.6/operators/admin/olm-restricted-networks.html#olm-mirror-catalog_olm-restricted-networks

Step 4 mentions:

$ echo "select * from related_image \
    where operatorbundle_name like '%jaeger%';" \
    | sqlite3 -line /tmp/153048078/index.db 


This change will affect those doing that.


With that in mind I'd say to only merge it in the latest version, but that implies that the above won't be possible from that version forward. What would be the procedure to request an additional option to write those files in a specific directory, that way anyone using those files would be able to still access them.

Thanks!

Comment 10 Robin Cernin 2021-02-25 20:54:50 UTC
Hi Tony,

With the change, one needs to add `--path` argument for example:

mkdir <dir>
oc adm catalog mirror --path <dir> \
    <index_image> \
    <mirror_registry>:<port> \
    [-a ${REG_CREDS}] \
    [--insecure] \
    --filter-by-os='.*' \
    [--manifests-only] 

echo "select * from related_image \
    where operatorbundle_name like '%jaeger%';" \
    | sqlite3 -line <dir>/index.db

Comment 11 tonyg 2021-02-25 21:26:11 UTC
Hi Robin,

That's great then in that case I see no problem if it's backported. Thanks again!

Comment 22 Tom Buskey 2021-10-08 15:48:58 UTC
 oc adm catalog mirror quay.io/tbuskey/tpb-operator-index:latest file:///local/index --insecure --manifests-only 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! DEPRECATION NOTICE:
!!   Sqlite-based catalogs are deprecated. Support for them will be removed in a
!!   future release. Please migrate your catalog workflows to the new file-based
!!   catalog format.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

src image has index label for database path: /database/index.db
using index path mapping: /database/index.db:/tmp/540893272
wrote database to /tmp/540893272
using database at: /tmp/540893272/index.db
wrote mirroring manifests to manifests-tpb-operator-index-1633707218

To upload local images to a registry, run:

	oc adm catalog mirror file://local/index/tbuskey/tpb-operator-index:latest REGISTRY/REPOSITORY
deleted dir /tmp/540893272

Comment 26 errata-xmlrpc 2022-03-10 16:02:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.