Bug 2150116

Reproducer using whole Satellite (and not communicating with candlepin directly):

0) configure candlepin to accept API requests from cmdline, per https://access.redhat.com/solutions/2955931 - alternativelly, configure OrphanCleanup per https://bugzilla.redhat.com/show_bug.cgi?id=1931027#c110 to be fired every 5 minutes. But invoking the OrphanCleanup from cmdline is more reliable.


1) Create 1 product and 20 repos:

prodNames="CUSTOM__ZOO_"
repoNames="CUSTOM__ZOO_"
products=1
repos=20

for productId in $(seq 1 $products); do
	productName="${prodNames}${productId}"
	hammer product create --name $productName --label $productName --organization-id 1
	for repoId in $(seq 1 $repos); do
		repoName=${repoNames}${productId}-${repoId}
		hammer repository create --product $productName --organization-id 1 --name ${repoName} --content-type=yum --url=https://repos.fedorapeople.org/repos/pulp/pulp/demo_repos/zoo/ --download-policy on_demand &
	done
done

wait


2) Have a Content Host assigned to Default CV / Library - check that:

subscription-manager refresh; subscription-manager repos --list | grep RedHat_CUSTOM__ZOO | sort | nl

returns all 20 repos

3) *Repeatedly* run below to update repos concurrently & call OrphansCleanup afterwards:

prodNames="CUSTOM__ZOO_"
repoNames="CUSTOM__ZOO_"
products=1
repos=20

for arch in noarch "\"\"" x86_64; do
	echo "$(date): concurrently modifying repos"
	for productId in $(seq 1 $products); do
		productName="${prodNames}${productId}"
		for repoId in $(seq 1 $repos); do
			repoName=${repoNames}${productId}-${repoId}
			hammer repository update --product $productName --organization-id 1 --name ${repoName} --arch $arch &
		done
	done
	wait
	echo "$(date): running orphan cleanup"
	sleep 1
	curl -H "Content-Type:application/json" -X POST -u admin:admin -ks https://localhost:23443/candlepin/jobs/schedule/OrphanCleanupJob
	echo
	sleep 2
done


4) Time to time, check on the Content Host if it still sees all 20 custom repos - it takes random number of iterations (5-10) of step 3 to see a repo is missing.

The issue was successfully reproduced in 6.12.0 (candlepin-4.1.15-1.el8sat.noarch) after ~30 runs of 3) from comment#1 - some of the repos disappeared on the content host side, as well as from the cp db:

candlepin=# SELECT product.uuid, product.name, content.uuid, content.name FROM cp2_content content LEFT JOIN (cp2_product_content pc JOIN cp2_products product ON product.uuid = pc.product_uuid) ON pc.content_uuid = content.uuid WHERE content.contenturl LIKE '%ZOO%';
               uuid               |     name      |               uuid               |       name       
----------------------------------+---------------+----------------------------------+------------------
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc6881d7419 | CUSTOM__ZOO_1-11
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc6886d7429 | CUSTOM__ZOO_1-17
                                  |               | 8a818310857d49fb01857dc688af7439 | CUSTOM__ZOO_1-12
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68a0c743a | CUSTOM__ZOO_1-15
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68b17744b | CUSTOM__ZOO_1-6
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68b72745b | CUSTOM__ZOO_1-7
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68bb6746b | CUSTOM__ZOO_1-10
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68bd6747b | CUSTOM__ZOO_1-13
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68c1c748b | CUSTOM__ZOO_1-14
                                  |               | 8a818310857d49fb01857dc68c7f749b | CUSTOM__ZOO_1-16
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68cc2749c | CUSTOM__ZOO_1-5
                                  |               | 8a818310857d49fb01857dc68cfe74ac | CUSTOM__ZOO_1-20
                                  |               | 8a818310857d49fb01857dc68acc744a | CUSTOM__ZOO_1-18
                                  |               | 8a818310857d49fb01857dc68de474ce | CUSTOM__ZOO_1-19
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68e0674cf | CUSTOM__ZOO_1-3
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68e3c74df | CUSTOM__ZOO_1-1
                                  |               | 8a818310857d49fb01857dc68d1c74ad | CUSTOM__ZOO_1-8
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68d5674ae | CUSTOM__ZOO_1-4
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc68dab74be | CUSTOM__ZOO_1-2
 8a818310857d49fb01857dc68e4374e0 | CUSTOM__ZOO_1 | 8a818310857d49fb01857dc3863e6186 | CUSTOM__ZOO_1-9
(20 rows)

On the contrary I was not able to reproduce the issue in 6.12.1 (candlepin-selinux-4.1.19-1.el8sat.noarch) even after 300 runs of the same - content host still shows all the repos, as well as the cp db:
candlepin=# SELECT product.uuid, product.name, content.uuid, content.name FROM cp2_content content LEFT JOIN (cp2_product_content pc JOIN cp2_products product ON product.uuid = pc.product_uuid) ON pc.content_uuid = content.uuid WHERE content.contenturl LIKE '%ZOO%';
               uuid               |     name      |               uuid               |       name       
----------------------------------+---------------+----------------------------------+------------------
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71ab9c26b0 | CUSTOM__ZOO_1-17
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71abdd26dc | CUSTOM__ZOO_1-5
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71b5472708 | CUSTOM__ZOO_1-2
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e719c9a2592 | CUSTOM__ZOO_1-7
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a05925ea | CUSTOM__ZOO_1-8
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a91e266e | CUSTOM__ZOO_1-18
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a2ce2616 | CUSTOM__ZOO_1-15
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e719cb325a8 | CUSTOM__ZOO_1-6
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e719dba25d4 | CUSTOM__ZOO_1-13
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71b9c4271e | CUSTOM__ZOO_1-1
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71b4f426f2 | CUSTOM__ZOO_1-14
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71ab10269a | CUSTOM__ZOO_1-9
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a4612642 | CUSTOM__ZOO_1-11
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e719b43257c | CUSTOM__ZOO_1-12
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a97b2684 | CUSTOM__ZOO_1-20
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a0d22600 | CUSTOM__ZOO_1-3
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a3fd262c | CUSTOM__ZOO_1-4
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e719cf625be | CUSTOM__ZOO_1-10
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71a7f02658 | CUSTOM__ZOO_1-16
 8a8183c3857c8cae01857e71b9c9271f | CUSTOM__ZOO_1 | 8a8183c3857c8cae01857e71abbb26c6 | CUSTOM__ZOO_1-19
(20 rows)

Given this result, I consider this particular reproducer as resolved in 6.12.1, although it may not be the only reproducer of the issue (see https://bugzilla.redhat.com/show_bug.cgi?id=1931027#c112).

Re-verified in 6.12.2 snap 1.0 (candlepin-4.1.19-1.el8sat.noarch) - after 300 runs of 3) from comment#1 the content host still shows all the repos, as well as the cp db.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.12.2 Async Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:0668