Bug 1881181

Summary: Refresh pools job NPE caused by derived pool
Product: [Community] Candlepin Reporter: Chris "Ceiu" Rog <crog>
Component: candlepinAssignee: Chris "Ceiu" Rog <crog>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: urgent    
Version: 2.9CC: crog, nmoumoul, redakkan, skallesh, thsimmon
Target Milestone: ---Keywords: Triaged
Target Release: 2.9   
Hardware: noarch   
OS: Linux   
Whiteboard:
Fixed In Version: candlepin-2.9.30-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1871873 Environment:
Last Closed: 2020-09-22 09:00:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1871873    
Bug Blocks: 1881186    

Description Chris "Ceiu" Rog 2020-09-21 17:35:02 UTC 
+++ This bug was initially created as a clone of Bug #1871873 +++

Description of problem:

Every execution of a refresh pools job for organization 11551121 fails in production due to a null pointer exception (see https://docs.google.com/document/d/1Ka2Ct1MYA8fleSLlI8CnyZVeV7DrFFTbme5lU5J1azs/edit?usp=sharing)

The root cause on most stack traces is elided, possibly because of a JVM optimization. The root cause is included in older stack traces on splunk and appears to indicate that a derived product or its ID is null for either the master pool or an existing pool, as noted in the document above.

Find the stack trace on 2020-AUG-18 in splunk: `candlepin_hosts(env=prod)` job=8a85f99173e8810e0173ee37a4a03d4a | transaction job

Version-Release number of selected component (if applicable): 3.1.16
How reproducible: Always for org 11551121, but reason is unknown.

Note also that this is associated with INC: https://redhat.service-now.com/surl.do?n=INC1397105