Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1735715

Summary: cdn-sync fails with LOB variable no longer valid after subsequent fetch
Product: Red Hat Satellite 5 Reporter: Kenny Tordeurs <ktordeur>
Component: OtherAssignee: Michael Mráka <mmraka>
Status: CLOSED ERRATA QA Contact: Red Hat Satellite QA List <satqe-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 580CC: rdrazny, saime, tlestach
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: spacewalk-backend-2.5.3-178-sat Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-18 02:49:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
patches none

Description Kenny Tordeurs 2019-08-01 10:18:20 UTC
Description of problem:

cdn-sync is failing with:

# cdn-sync -vvv -c rhel-x86_64-server-6
15:12:19 WARNING: Problem with dates in certificate 'CDN-CLIENT-CERT-182c8e782ee0472bb200d1434d1088ab'. Please check validity of this certificate.
15:12:19 WARNING: Problem with dates in certificate 'CDN-CLIENT-CERT-2a087aa10d5c47c0a32b25b49fbc62c2'. Please check validity of this certificate.
...
15:12:41 ======================================
15:12:41 | Channel: rhel-x86_64-server-6
15:12:41 ======================================
15:12:41 Sync of channel started.
15:12:42
15:12:42   Processing repository with URL: https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os
15:12:42 ERROR: LOB variable no longer valid after subsequent fetch
15:12:42
15:12:42   Processing repository with URL: https://cdn.redhat.com/content/dist/rhel/server/6/6.9/x86_64/kickstart
15:12:42 ERROR: LOB variable no longer valid after subsequent fetch
15:12:42
15:12:42   Processing repository with URL: https://cdn.redhat.com/content/dist/rhel/server/6/6.8/x86_64/kickstart
15:12:42 ERROR: LOB variable no longer valid after subsequent fetch
...
~~~

The problem seems to be coming from being unable to read the certs:
~~~
# cdn-sync --cdn-certs
15:10:10 ======================================
15:10:10 | Certificate/Key: CDN-CA-CERT
15:10:10 ======================================
Traceback (most recent call last):
  File "/usr/bin/cdn-sync", line 230, in <module>
    sys.exc_info()[2])
  File "/usr/bin/cdn-sync", line 162, in <module>
    cdnsync.print_cdn_certificates_info(repos=args.show_repos)
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/cdnsync.py", line 832, in print_cdn_certificates_info
    if not verify_certificate_dates(str(key['key'])):
Exception: SYNC ERROR: attempting to display as much information as possible
 LOB variable no longer valid after subsequent fetch
~~~


Add some extra prints to the code:

/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/cdnsync.py

~~~
...
for key in keys:
+   print('this is the key:', key)
+   print('to str:', str(key['key']))
...
~~~

Results in:
~~~
# cdn-sync --cdn-certs
('this is the key:', {'id': 826, 'key': <cx_Oracle.LOB object at 0x342bd50>, 'description': 'CDN-CA-CERT'})
Traceback (most recent call last):
  File "/usr/bin/cdn-sync", line 230, in <module>
    sys.exc_info()[2])
  File "/usr/bin/cdn-sync", line 162, in <module>
    cdnsync.print_cdn_certificates_info(repos=args.show_repos)
  File "/usr/lib/python2.6/site-packages/spacewalk/cdn_tools/cdnsync.py", line 829, in print_cdn_certificates_info
    print('to str:', str(key['key']))
Exception: SYNC ERROR: attempting to display as much information as possible
 LOB variable no longer valid after subsequent fetch
~~~


Satellite uses cx_Oracle-5.1.2-6.el6sat.x86_64
 
According to https://cx-oracle.readthedocs.io/en/latest/releasenotes.html cx_oracle version 6 solved this issue:
~~~
Removed restriction on fetching LOBs across round trips to the database (eliminates error “LOB variable no longer valid after subsequent fetch”).
~~~
 
https://bugzilla.redhat.com/show_bug.cgi?id=1459536
~~~
rhn-satellite-activation fails on satellite with oracle backend.
The issue was temporarily fixed by workaround in bug 1449635.
Preffered fix is to use newer version of cx_Oracle.
 
Version-Release number of selected component (if applicable):
cx_Oracle-5.1.2-6
spacewalk-backend-2.5.3-140
 
CLOSED CANTFIX
~~~
 
Code:  /usr/lib/python2.6/site-packages/spacewalk/cdn_tools/cdnsync.py
 
~~~
...
   @staticmethod
    def _get_cdn_certificate_keys_and_certs():
        h = rhnSQL.prepare("""
            SELECT ck.id, ck.description, ck.key
            FROM rhnCryptoKeyType ckt,
                 rhnCryptoKey ck
            WHERE ckt.label = 'SSL'
              AND ckt.id = ck.crypto_key_type_id
              AND ck.description LIKE 'CDN_%'
              AND ck.org_id is NULL
            ORDER BY ck.description
        """)
        h.execute()
        return h.fetchall_dict() or []
...
~~~


Version-Release number of selected component (if applicable):
satellite-schema-5.8.0.51-1.el6sat.noarch
cx_Oracle-5.1.2-6.el6sat.x86_64
Oracle Release 11.2.0.4

How reproducible:
Unable to reproduce the issue with Oracle Release 12.2.0.1.0

Actual results:
~~~
LOB variable no longer valid after subsequent fetch
~~~

Expected results:
No errors

Additional info:

Comment 2 Kenny Tordeurs 2019-08-02 07:29:25 UTC
I have tested these changes on my instance with Oracle Release 12.2.0.1.0 and was still able to perform cdn-sync and check the certs info.

I put them into patch files for easier usage.

One can apply/revert the patches with:
# patch -d /usr/lib/python2.6/site-packages/spacewalk/cdn_tools/ -p7 < cdnsync.patch 
# patch -d /usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/ -p8 < init_base.patch
# patch -d /usr/lib/python2.6/site-packages/spacewalk/satellite_tools/ -p7 < reposync.patch

Restart the Satellite services:
# rhn-satellite restart

Comment 3 Kenny Tordeurs 2019-08-02 07:31:10 UTC
Created attachment 1599527 [details]
patches

cdnsync.patch  init_base.patch  reposync.patch

Comment 6 Michael Mráka 2019-08-14 11:00:15 UTC
Fixed in upstream spacewalk git by
commit ad8c90e80a4ae1a6393854f0826b33c94c962500
    1735715 - read LOBs explicitly

Comment 9 Radovan Drazny 2019-09-02 08:28:48 UTC
Verified with spacewalk-backend-2.5.3-178.el6sat.noarch using the reproducer from the initial report and using the script from comment #5. cdn-sync --cdn-certs successfully shows the list of certificate keys. cdn-sync was able to sync channels without any issues. 

VERIFIED

Comment 11 errata-xmlrpc 2019-09-18 02:49:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2783