Bug 2052815

Summary:	dynflow fails with "403 extra bytes after the deserialized object"
Product:	Red Hat Satellite	Reporter:	Devendra Singh <desingh>
Component:	Dynflow	Assignee:	Adam Ruzicka <aruzicka>
Status:	CLOSED ERRATA	QA Contact:	Lukas Pramuk <lpramuk>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	6.11.0	CC:	aruzicka, egolov, ehelms, lpramuk, smallamp
Target Milestone:	6.11.0	Keywords:	Regression, Triaged, UpgradeBlocker, Upgrades
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	tfm-rubygem-dynflow-1.6.4	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-07-05 14:32:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Devendra Singh 2022-02-10 05:49:04 UTC

Description of problem: Satellite installer failed in the upgrade due to some extra bytes seen after the deserialized object


Version-Release number of selected component (if applicable):
7.0 Snap8

How reproducible:
always

Steps to Reproduce:
1. Restore Satellite 6.10.z by using the backup data.
2. Setup the repository to perform the satellite upgrade
3. Run the Satellite upgrade but it got failed in "Procedures::Installer::Upgrade: " upgrade stage.
Running preparation steps required to run the next scenarios
 ================================================================================
 Check if tooling for package locking is installed:                    [32m[1m[OK][0m
 --------------------------------------------------------------------------------
 
 
 Running Migration scripts to Satellite 7.0
 ================================================================================
 Setup repositories:                                                   [33m[1m[SKIPPED][0m                                
 --------------------------------------------------------------------------------
 Unlock packages:                                                      [32m[1m[OK][0m                                     
 --------------------------------------------------------------------------------
 Update package(s) :                                                   [32m[1m[OK][0m                                     
 --------------------------------------------------------------------------------
 Procedures::Installer::Upgrade:                                       [31m[1m[FAIL][0m                                   
 Failed executing LANG=en_US.utf-8 satellite-installer  --disable-system-checks, exit status 6:
  [34m2022-02-09 07:11:14[0m [[32mNOTICE[0m] [[36mroot[0m] Loading installer configuration. This will take some time. 
 [34m2022-02-09 07:11:21[0m [[32mNOTICE[0m] [[36mroot[0m] Running installer with log based terminal output at level NOTICE.
 [34m2022-02-09 07:11:21[0m [[32mNOTICE[0m] [[36mroot[0m] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
 [34m2022-02-09 07:11:28[0m [[33mWARN  [0m] [[36mpre[0m] Skipping system checks.
 [34m2022-02-09 07:11:28[0m [[33mWARN  [0m] [[36mpre[0m] Skipping system checks.
 [34m2022-02-09 07:11:40[0m [[32mNOTICE[0m] [[36mconfigure[0m] Starting system configuration.
 [34m2022-02-09 07:11:56[0m [[32mNOTICE[0m] [[36mconfigure[0m] 250 configuration steps out of 2107 steps complete.
 [34m2022-02-09 07:12:23[0m [[32mNOTICE[0m] [[36mconfigure[0m] 500 configuration steps out of 2111 steps complete.
 [34m2022-02-09 07:12:30[0m [[32mNOTICE[0m] [[36mconfigure[0m] 750 configuration steps out of 2113 steps complete.
 [34m2022-02-09 07:12:42[0m [[32mNOTICE[0m] [[36mconfigure[0m] 1000 configuration steps out of 2118 steps complete.
 [34m2022-02-09 07:12:57[0m [[32mNOTICE[0m] [[36mconfigure[0m] 1250 configuration steps out of 2122 steps complete.
 [34m2022-02-09 07:23:25[0m [[32mNOTICE[0m] [[36mconfigure[0m] 1500 configuration steps out of 2123 steps complete.
 [34m2022-02-09 07:24:34[0m [[32mNOTICE[0m] [[36mconfigure[0m] 1750 configuration steps out of 2123 steps complete.
 [34m2022-02-09 07:25:18[0m [[31mERROR [0m] [[36mconfigure[0m] Systemd start for dynflow-sidekiq@orchestrator failed!
 [34m2022-02-09 07:25:18[0m [[31mERROR [0m] [[36mconfigure[0m] journalctl log for dynflow-sidekiq@orchestrator:
 [34m2022-02-09 07:25:18[0m [[31mERROR [0m] [[36mconfigure[0m] -- Logs begin at Wed 2022-02-09 05:37:37 EST, end at Wed 2022-02-09 07:25:18 EST. --
~                                                                                                                                                               


Actual results:
Installer failed in the satellite upgrade

Expected results:
Installer should be completed successfully.

Additional info:

Comment 3 Evgeni Golov 2022-02-10 08:00:38 UTC

We had this issue upstream (see https://community.theforeman.org/t/foreman-nightly-rpm-pipeline-1222-failed/25712/) and thought we solved it by using https://github.com/theforeman/foreman/pull/8856

Something in the upgrade process seems to still trigger this issue, but it's certainly not the installer.
Reassigning to Dynflow component.

Comment 11 Lukas Pramuk 2022-04-26 05:42:23 UTC

VERIFIED.

@Satellite 6.11.0 Snap17
tfm-rubygem-dynflow-1.6.4-1.el7sat.noarch

by the manual reproducer described in comment#0:

1) Restore to Satellite 6.10.5 a specific backup data

2) Upgrade to 6.11.0 using internal repositories (applied upstream fix for db:migrate blocker BZ#2075519)

# satellite-maintain upgrade run --target-version $ver -y -w repositories-validate,repositories-setup
...

Running Migration scripts to Satellite 6.11
================================================================================
Setup repositories:                                                   [SKIPPED]
--------------------------------------------------------------------------------
Unlock packages:                                                      [OK]
--------------------------------------------------------------------------------
Update package(s) :                                                   [OK]
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [OK]
--------------------------------------------------------------------------------
Execute upgrade:run rake task:                                        [OK]
--------------------------------------------------------------------------------
...
Upgrade finished.

>>> upgrade with the specific backup data finished successfully

Comment 14 errata-xmlrpc 2022-07-05 14:32:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498