Bug 1524279

Summary:	[RFE] Foreman-maintain should provide an option to clean its cache as we store previously completed checks.
Product:	Red Hat Satellite	Reporter:	Ashish Humbe <ahumbe>
Component:	Satellite Maintain	Assignee:	Anurag Patel <apatel>
Status:	CLOSED WONTFIX	QA Contact:	Nikhil Kathole <nkathole>
Severity:	high	Docs Contact:
Priority:	high
Version:	6.3.0	CC:	ahumbe, apatel, bbuckingham, bkearney, inecas, kgaikwad, mmccune
Target Milestone:	Unspecified	Keywords:	FutureFeature
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-08-01 18:34:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1496794

Description Ashish Humbe 2017-12-11 06:42:40 UTC

Description of problem:
Foreman-maintain should provide an option to clean its cache as we store previously completed checks.

This will be helpful in the situations where upgrade failed in between while running "satellite-installer", if we want to start the upgrade from scratch there is no way we could see. 

The challenge we faced is :

First upgrade attempt failed with below error: 

Upgrade Step: Running installer...
 pulp-manage-db && touch /var/lib/pulp/init.flag returned 70 instead of one of [0]
 /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: change from notrun to 0 failed: pulp-manage-db && touch /var/lib/pulp/init.flag returned 70 instead of one of [0]

While diagnosing issue we flushed iptable rules along with fixing pulp issue and that caused failure with upgrade via foreman maintain, upgrade failed with below error: 



Running Procedures after migrating to Satellite 6.2.z
================================================================================
katello-service start: 
/ No katello service to start                                         [OK]      
--------------------------------------------------------------------------------
Turn off maintenance mode:                                            [FAIL]
Failed executing iptables -D INPUT -j FOREMAN_MAINTAIN, exit status 1:
 iptables: No chain/target/match by that name.
--------------------------------------------------------------------------------
Scenario [Procedures after migrating to Satellite 6.2.z] failed.

The following steps ended up in failing state:

  [maintenance-mode-disable]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="maintenance-mode-disable"


To get the firewall rule created we tried to run the installer with " --phase pre_migrations " but in multiple attempts, it failed with error "Illegal quoting in line 3. while disabling sync plan." 

# foreman-maintain upgrade run  --target-version 6.2.z --whitelist="disk-io" --phase pre_migrations 
Running Procedures before migrating to Satellite 6.2.z
================================================================================
Turn on maintenance mode:                                             [OK]
--------------------------------------------------------------------------------
disable active sync plans: 
/ disabling sync plans                                                [FAIL]    
Illegal quoting in line 3.
--------------------------------------------------------------------------------
Scenario [Procedures before migrating to Satellite 6.2.z] failed.

The following steps ended up in failing state:

  [sync-plans-disable]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="sync-plans-disable"

On the final attempt of executing foreman-maintain it failed with an error and got the message "The upgrade failed and system was restored to pre-upgrade state."

******************************************************************************************
# foreman-maintain upgrade run  --target-version 6.2.z --whitelist="disk-io"
Running Checks before upgrading to Satellite 6.2.z
================================================================================
Skipping pre_upgrade_checks phase as it was already run before.
To enforce to run the phase, use `upgrade run --phase pre_upgrade_checks`

Scenario [Checks before upgrading to Satellite 6.2.z] failed.

The following steps ended up in failing state:

  [disk-io]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="disk-io"



Running Procedures before migrating to Satellite 6.2.z
================================================================================
Turn on maintenance mode:                                             [OK]
--------------------------------------------------------------------------------
disable active sync plans: 
/ disabling sync plans                                                [FAIL]    
Illegal quoting in line 3.
--------------------------------------------------------------------------------
Scenario [Procedures before migrating to Satellite 6.2.z] failed.

The following steps ended up in failing state:

  [sync-plans-disable]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="sync-plans-disable"



Running Procedures after migrating to Satellite 6.2.z
================================================================================
katello-service start: 
| Starting katello services..                                         [OK]      
--------------------------------------------------------------------------------
Turn off maintenance mode:                                            [OK]
--------------------------------------------------------------------------------
re-enable sync plans: 
/ Total 1 sync plans are now enabled.                                 [OK]      
--------------------------------------------------------------------------------


Running Checks after upgrading to Satellite 6.2.z
================================================================================
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using hammer ping:             [OK]
--------------------------------------------------------------------------------


The upgrade failed and system was restored to pre-upgrade state.
******************************************************************************************

In the next execution of "foreman-maintain upgrade run  --target-version 6.2.z --whitelist="disk-io"  " upgrade was completed successfully without any problem, there were no extra steps/commands executed in between earlier and this successful attempt.

Comment 3 Anurag Patel 2017-12-15 12:09:34 UTC

Discussed this issue with Ashish. This BZ can be broken down into the following themes:

1. Cleaning data files - /var/lib/foreman-maintain/data.yml is the file that contains state data. Renaming this file would clear previously completed check state.
2. Turning maintenance mode ON/OFF can be better done by the tool itself. 
3. 'Illegal quoting in line 3.' This seems to be an error (improper escaping?) that we should investigate and fix.

We'll assign BZs.

Comment 4 Ashish Humbe 2017-12-17 06:36:40 UTC

Thank you Anurag,

We can track 1st issue in this bz for other 2 points we have different bz created. 

Turning maintenance mode ON/OFF can be better done by the tool itself.
https://bugzilla.redhat.com/show_bug.cgi?id=1526749

While running foreman-maintain it fails with "Illegal quoting in line 3."
https://bugzilla.redhat.com/show_bug.cgi?id=1523947

Comment 5 Anurag Patel 2018-01-10 13:46:06 UTC

Ashish, would providing clear instructions in docs on how to clear state data by renaming data.yml be sufficient to address point #1?

Comment 6 Ashish Humbe 2018-01-11 08:25:03 UTC

Hi Anurag,

Yeah, I think we should add it in two places:

1. In the formeman-maintain code, maybe yes/no option for the customer to clear old data at "Running Checks before upgrading to Satellite" step or at the end where upgrade fail message is displayed. 

2. In the official product guide.

Many time customers do not read the product guide properly so having an option to clean data.yml will make a good impact. 

What do you think?

Comment 7 Mike McCune 2018-01-29 18:24:07 UTC

Moving to 1.1 Milestone as we are out of time for 1.0

Comment 8 Kavita 2018-04-03 12:38:21 UTC

We can track first point here & second point, will track in other BZ under Documentation.


(In reply to Ashish Humbe from comment #6)
> Hi Anurag,
> 
> Yeah, I think we should add it in two places:
> 
> 1. In the formeman-maintain code, maybe yes/no option for the customer to
> clear old data at "Running Checks before upgrading to Satellite" step or at
> the end where upgrade fail message is displayed. 
> 
> 2. In the official product guide.
> 
> Many time customers do not read the product guide properly so having an
> option to clean data.yml will make a good impact. 
> 
> What do you think?

Comment 9 Bryan Kearney 2019-07-02 18:01:43 UTC

The Satellite Team is attempting to provide an accurate backlog of bugzilla requests which we feel will be resolved in the next few releases. We do not believe this bugzilla will meet that criteria, and have plans to close it out in 1 month. This is not a reflection on the validity of the request, but a reflection of the many priorities for the product. If you have any concerns about this, feel free to contact Red Hat Technical Support or your account team. If we do not hear from you, we will close this bug out. Thank you.

Comment 10 Bryan Kearney 2019-08-01 18:34:58 UTC

Thank you for your interest in Satellite 6. We have evaluated this request, and while we recognize that it is a valid request, we do not expect this to be implemented in the product in the foreseeable future. This is due to other priorities for the product, and not a reflection on the request itself. We are therefore closing this out as WONTFIX. If you have any concerns about this, please do not reopen. Instead, feel free to contact Red Hat Technical Support. Thank you.