Bug 2227276

Document URL: 

https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.14/html-single/administering_red_hat_satellite/index?lb_target=preview#restoring-satellite-server-or-capsule-from-a-backup_admin

Section Number and Name: 

Chapter 12. Restoring Satellite Server or Capsule Server from a Backup

Describe the issue: 

We support restoring a backup that was taken without pulp content but we don't explain how to recover the pulp content afterwards. 

Sat 6.14 has improved quite a bit now ( the pulp Repair API functionality improvements ) and we can actually get back the Pulp data after such a restoration has been done.



Suggestions for improvement: 

Add a new section called "12.3 Recovering pulp data after a successful restoration" 

This section should be dedicated to the scenario where a backup was taken without pulp data and was restored for a satellite or capsule 6.14 server and user now wants to recover that Pulp data.


## In case of Satellite ##


* For all yum type repos that has an Upstream URL specified:

--> Perform "Complete Sync" 
--> Perform "Validate Content Checksum" action\sync

URL: https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.14/html-single/managing_content/index?lb_target=preview#Recovering_a_Corrupted_Repository_content-management

-> To identify such repos:

# hammer --csv --no-headers repository list --search "content_type = yum" --fields Id,Name,Url,Product | awk -F, '{ if ($4) print $1","$2","$3 ;}'


-> To run the expected action on them via hammer_cli :

IFS=$'\n'; for i in $(hammer --csv --no-headers repository list --search "content_type = yum" --fields Id,Name,Url | awk -F, '{ if ($3) print $1","$2 ;}'); do ID=$(echo $i | cut -d, -f1); REPO=$(echo $i | cut -d, -f2); echo "Working on $REPO"; hammer repository synchronize --id $ID --organization RedHat --skip-metadata-check true; sleep 2; hammer repository synchronize --id $ID --organization RedHat --validate-contents true;echo -e "\n\n";done


------


* For all docker type repos that has an Upstream URL specified:

--> Perform "Validate Content Checksum" action\sync
--> And then simply re-sync the repository.

URL: https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.14/html-single/managing_content/index?lb_target=preview#Recovering_a_Corrupted_Repository_content-management


-> To identify such repos:

# hammer --csv --no-headers repository list --search "content_type = docker" --fields Id,Name,Url,Product | awk -F, '{ if ($4) print $1","$2","$3 ;}'


------


* For all File type repos that has an Upstream URL specified:

--> Perform "Validate Content Checksum" action\sync
--> Perform "Republish Repository Metadata" action ( to regenerate the PULP_MANIFEST ) by following "5.16. Republishing Repository Metadata"
--> And then simply re-sync the repository to confirm nothing is missing.

URL: 
  https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.14/html-single/managing_content/index?lb_target=preview#Recovering_a_Corrupted_Repository_content-management
  https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.14/html-single/managing_content/index?lb_target=preview#Republishing_Repository_Metadata_content-management

-> To identify such repos:

# hammer --csv --no-headers repository list --search "content_type = file" --fields Id,Name,Url,Product | awk -F, '{ if ($4) print $1","$2","$3 ;}'



* For any yum type repos, having no upstream URL specified and the contents were manually managed\uploaded, Re-upload the expected contents in the same way. 

-> To identify such repos:

# hammer --csv --no-headers repository list --search "content_type = yum" --fields Id,Name,Url,Product | awk -F, '{ if (! $4) print $1","$2","$3 ;}'


* For any file type repos, having no upstream URL specified and the contents were manually managed\uploaded, completely delete the repo and recreate it. Once done, reupload the files.

( We perhaps could have fixed it by simply reuploading the file and republishing the CV metadata as well but https://bugzilla.redhat.com/show_bug.cgi?id=2227289 blocks it )

-> To identify such repos:

# hammer --csv --no-headers repository list --search "content_type = file" --fields Id,Name,Url,Product | awk -F, '{ if (! $4) print $1","$2","$3 ;}'



* For the component and composite content-view versions promoted to lifecycle environments, Republish their repository metadata by following "5.17. Republishing Content View Metadata"

https://dxp-docp-prod.apps.ext-waf.spoke.prod.us-west-2.aws.paas.redhat.com/documentation/en-us/red_hat_satellite/6.14/html-single/managing_content/index?lb_target=preview#Republishing_Content_View_Metadata_content-management

(Until https://bugzilla.redhat.com/show_bug.cgi?id=2227271 is fixed, "Republishing Content View Metadata" would not work. So users have to perform a new CV version publish and promote )


## In case of capsules ##

* If any external capsules were restored in a similar manner i.e. without the pulp data then:

  --> Execute the "Pulp Repair" action on the capsule server as explained in the article https://access.redhat.com/solutions/6961905 , under the segment "Can the same repair action be performed in a Red Hat Satellite Capsule 6.10 and above?" and wait for it to complete.
  --> Execute a "Complete Sync" for the same capsule to ensure all repository metadata has been properly republished.


Additional information: 


Please discuss this with Katello and Pulp SMEs and feel free to let me know if any further information is needed from my end.