Bug 2104339

Summary: Failed to upgrade: ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_puppetclasses", "view_environments", :view_environments, :view_puppetclasses] (Foreman::PermissionMissingException)
Product: Red Hat Satellite Reporter: Paul Armstrong <parmstro>
Component: Discovery PluginAssignee: Lukas Zapletal <lzap>
Status: CLOSED ERRATA QA Contact: Griffin Sullivan <gsulliva>
Severity: high Docs Contact:
Priority: high    
Version: 6.11.0CC: addubey, ahumbe, ehelms, jbhatia, lpramuk, lstejska, lzap, osousa, pbadguja, pdwyer, rabajaj, rjerrido, rlavi, saydas, smajumda, vsedmik
Target Milestone: 6.11.1Keywords: PrioBumpGSS, Triaged, Upgrades
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: tfm-rubygem-foreman_discovery-19.0.5 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-27 17:27:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
foreman-installer log from failed attempt none

Description Paul Armstrong 2022-07-06 02:25:12 UTC
Created attachment 1894797 [details]
foreman-installer log from failed attempt

Description of problem:
Upgrade from 6.10.5 to 6.11 fails with foreman restart

ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_puppetclasses", "view_environments", :view_environments, :view_puppetclasses] (Foreman::PermissionMissingException)

Version-Release number of selected component (if applicable):
6.11

How reproducible:
always

Steps to Reproduce:
1. Run satellite upgrade procedures as per documenation on Satellite 6.10.5
2. upgrade fails at configuration step 1795
3. see below

Actual results:
Long ugly failure results.
3 sidekiq processes and 1 pulp process consume near 100% CPU
installation exits as failed

Expected results:
upgrade succeeds.

Additional info:
workaround from https://community.theforeman.org/t/foreman-upgrade-issue-from-2-5-4-to-3-0-1/26567/2 results in successful installation

Comment 1 Paul Armstrong 2022-07-06 02:26:57 UTC
Error occurring here. Seems like we are expecting permissions that have already been removed.

2022-07-05 18:56:17 [DEBUG ] [configure] /Service[foreman]: Starting to evaluate the resource (1795 of 2306)
2022-07-05 18:56:17 [DEBUG ] [configure] Executing: '/bin/systemctl is-active -- foreman'
2022-07-05 18:56:17 [DEBUG ] [configure] Executing: '/bin/systemctl is-enabled -- foreman'
2022-07-05 18:56:17 [DEBUG ] [configure] Executing: '/bin/systemctl show --property=NeedDaemonReload -- foreman'
2022-07-05 18:56:17 [DEBUG ] [configure] Executing: '/bin/systemctl unmask -- foreman'
2022-07-05 18:56:17 [DEBUG ] [configure] Executing: '/bin/systemctl start -- foreman'
2022-07-05 18:56:35 [DEBUG ] [configure] Running journalctl command to get logs for systemd start failure: journalctl -n 50 --since '5 minutes ago' -u foreman --no-pager
2022-07-05 18:56:35 [DEBUG ] [configure] Executing: 'journalctl -n 50 --since '5 minutes ago' -u foreman --no-pager'
2022-07-05 18:56:35 [ERROR ] [configure] Systemd start for foreman failed!
2022-07-05 18:56:35 [ERROR ] [configure] journalctl log for foreman:
2022-07-05 18:56:35 [ERROR ] [configure] -- Logs begin at Mon 2022-06-27 18:10:01 EDT, end at Tue 2022-07-05 18:56:35 EDT. --
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:51:50 sat6.parmstrong.ca foreman[17275]: => Booting Puma
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:51:50 sat6.parmstrong.ca foreman[17275]: => Rails 6.0.4.7 application starting in production
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:51:50 sat6.parmstrong.ca foreman[17275]: => Run `rails server --help` for more startup options
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:52:05 sat6.parmstrong.ca foreman[17275]: Exiting
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:52:05 sat6.parmstrong.ca foreman[17275]: /usr/share/foreman/app/models/role.rb:336:in `permission_records': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: ["view_puppetclasses", "view_environments", :view_environments, :view_puppetclasses] (Foreman::PermissionMissingException)
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:52:05 sat6.parmstrong.ca foreman[17275]: from /usr/share/foreman/app/models/role.rb:172:in `find_for_permission_removal'
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:52:05 sat6.parmstrong.ca foreman[17275]: from /usr/share/foreman/app/models/role.rb:213:in `remove_permissions!'
2022-07-05 18:56:35 [ERROR ] [configure] Jul 05 18:52:05 sat6.parmstrong.ca foreman[17275]: from /usr/share/foreman/app/services/foreman/plugin/role_lock.rb:56:in `block in update_plugin_role_permissions'

Comment 4 soham 2022-07-07 12:56:20 UTC
*** Bug 2104904 has been marked as a duplicate of this bug. ***

Comment 14 Ron Lavi 2022-07-10 09:28:58 UTC
Created redmine issue https://projects.theforeman.org/issues/35188 from this bug

Comment 15 Leos Stejskal 2022-07-11 06:29:49 UTC
I'll take a look at the issues and take care of them

Comment 16 Bryan Kearney 2022-07-11 12:04:50 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35188 has been resolved.

Comment 17 Paul Armstrong 2022-07-13 15:12:54 UTC
Hi All,

I wanted to update the notes. I reverted the workaround mentioned in comment 3. I implemented the one liner from the commit upstream 

https://github.com/theforeman/foreman_discovery/pull/573 

This resolved my problems and my system is now functioning normally.

Please note:
Satellite was unable to define template capsule for subnet managed by the Satellite after upgrade. Capsule automatically picked up the configuration.
Satellite stopped listening on port 8000 so provisioning failed.
These were resolved by re-running installer with:

satellite-installer --scenario=satellite --foreman-proxy-templates=true --foreman-proxy-template-url=http://sat6.parmstrong.ca:8000 --foreman-proxy-http=true

I can open separate BZs for the above as necessary.

Comment 18 Ron Lavi 2022-07-13 15:59:20 UTC
Thanks for the clarification @Paul!
Could you open a new BZ for the second issue?

Comment 19 Paul Armstrong 2022-07-13 17:32:32 UTC
Created BZ
Satellite upgrade fails to set template capsule for satellite managed subnet
https://bugzilla.redhat.com/show_bug.cgi?id=2106864

Comment 20 Griffin Sullivan 2022-07-18 16:05:20 UTC
6.11 upgrades successfully with foreman discovery image and puppet plugin installed.

Steps to Reproduce:
1. On 6.10.5 install and setup discovery environment
2. Upgrade to 6.11.1

Expected Results:
Upgrade runs successfully without puppet permission errors.

Actual Results:
Upgrade runs successfully without puppet permission errors.

Comment 28 errata-xmlrpc 2022-07-27 17:27:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.11.1 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:5742