Bug 1490070

Summary: API controllers newer than Apipie cache
Product: Red Hat Satellite Reporter: Reartes Guillermo <greartes>
Component: InstallationAssignee: Martin Bacovsky <mbacovsk>
Status: CLOSED ERRATA QA Contact: Jan Hutaƙ <jhutar>
Severity: low Docs Contact:
Priority: unspecified    
Version: 6.2.11CC: bkearney, greartes, inecas, jangerrit.kootstra, jhutar, mbacovsk, michiel.smit, smane
Target Milestone: 6.4.0Keywords: Triaged
Target Release: Unused   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-16 19:30:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
foreman-debug none

Description Reartes Guillermo 2017-09-09 21:18:53 UTC
Created attachment 1324175 [details]
foreman-debug

Description of problem:

On Sat 6.2.11, i was getting the message: "API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache."

I did not realize that this was an error. (maybe the message should be red colored instead of orange.)

Also, i do not know if this breaks something or not.

I had to execute the "apipie:cache" rake task manually and it went away.
While it went away, i did not realize that is was something that i needed to act upon.

I do expect that to be performed by the "satellite-installer" program in te proper order and time.

Thanks in advance.


Version-Release number of selected component (if applicable):
6.2.11

How reproducible:
always

Steps to Reproduce:
1. Install Sat 6.2.x

2. Stop Sat6 services 
   # katello-service stop

3. Upgrade Packages
   # yum update -y

4. Run Sat6 installer in upgrade mode
   # satellite-installer --scenario=satellite --upgrade
   I was getting the message at this point.  

5. Check Services
   # katello-service status
   # systemctl list-units --state failed
 
6. Reboot and check services again 
   # systemctl reboot
   # katello-service status
   # systemctl list-units --state failed


Actual results:

I was getting this message (repeated several times) during the step #4 -> "API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache."
I don't know if the messages does have any particual effect or not on the product.

Expected results:

As it is, i do expect that the "satellite-installer" in "upgrade mode" to actually "upgrade" itself and perform the necessary migrations. 

If that is an actual failure of a step of "satellite-installer", then please adjust the message somewhat. It looks unimportant. (change the color to red?)

Also make sure that the exit code of "satellite-installer" is not 0 when the message is emitted. (i did not test it)


Additional info:

After executing:
# satellite-installer --scenario=satellite --upgrade

I executed (no errrors):
# foreman-rake apipie:cache:index --trace

And then inmediatelly i executed:
# satellite-installer --scenario=satellite --upgrade

The messages were not shown this time.

Comment 2 Martin Bacovsky 2017-09-15 19:38:35 UTC
I tried to reproduce the issue following the steps described. It is not clear what was the original Satellite version so I tested upgrades from 6.2.11 and 6.2.7 to 6.2.11.

I noticed several occurrences of the message during the upgrade from 6.2.7 but in the end both upgrades finished successfully with all Satellite services up and 
'systemctl list-units --state failed' giving empty list.

@greartes, could you please specify what version of Satellite you were upgrading from? What services were not running and what units were listed as failed?

In general the massage is saying that installed controllers are different from what is contained in apipie cache. This is expected situation during the upgrade and it shouldn't have any impact on the upgrade process. However after the upgrade the cache should be in sync which seems it was not in your case.

Looking in the foreman-debug I found out that the cache was generated properly but right after that the installer installed the plugins (satellite.1.log L 4262-4276) which could change the API. It seems the cache was not re-generated after that. I guess this caused the later issues and will be investigating it further in the meantime.

Comment 3 Reartes Guillermo 2017-09-19 20:19:21 UTC
There was no failure that i can remember.

I just noticed the message and always assumed that it was being taken care by the installer on each update.

Also i do not remember on which version it did start. There are only logs.

Does current code in the installer make sure that the apipie cache is regenerated on each upgrade?  If so then the bz is fixed.

If not, then it should be improved somehow. I feel that if i did not perform the apipie cache rebuild manually, it would have remained out of date permanently.

Comment 4 Brad Buckingham 2017-09-20 17:39:56 UTC
Martin, can you confirm if the cache is regenerated by installer during upgrade as mentioned in comment 3?

Comment 5 Martin Bacovsky 2017-09-20 18:24:30 UTC
Brad, the cache is regenerated by installer as mentioned in c#2. I was not able to reproduce the issue, but from the attached logs it seems that the cache was regenerated before installation of the plugins which might have caused change of the API and invalidation of the cache.

I'm looking into what is actually going on during the plugin installation and if it is possible to move the cache regeneration after plugin installation.

Comment 6 Martin Bacovsky 2017-09-22 12:42:12 UTC
I've find possible solution, I'll send PR upstream for further discussion.Changing component to installer as this is not 'Upgrade' specific.

Comment 7 Martin Bacovsky 2017-09-22 12:42:53 UTC
Created redmine issue http://projects.theforeman.org/issues/21072 from this bug

Comment 9 Marek Hulan 2017-09-25 12:06:55 UTC
*** Bug 1479900 has been marked as a duplicate of this bug. ***

Comment 10 Satellite Program 2017-12-14 17:15:19 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/21072 has been resolved.

Comment 11 michiel.smit 2018-02-12 21:06:52 UTC
Still occurs with my 6.2.13 to 6.2.14 upgrade.
The error is displayed 8 times. If you're not aware of the severity of the problem, seeing the error 8 times makes you think that your upgrade is not going well.

So having this checked and fixed **before** upgrading will alleviate stress.

Comment 12 Martin Bacovsky 2018-02-13 00:34:27 UTC
michiel.smit, thanks for the feedback. I agree the reoccuring warnings are disturbing. The fix will make it to 6.4.0 and with regard to low severity there is not a big chance to qualify for 6.2.z or 6.3.z.

Comment 13 ir. Jan Gerrit Kootstra 2018-02-27 09:49:54 UTC
All,

Be aware that this issue does manifest itself during an upgrade from 6.2.14 to 6.3.0.

Red Hat case 02042041

Regards,

Jan Gerrit Kootstra

Comment 16 Bryan Kearney 2018-10-16 19:30:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2927