Description of problem: I just upgraded my Satellite from 6.3.5 to 6.4.2. Plenty of errors and warnings there which I'll need to address in other BZs, but just hit a really big UI confusion which meant that I updated ALL packages across all hosts rather than just updating katello-agent. This could potentially be a big problem even with Red Hat's assertion that package updates/changes within major releases shouldn't change APIs, etc. My managers aren't happy. Version-Release number of selected component (if applicable): Satellite 6.4.2 How reproducible: Every single time you use the GUI to manage packages on content hosts. Steps to Reproduce: 1. Log in to GUI and go to Hosts -> Content Hosts 2. Select one or more content hosts 3. Click Select Action -> Manage Packages 4. Enter "katello-agent" in the text field of the "Update Package" pop-up 5. Click on the down arrow next to the "Update" button in the middle of the "Update Package" pop-up to select a method of doing the update 6. Select method (in this case I chose "via Katello Agent") 7. Note the immediate pop-up message to the right which says "Successfully scheduled package update" Actual results: This action updates ALL packages on the content host! Not just the one input into the text box. Expected results: Only update the input package name ("katello-agent" in this case). Additional info: Apparently what I should have done at step 5. is click the "Update" button in the middle of the "Update Package" pop-up itself (but then how do you choose the update method?) and then click on the "Yes" choice which would appear below. Please can you confirm that I'm not going mad and this is bad GUI design. If I've misunderstood how this functionality works please accept my apologies for the tone of this BZ (-:
Note also that this problem functionality left many of my Satellite-subscribed content hosts with incomplete yum transactions due to the bug where the gofer package is part of the update and therefore restarts half way through the update, thus truncating/terminating it before it's finished.
Note also also that I discovered this issue by following https://access.redhat.com/labs/satelliteupgradehelper/ 1. Use current 6.3, target 6.4 2. Agree in the tickybox and click Next 3. Select Connected, No, No, No, No, click Next 4. Step 2.11 is where things go wrong Step 2.11 is my point 6. in the initial bug report above. This did a complete yum update, not an update of the typed in package name.
Ben, Could you provide the foreman debug logs for the server on which you are seeing this happen. Could you also verify the version of katello-agent on the client?
>Additional info: >Apparently what I should have done at step 5. is click the "Update" button in the middle of the "Update Package" pop-up itself (but then how do you choose the >update method?) and then click on the "Yes" choice which would appear below. Also, are you seeing a difference in behavior on clicking update vs update dropdown > via katello agent?
I'm afraid the /var/log/foreman/production.log files have rotated out by this point. Given the size of them at the moment (paritially due to the bug about content hosts mutiply contacting Satellite with every yum transaction, I imagine), I only have logs back to 6 March (I keep current-plus-1-to-14). The Satellite upgrade was on 19 February. I can tell you that in all cases the katello-agent version was katello-agent-3.1.0-2 and is now katello-agent-3.3.5-3. Sorry I can't give you the logs. I also can't tell you if I'm seeing a difference in behaviour between the two methods as I've been banned from pushing any updates from Satellite (manual yum updates from content hosts only) until we have confirmation this bug has been fixed (or similar). Management is very wary of pushing anything at this point.
Hey Ben, We tried reproducing this with a 6.4.2 box and agent katello-agent-3.1.0-2 but were not able to see the issue you describe with the full string katello-agent or any other package name. The filtering there seems to work. Would it be possible for you to test this with a test content host and get the logs? There is one edge case we noticed where something like katello-agent* (appending a star to the package name) does lead to updating all packages. Is it possible that you hit this case?
Hi Samir, It is entirely possible I added a wildcard to the end of the package name, but I don't _think_ I did. Would I be right in thinking that your edge case is also incorrect functionality, though? I will try doing as you ask with a new content host and see if I can get any joy. Hopefully before the end of today (GMT).
Created attachment 1546020 [details] /var/log/foreman/production.log Actions detailed start at 2019-03-20 11:02:21
In an attempt to replicate what I saw before, I set up content host as a Satellite Tools 6.3 content host would have been. However, this time the failure was of a different kind. See below. I created a new VM and did the following: # yum localinstall -y http://{satellite host}/pub/katello-ca-consumer-latest.noarch.rpm # subscription-manager clean # subscription-manager register --org="{org}" --activationkey="RHEL7 Servers" # yum install \ katello-agent-3.1.0-2.el7sat.noarch.rpm \ katello-host-tools-3.1.0-2.el7sat.noarch.rpm \ katello-host-tools-fact-plugin-3.1.0-2.el7sat.noarch.rpm \ gofer-2.7.8-1.el7.noarch.rpm \ python-gofer-2.7.8-1.el7.noarch.rpm \ python-gofer-proton-2.7.8-1.el7.noarch.rpm \ pulp-rpm-handlers-2.13.4.9-1.el7sat.noarch.rpm \ python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch.rpm \ python-pulp-common-2.13.4.16-1.el7sat.noarch.rpm \ python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch.rpm # service goferd start ###################### In Satellite GUI: 1) Go to Content Hosts 2) Search for "installed_package = katello-agent-3.1.0-2.el7sat.noarch" 3) Tickybox select the content host which appears 4) Click "Select Action", choose "Manage Packages" from dropdown 5) Enter "katello-agent" (with no * character) in the text field 6) Click the dropdown arrow on the middle button ("Update") and select "via Katello Agent" 7) Timestamp 11:02:21: Observe "Successfully scheduled package update" message 8) Click "Done" ###################### Note update failure: # rpm -qa | egrep 'katello|gofer|pulp' | sort gofer-2.12.3-1.el7.noarch gofer-2.7.8-1.el7.noarch <-- katello-agent-3.3.5-3.el7sat.noarch katello-ca-consumer-{satellite host}-1.0-5.noarch katello-host-tools-3.1.0-2.el7sat.noarch katello-host-tools-fact-plugin-3.1.0-2.el7sat.noarch python-gofer-2.12.3-1.el7.noarch python-gofer-2.7.8-1.el7.noarch <-- python-gofer-proton-2.12.3-1.el7.noarch python-gofer-proton-2.7.8-1.el7.noarch <-- python-isodate-0.5.0-5.pulp.el7sat.noarch python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch python-pulp-common-2.13.4.16-1.el7sat.noarch python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch yum update now produces: # yum update Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager UIS_Red_Hat_Enterprise_Linux_EPEL_7Server_x86_64 | 2.1 kB 00:00:00 rhel-7-server-extras-rpms | 2.0 kB 00:00:00 rhel-7-server-optional-rpms | 2.0 kB 00:00:00 rhel-7-server-rh-common-rpms | 2.1 kB 00:00:00 rhel-7-server-rhn-tools-rpms | 2.1 kB 00:00:00 rhel-7-server-rpms | 2.0 kB 00:00:00 rhel-7-server-satellite-tools-6.4-rpms | 2.1 kB 00:00:00 rhel-7-server-supplementary-rpms | 2.0 kB 00:00:00 Resolving Dependencies There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help). [Long list of packages to update] Error: Package: python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch (installed) Requires: python-pulp-common = 2.13.4.16 Removing: python-pulp-common-2.13.4.16-1.el7sat.noarch (installed) python-pulp-common = 2.13.4.16-1.el7sat Updated By: python-pulp-common-2.16.4.1-1.el7sat.noarch (rhel-7-server-satellite-tools-6.4-rpms) python-pulp-common = 2.16.4.1-1.el7sat Available: python-pulp-common-2.4.1-0.7.beta.el7sat.noarch (rhel-7-server-rh-common-rpms) python-pulp-common = 2.4.1-0.7.beta.el7sat Error: Package: python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch (installed) Requires: python-pulp-common < 2.13.5 Removing: python-pulp-common-2.13.4.16-1.el7sat.noarch (installed) python-pulp-common = 2.13.4.16-1.el7sat Updated By: python-pulp-common-2.16.4.1-1.el7sat.noarch (rhel-7-server-satellite-tools-6.4-rpms) python-pulp-common = 2.16.4.1-1.el7sat Available: python-pulp-common-2.4.1-0.7.beta.el7sat.noarch (rhel-7-server-rh-common-rpms) python-pulp-common = 2.4.1-0.7.beta.el7sat You could try using --skip-broken to work around the problem ** Found 3 pre-existing rpmdb problem(s), 'yum check' output follows: gofer-2.12.3-1.el7.noarch is a duplicate with gofer-2.7.8-1.el7.noarch python-gofer-2.12.3-1.el7.noarch is a duplicate with python-gofer-2.7.8-1.el7.noarch python-gofer-proton-2.12.3-1.el7.noarch is a duplicate with python-gofer-proton-2.7.8-1.el7.noarch Uploading Enabled Repositories Report Loaded plugins: product-id, subscription-manager This is different to the previous issue, and seems to be more connected with https://bugzilla.redhat.com/show_bug.cgi?id=1653040. I can't seem to get past it to test our bug. Attached is /var/log/foreman/production.log
Are you able to manually remove the older version of the packages? gofer-2.12.3-1.el7.noarch is a duplicate with gofer-2.7.8-1.el7.noarch python-gofer-2.12.3-1.el7.noarch is a duplicate with python-gofer-2.7.8-1.el7.noarch python-gofer-proton-2.12.3-1.el7.noarch is a duplicate with python-gofer-proton-2.7.8-1.el7.noarch Based on the other bug you are tracking, updating goferd on the host manually before updating via satellite should let you move forward.
I removed them fairly straightforwardly: Mar 20 15:24:27 Erased: gofer-2.7.8-1.el7.noarch Mar 20 15:24:27 Erased: python-gofer-proton-2.7.8-1.el7.noarch Mar 20 15:24:27 Erased: python-gofer-2.7.8-1.el7.noarch But saw this: Mar 20 15:24:27 otto systemd[1]: Started Gofer Agent. Mar 20 15:24:27 otto goferd[42857]: [INFO][Thread-1] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/demo Mar 20 15:24:27 otto goferd[42857]: [WARNING][MainThread] gofer.agent.plugin:647 - plugin:demo, DISABLED Mar 20 15:24:27 otto goferd[42857]: [INFO][Thread-2] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/katello Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - plugin:katello, import failed Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - Traceback (most recent call last): Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - File "/usr/lib/python2.7/site-packages/gofer/agent/plugin.py", lin..., in _load Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - plugin.impl = __import__(path, {}, {}, [path.split('.')[-1]]) Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - ImportError: No module named katello.agent.goferd.plugin Mar 20 15:24:27 otto goferd[42857]: [INFO][MainThread] gofer.agent.main:92 - agent started. To deal with the python-pulp* issues I had to do the following: Mar 20 15:26:33 Updated: katello-host-tools-fact-plugin-3.3.5-3.el7sat.noarch Mar 20 15:26:34 Updated: katello-host-tools-3.3.5-3.el7sat.noarch Mar 20 15:28:15 Erased: katello-agent-3.3.5-3.el7sat.noarch Mar 20 15:28:27 Erased: python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch Mar 20 15:28:27 Erased: python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch Mar 20 15:28:27 Erased: python-pulp-common-2.13.4.16-1.el7sat.noarch Mar 20 15:28:42 Installed: katello-agent-3.3.5-3.el7sat.noarch I.e. Remove the katello-agent completely, then remove the python-pulp* packages, and then finally put the agent (version 3.3.5-3) back on again. At that point attempting the update process we've been talking about doesn't result in anything happening at all. Likely because the package is already at the current/latest version. I have no proof or basis for assumption, but I don't think this bug (if it exists) is tickled if Satellite knows the content host doesn't need to update anything. I downgraded the katello-agent to 3.1.0-2 again, reinstalling the python-pulp* packages it requires Mar 20 15:40:37 Erased: katello-agent-3.3.5-3.el7sat.noarch Mar 20 15:40:58 Installed: python-pulp-common-2.13.4.16-1.el7sat.noarch Mar 20 15:40:58 Installed: python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch Mar 20 15:40:58 Installed: python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch Mar 20 15:40:58 Installed: pulp-rpm-handlers-2.13.4.9-1.el7sat.noarch Mar 20 15:40:58 Installed: katello-agent-3.1.0-2.el7sat.noarch Reissuing the update process we've been talking about resulted in nothing happening until I restarted the goferd, which I think was in a bit of a state given the katello files changing from under it and an older version of it being erased. Once I'd restarted goferd the update of katello-agent happened almost instantly: # rpm -qa | egrep 'katello|gofer|pulp' | sort gofer-2.12.3-1.el7.noarch katello-agent-3.3.5-3.el7sat.noarch katello-ca-consumer-{satellite host}-1.0-5.noarch katello-host-tools-3.3.5-3.el7sat.noarch katello-host-tools-fact-plugin-3.3.5-3.el7sat.noarch python-gofer-2.12.3-1.el7.noarch python-gofer-proton-2.12.3-1.el7.noarch python-isodate-0.5.0-5.pulp.el7sat.noarch python-pulp-common-2.13.4.16-1.el7sat.noarch ... and none of the other OS packages were touched/updated. So now I'm really confused, even though we haven't really done things with the way my content hosts were installed when I logged this bug. Where do we go from here, please?
>> At that point attempting the update process we've been talking about doesn't result in anything happening at all. Likely because the package is already at the current/latest version. That's correct. Also as you stated, restarting goferd does seem to resolve a lot of issues. >> ... and none of the other OS packages were touched/updated. That is the intended behavior. >> Where do we go from here, please? If you suspect your boxes are still broken, I would suggest running yum check on one of your boxes that had all packages updated from before. Check if you see the duplicate package issues. If you do, we can track that progress here https://bugzilla.redhat.com/show_bug.cgi?id=1653040. If you (fingers crossed) don't have that issue on your hosts, you could try running a package update with the bulk manage page as before on *one* of those boxes and see if everything works as expected. Let us know if you face any issues.
Hi Ben, Based on the discussion, would it be reasonable for us to assume that a wild card was used as mentioned in comment 9? If so, may we use this bugzilla to address that edge case? Thanks for your support and collaboration on this bugzilla.
Honestly, I can't swear to it either way. Obviously I want it to be Satellite's fault as that makes me look like less of a finger-fumbly idiot, but it's entirely likely I added the wildcard character (-: But yes, please do go ahead and use this BZ for the edge case. Just so long as there's also a fix for multi-package updates that include a gofer update not failing at the point gofer is updated and restarted, too!
Thanks Ben!
Hi Ben, One of my teammates (Samir) was taking another look at the wild-card issue (ref: comment 16). It seems that the issue is no longer reproducible in Satellite 6.5 (current release) and based upon his investigation it should be solved starting with katello-agent version 3.3.1. Based upon this, I am going to go ahead and close the bugzilla; however, please feel free to re-open it if you see the same issue with the above versions. Thanks again for raising the bugzilla.