Bug 1678666

Note also that this problem functionality left many of my Satellite-subscribed content hosts with incomplete yum transactions due to the bug where the gofer package is part of the update and therefore restarts half way through the update, thus truncating/terminating it before it's finished.

Note also also that I discovered this issue by following https://access.redhat.com/labs/satelliteupgradehelper/

1. Use current 6.3, target 6.4
2. Agree in the tickybox and click Next
3. Select Connected, No, No, No, No, click Next
4. Step 2.11 is where things go wrong

Step 2.11 is my point 6. in the initial bug report above.  This did a complete yum update, not an update of the typed in package name.

Ben, Could you provide the foreman debug logs for the server on which you are seeing this happen. Could you also verify the version of katello-agent on the client?

>Additional info:
>Apparently what I should have done at step 5. is click the "Update" button in the middle of the "Update Package" pop-up itself (but then how do you choose the >update method?) and then click on the "Yes" choice which would appear below.

Also, are you seeing a difference in behavior on clicking update vs update dropdown > via katello agent?

I'm afraid the /var/log/foreman/production.log files have rotated out by this point.  Given the size of them at the moment (paritially due to the bug about content hosts mutiply contacting Satellite with every yum transaction, I imagine), I only have logs back to 6 March (I keep current-plus-1-to-14).  The Satellite upgrade was on 19 February.

I can tell you that in all cases the katello-agent version was katello-agent-3.1.0-2 and is now katello-agent-3.3.5-3.  Sorry I can't give you the logs.

I also can't tell you if I'm seeing a difference in behaviour between the two methods as I've been banned from pushing any updates from Satellite (manual yum updates from content hosts only) until we have confirmation this bug has been fixed (or similar).  Management is very wary of pushing anything at this point.

Hey Ben,

We tried reproducing this with a 6.4.2 box and agent katello-agent-3.1.0-2 but were not able to see the issue you describe with the full string katello-agent or any other package name. The filtering there seems to work. Would it be possible for you to test this with a test content host and get the logs?

There is one edge case we noticed where something like katello-agent* (appending a star to the package name) does lead to updating all packages. Is it possible that you hit this case?

Hi Samir,

It is entirely possible I added a wildcard to the end of the package name, but I don't _think_ I did.  Would I be right in thinking that your edge case is also incorrect functionality, though?

I will try doing as you ask with a new content host and see if I can get any joy.  Hopefully before the end of today (GMT).

Created attachment 1546020 [details]
/var/log/foreman/production.log

Actions detailed start at 2019-03-20 11:02:21

In an attempt to replicate what I saw before, I set up content host as a Satellite Tools 6.3 content host would have been.  However, this time the failure was of a different kind.  See below.

I created a new VM and did the following:

# yum localinstall -y http://{satellite host}/pub/katello-ca-consumer-latest.noarch.rpm
# subscription-manager clean
# subscription-manager register --org="{org}" --activationkey="RHEL7 Servers"
# yum install \
katello-agent-3.1.0-2.el7sat.noarch.rpm \
katello-host-tools-3.1.0-2.el7sat.noarch.rpm \
katello-host-tools-fact-plugin-3.1.0-2.el7sat.noarch.rpm \
gofer-2.7.8-1.el7.noarch.rpm \
python-gofer-2.7.8-1.el7.noarch.rpm \
python-gofer-proton-2.7.8-1.el7.noarch.rpm \
pulp-rpm-handlers-2.13.4.9-1.el7sat.noarch.rpm \
python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch.rpm \
python-pulp-common-2.13.4.16-1.el7sat.noarch.rpm \
python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch.rpm
# service goferd start

######################
In Satellite GUI:

1) Go to Content Hosts
2) Search for "installed_package = katello-agent-3.1.0-2.el7sat.noarch"
3) Tickybox select the content host which appears
4) Click "Select Action", choose "Manage Packages" from dropdown
5) Enter "katello-agent" (with no * character) in the text field
6) Click the dropdown arrow on the middle button ("Update") and select "via
Katello Agent"
7) Timestamp 11:02:21: Observe "Successfully scheduled package update" message
8) Click "Done"

######################
Note update failure:

# rpm -qa | egrep 'katello|gofer|pulp' | sort
gofer-2.12.3-1.el7.noarch
gofer-2.7.8-1.el7.noarch <--
katello-agent-3.3.5-3.el7sat.noarch
katello-ca-consumer-{satellite host}-1.0-5.noarch
katello-host-tools-3.1.0-2.el7sat.noarch
katello-host-tools-fact-plugin-3.1.0-2.el7sat.noarch
python-gofer-2.12.3-1.el7.noarch
python-gofer-2.7.8-1.el7.noarch <--
python-gofer-proton-2.12.3-1.el7.noarch
python-gofer-proton-2.7.8-1.el7.noarch <--
python-isodate-0.5.0-5.pulp.el7sat.noarch
python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch
python-pulp-common-2.13.4.16-1.el7sat.noarch
python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch

yum update now produces:

# yum update
Loaded plugins: enabled_repos_upload, package_upload, product-id, search-disabled-repos, subscription-manager
UIS_Red_Hat_Enterprise_Linux_EPEL_7Server_x86_64                                                                                           | 2.1 kB  00:00:00     
rhel-7-server-extras-rpms                                                                                                                  | 2.0 kB  00:00:00     
rhel-7-server-optional-rpms                                                                                                                | 2.0 kB  00:00:00     
rhel-7-server-rh-common-rpms                                                                                                               | 2.1 kB  00:00:00     
rhel-7-server-rhn-tools-rpms                                                                                                               | 2.1 kB  00:00:00     
rhel-7-server-rpms                                                                                                                         | 2.0 kB  00:00:00     
rhel-7-server-satellite-tools-6.4-rpms                                                                                                     | 2.1 kB  00:00:00     
rhel-7-server-supplementary-rpms                                                                                                           | 2.0 kB  00:00:00     
Resolving Dependencies
There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help).
[Long list of packages to update]
Error: Package: python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch (installed)
           Requires: python-pulp-common = 2.13.4.16
           Removing: python-pulp-common-2.13.4.16-1.el7sat.noarch (installed)
               python-pulp-common = 2.13.4.16-1.el7sat
           Updated By: python-pulp-common-2.16.4.1-1.el7sat.noarch (rhel-7-server-satellite-tools-6.4-rpms)
               python-pulp-common = 2.16.4.1-1.el7sat
           Available: python-pulp-common-2.4.1-0.7.beta.el7sat.noarch (rhel-7-server-rh-common-rpms)
               python-pulp-common = 2.4.1-0.7.beta.el7sat
Error: Package: python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch (installed)
           Requires: python-pulp-common < 2.13.5
           Removing: python-pulp-common-2.13.4.16-1.el7sat.noarch (installed)
               python-pulp-common = 2.13.4.16-1.el7sat
           Updated By: python-pulp-common-2.16.4.1-1.el7sat.noarch (rhel-7-server-satellite-tools-6.4-rpms)
               python-pulp-common = 2.16.4.1-1.el7sat
           Available: python-pulp-common-2.4.1-0.7.beta.el7sat.noarch (rhel-7-server-rh-common-rpms)
               python-pulp-common = 2.4.1-0.7.beta.el7sat
 You could try using --skip-broken to work around the problem
** Found 3 pre-existing rpmdb problem(s), 'yum check' output follows:
gofer-2.12.3-1.el7.noarch is a duplicate with gofer-2.7.8-1.el7.noarch
python-gofer-2.12.3-1.el7.noarch is a duplicate with python-gofer-2.7.8-1.el7.noarch
python-gofer-proton-2.12.3-1.el7.noarch is a duplicate with python-gofer-proton-2.7.8-1.el7.noarch
Uploading Enabled Repositories Report
Loaded plugins: product-id, subscription-manager

This is different to the previous issue, and seems to be more connected with https://bugzilla.redhat.com/show_bug.cgi?id=1653040.  I can't seem to get past it to test our bug.  Attached is /var/log/foreman/production.log

Are you able to manually remove the older version of the packages? 

gofer-2.12.3-1.el7.noarch is a duplicate with gofer-2.7.8-1.el7.noarch
python-gofer-2.12.3-1.el7.noarch is a duplicate with python-gofer-2.7.8-1.el7.noarch
python-gofer-proton-2.12.3-1.el7.noarch is a duplicate with python-gofer-proton-2.7.8-1.el7.noarch

Based on the other bug you are tracking, updating goferd on the host manually before updating via satellite should let you move forward.

I removed them fairly straightforwardly:

Mar 20 15:24:27 Erased: gofer-2.7.8-1.el7.noarch
Mar 20 15:24:27 Erased: python-gofer-proton-2.7.8-1.el7.noarch
Mar 20 15:24:27 Erased: python-gofer-2.7.8-1.el7.noarch

But saw this:

Mar 20 15:24:27 otto systemd[1]: Started Gofer Agent.
Mar 20 15:24:27 otto goferd[42857]: [INFO][Thread-1] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/demo
Mar 20 15:24:27 otto goferd[42857]: [WARNING][MainThread] gofer.agent.plugin:647 - plugin:demo, DISABLED
Mar 20 15:24:27 otto goferd[42857]: [INFO][Thread-2] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/katello
Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - plugin:katello, import failed
Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - Traceback (most recent call last):
Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 -   File "/usr/lib/python2.7/site-packages/gofer/agent/plugin.py", lin..., in _load
Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 -     plugin.impl = __import__(path, {}, {}, [path.split('.')[-1]])
Mar 20 15:24:27 otto goferd[42857]: [ERROR][MainThread] gofer.agent.plugin:703 - ImportError: No module named katello.agent.goferd.plugin
Mar 20 15:24:27 otto goferd[42857]: [INFO][MainThread] gofer.agent.main:92 - agent started.

To deal with the python-pulp* issues I had to do the following:

Mar 20 15:26:33 Updated: katello-host-tools-fact-plugin-3.3.5-3.el7sat.noarch
Mar 20 15:26:34 Updated: katello-host-tools-3.3.5-3.el7sat.noarch
Mar 20 15:28:15 Erased: katello-agent-3.3.5-3.el7sat.noarch
Mar 20 15:28:27 Erased: python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch
Mar 20 15:28:27 Erased: python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch
Mar 20 15:28:27 Erased: python-pulp-common-2.13.4.16-1.el7sat.noarch
Mar 20 15:28:42 Installed: katello-agent-3.3.5-3.el7sat.noarch

I.e. Remove the katello-agent completely, then remove the python-pulp* packages, and then finally put the agent (version 3.3.5-3) back on again.

At that point attempting the update process we've been talking about doesn't result in anything happening at all.  Likely because the package is already at the current/latest version.  I have no proof or basis for assumption, but I don't think this bug (if it exists) is tickled if Satellite knows the content host doesn't need to update anything.  I downgraded the katello-agent to 3.1.0-2 again, reinstalling the python-pulp* packages it requires

Mar 20 15:40:37 Erased: katello-agent-3.3.5-3.el7sat.noarch
Mar 20 15:40:58 Installed: python-pulp-common-2.13.4.16-1.el7sat.noarch
Mar 20 15:40:58 Installed: python-pulp-agent-lib-2.13.4.16-1.el7sat.noarch
Mar 20 15:40:58 Installed: python-pulp-rpm-common-2.13.4.9-1.el7sat.noarch
Mar 20 15:40:58 Installed: pulp-rpm-handlers-2.13.4.9-1.el7sat.noarch
Mar 20 15:40:58 Installed: katello-agent-3.1.0-2.el7sat.noarch

Reissuing the update process we've been talking about resulted in nothing happening until I restarted the goferd, which I think was in a bit of a state given the katello files changing from under it and an older version of it being erased.  Once I'd restarted goferd the update of katello-agent happened almost instantly:

# rpm -qa | egrep 'katello|gofer|pulp' | sort
gofer-2.12.3-1.el7.noarch
katello-agent-3.3.5-3.el7sat.noarch
katello-ca-consumer-{satellite host}-1.0-5.noarch
katello-host-tools-3.3.5-3.el7sat.noarch
katello-host-tools-fact-plugin-3.3.5-3.el7sat.noarch
python-gofer-2.12.3-1.el7.noarch
python-gofer-proton-2.12.3-1.el7.noarch
python-isodate-0.5.0-5.pulp.el7sat.noarch
python-pulp-common-2.13.4.16-1.el7sat.noarch

... and none of the other OS packages were touched/updated.

So now I'm really confused, even though we haven't really done things with the way my content hosts were installed when I logged this bug.  Where do we go from here, please?

>> At that point attempting the update process we've been talking about doesn't result in anything happening at all.  Likely because the package is already at the current/latest version.
That's correct. Also as you stated, restarting goferd does seem to resolve a lot of issues.

>> ... and none of the other OS packages were touched/updated.
That is the intended behavior.

>> Where do we go from here, please?
If you suspect your boxes are still broken, I would suggest running yum check on one of your boxes that had all packages updated from before. Check if you see the duplicate package issues. If you do, we can track that progress here https://bugzilla.redhat.com/show_bug.cgi?id=1653040. 
If you (fingers crossed) don't have that issue on your hosts, you could try running a package update with the bulk manage page as before on *one* of those boxes and see if everything works as expected. Let us know if you face any issues.

Hi Ben,

Based on the discussion, would it be reasonable for us to assume that a wild card was used as mentioned in comment 9?

If so, may we use this bugzilla to address that edge case?

Thanks for your support and collaboration on this bugzilla.

Honestly, I can't swear to it either way.  Obviously I want it to be Satellite's fault as that makes me look like less of a finger-fumbly idiot, but it's entirely likely I added the wildcard character (-:

But yes, please do go ahead and use this BZ for the edge case.  Just so long as there's also a fix for multi-package updates that include a gofer update not failing at the point gofer is updated and restarted, too!

Thanks Ben!

Hi Ben,

One of my teammates (Samir) was taking another look at the wild-card issue (ref: comment 16).  It seems that the issue is no longer reproducible in Satellite 6.5 (current release) and based upon his investigation it should be solved starting with katello-agent version 3.3.1.

Based upon this, I am going to go ahead and close the bugzilla; however, please feel free to re-open it if you see the same issue with the above versions.

Thanks again for raising the bugzilla.