Bug 1595960

chatting with beav in irc, the following could have happened:
- 1. The content host registered successfully
- 2. fired generate applicability task
- 3. Content host is unregistered before Step 2 above is processed

I was able to reproduce the issue by the following:
1. Register a content host
2. do several yum install/removals - I created a script like so:
#cat testinstall.sh
for i in `seq 1 6`; do
  echo "Installing vim"
  yum install -y vim
  echo "Removing vim"
  yum remove -y vim
done
3. Let the script run for 1 or 2 iterations and make sure that the generate applicability tasks run fine.
4. Stop foreman-tasks and let the script run to completion.
# systemctl stop foreman-tasks
5. Unregister the content host.
6. Start foreman-tasks and let the script run to completion.
# Start foreman tasks and wait until the pending tasks are executed.
systemctl stop foreman-tasks
6. Notice that all the pending regenerate applicability tasks go to Stopped/Warning state with the message in foreman tasks:
undefined method `each' for true:TrueClass

the log should be similar to what is posted in this bug's description.

Note:
The only difference is I see is the status of the foreman tasks:

In my test it went to Stopped/Warning status. In a customer environment, it went to Paused/Error status.  It may very well because of the way I which stop/start foreman-tasks explicitly. Since the trace is matching, fixing this should alleviate the customer issue for sure.

Apologies, in the above step, the tasks ended with warning are all - Package Profile update tasks and not Generate applicability tasks - but the stacktrace looks similar.

Now I am able to reproduce the Generate Applicability error exactly as noted in the customer environment:

1. Stop foreman-tasks
2. Register the client
3. Notice the client is registered. The General applicability task and the Package Profile Update tasks are in planned status.
4. Unregister the client
5. Start foreman-tasks
6. Notice the following:
Generate applicability task goes to Paused/Error
Package Profile Update goes to Stopped/Warning


This simulates the customer environment in which there is a lot of load on foreman-tasks and due to some reason - the client gets provisioned, registered to Satellite and immediately unregistered - probably because of a configuration issue.

*** Bug 1598736 has been marked as a duplicate of this bug. ***

*** Bug 1589791 has been marked as a duplicate of this bug. ***

Reproducer steps:

- have an activation key with a RHEL subscription (any sub works fine, AK itself is just increasing probability of the reproducer)

- in 1st terminal on a Content Host:

while true; do
  for act in enable disable; do
    subscription-manager repos --${act}=rhel-7-server-rpms
  done
  date
  sleep 0.1
done

- 2nd terminal on the C.H.:

while true; do
  subscription-manager register --activationkey=ak_test --force --org=RedHat
  date
  sleep 1
done

- wait a minute or two, and check paused,error tasks with label Actions::Katello::Host::GenerateApplicability


(hint for engineering: why _this_ task gets paused, while profile upload does not? what makes the profile upload task better?)

I can't reproduce this on 6.4.1. So I would suggest closing the BZ as resolved in 6.4.

This issue has been resolved in Satellite 6.4 and is no longer reproducible. 
Closing as CURRENTRELEASE. Please reopen if the issue still occurs in Satellite 6.4 or newer.