Bug 1637042

Description of problem:
When sending a virt-who report to Satellite a short time after a previous one (from even different set of hypervisors), Satellite's task Hypervisors fail on

undefined method `[]' for nil:NilClass

error.

I reproduced it on 6.3.4 snap 2, but older/other versions of 6.3 are expected to have the bug as well.


Version-Release number of selected component (if applicable):
Sat 6.3.4 (should be reproducible on older versions as well)
- satellite-6.3.4-2.el7sat.noarch
- tfm-rubygem-katello-3.4.5.86-1.el7sat.noarch
- candlepin-2.1.24-1.el7.noarch


How reproducible:
100% in several attempts


Steps to Reproduce:
1. Send concurrently (within 10 seconds or so) several virt-who reports - ideally, generate the json file via virt-who -op for each hypervisor (to have multiple reports), and then run:

for file in $(ls virt-who.report.*.json); do curl -ks -u admin:Password -X POST -H "Content-Type: text/plain" -d @${file} 'https://satellite.example.com/rhsm/hypervisors/ORGANIZATION' & done

2. Check status of Hypervisors tasks spawned


Actual results:
Some of the tasks fail with:

2018-10-08 14:34:49  [foreman-tasks/action] [E] undefined method `[]' for nil:NilClass (NoMethodError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.86/app/lib/actions/katello/host/hypervisors.rb:23:in `block in parse_hypervisors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.86/app/lib/actions/katello/host/hypervisors.rb:22:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.86/app/lib/actions/katello/host/hypervisors.rb:22:in `parse_hypervisors'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.86/app/lib/actions/candlepin/async_hypervisors.rb:14:in `poll_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.86/app/lib/actions/candlepin/async_hypervisors.rb:20:in `invoke_external_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action/polling.rb:83:in `initiate_external_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action/polling.rb:18:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.86/app/lib/actions/candlepin/abstract_async_task.rb:9:in `run'
..


Expected results:
No such nullpointer exception


Additional info:
The reason is, Candlepin responded with WAITING state of a task (not sure if it is a new task or an announcement candlepin is busy), what katello cant react on properly. That is evident when I put to /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.4.5.86/app/lib/actions/candlepin/async_hypervisors.rb :

      def poll_external_task(task_id = external_task[:id])
        task = ::Katello::Resources::Candlepin::Job.get(task_id, :result_data => true)
        unless ::Katello::Resources::Candlepin::Job.not_finished?(task)
          Rails.logger.info("PavelM: finished task=#{task.inspect}")     ################### add this debug line
          output[:hypervisors] = ::Actions::Katello::Host::Hypervisors.parse_hypervisors(task.delete('resultData'))
        end
        task
      end

that returned:

2018-10-08 15:39:26 4d9bdaeb [app] [I] PavelM: finished task={"id"=>"hypervisor_update_0376d314-c552-4920-b7f3-af2f1679bb8b", "state"=>"WAITING", "startTime"=>nil, "finishTime"=>nil, "result"=>nil, "principalName"=>"foreman_admin", "targetType"=>"owner", "targetId"=>"ORGANIZATION", "ownerId"=>"ORGANIZATION", "correlationId"=>"4d9bdaeb", "resultData"=>nil, "statusPath"=>"/jobs/hypervisor_update_0376d314-c552-4920-b7f3-af2f1679bb8b", "done"=>false, "group"=>"async group", "created"=>"2018-10-08T13:39:25+0000", "updated"=>"2018-10-08T13:39:25+0000"}

instead of typical:

2018-10-08 15:39:28 c9174a73 [app] [I] PavelM: finished task={"id"=>"hypervisor_update_b5bc3425-39ab-4792-98e2-37fecf2918c1", "state"=>"FINISHED", "startTime"=>"2018-10-08T13:39:22+0000", "finishTime"=>"2018-10-08T13:39:27+0000", "result"=>"Created: 0, Updated: 47, Unchanged:123, ..