Description of problem:
Via https://bugzilla.redhat.com/show_bug.cgi?id=1501852 we fixed a bunch of issues around puppet errors being ignored by the higher level parts of the stack. As it turns out there are other invocations where we ignore the errors reported. Namely in ansible playbooks like this:
- name: return code
  shell: |
    echo "fail 2"
    exit 2
  failed_when: false
  log_when: false
  register: outputs
- debug: msg="Broken (does not fail as expected)"
  when: outputs is defined
  failed_when: outputs|failed

The problem with the above playbooks is that 'failed_when: false' makes it so
that the 'failed' key in the outputs dictionary is set to false and so we do not ever fail correctly. The other problem is that 'outputs is defined' is always true even when the previous task was skipped.