Bug 2124419

+++ This bug was initially created as a clone of Bug #2124287 +++

The problem / reproducer:

1) stop yggdrasild on a client:
systemctl stop yggdrasild

2) invoke a pull-mqtt REX job to the client

3) start the service back:
systemctl start yggdrasild

4) check status of the job - it will be hung forever

(the reason is yggd start drops all incoming notifications)


We *need* to document this *somehow*. Not sure about the most appropriate way, if:
- in install/config guide with pull-mqtt config description, as a warning
- in release notes (where I assume the pull-mqtt feature will be mentioned either way), again as a warning
- as a KCS (that I am happy to write, but I feel it is rather insufficient post-mortem documentation)

--- Additional comment from Pavel Moravec on 2022-09-05 14:16:57 UTC ---

See:

https://issues.redhat.com/browse/SAT-11349  As a user, I expect jobs which were scheduled when a host was down to be processed once the machine comes up
https://issues.redhat.com/browse/SAT-7337 Tasks delivered during yggd start are dropped 

for the "missing feature behaviour"

Recall that katello-agent works well here, its jobs survive goferd and also qpidd restart, and customers would see this as a regression. Let at least notify them about it.

Brad, who shall decide about the best place to document it?

--- Additional comment from Sayan Das on 2022-09-05 17:23:49 UTC ---

+1 for :

~~

3) start the service back:
systemctl start yggdrasild

4) check status of the job - it will be hung forever

(the reason is yggd start drops all incoming notifications)

~~


I think this is a major flaw in mosquitto + uggdrasild communication i.e. if there is a disturbance and the yggdrasild was down when the Job was pushed to the smart-proxy, That job will remain hung forever even if yggdrasild was brought up online. 

I faced this issue number of times when I was playing around with "Change COntent SOurce" + pull-mqtt before and after .

The hung job is not even cancellable but It needs to be "Aborted". I feel apart from documentation, It should be considered a product bug as well as big customers will bite us back later for this behavior. 


CC'ing aruzicka

--- Additional comment from Brad Buckingham on 2022-09-05 22:19:53 UTC ---

I'd recommend this be documented as a Known Issue in the Release Notes, unless it is resolved prior to GA.  It could also be in a KCS for those that may miss the note.

Is there a related bugzilla to address the behavior with the workflow?

I agree that it would be viewed as a regression in behavior when moving to the REX Pull-Provider.  We had to go through similar growing pains with katello-agent to make it more resilient.