I'm still looking into this....
On the reproducer appliance I can see that one of the widgets in question had its schedule run -
[----] I, [2018-12-19T19:00:16.762194 #17130:1311108] INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [1000122110401], MiqWorker id: [1000007606871], Zone: [default], Role: [], Server: [], Ident: [generic], Target id: [], Instance id: [1000000000003], Task id: [], Command: [MiqSchedule.invoke_actions], Timeout: [1200], Priority: [100], State: [dequeue], Deliver On: [], Data: [], Args: ["action_generate_widget", 1545350400], Dequeued in: [5.509696072] seconds
[----] I, [2018-12-19T19:00:16.762402 #17130:1311108] INFO -- : MIQ(MiqQueue#deliver) Message id: [1000122110401], Delivering...
[----] I, [2018-12-19T19:00:16.767916 #17130:1311108] INFO -- : MIQ(MiqSchedule#invoke_actions) [chart_vendor_and_guest_os] invoking action: [generate_widget] for target: [chart_vendor_and_guest_os]
[----] I, [2018-12-19T19:00:16.833383 #17130:1311108] INFO -- : MIQ(MiqSchedule#action_generate_widget) Action [chart_vendor_and_guest_os] has been run for target type: [MiqWidget] with name: [Vendor and Guest OS Chart]
[----] I, [2018-12-19T19:00:16.849726 #17130:1311108] INFO -- : MIQ(MiqQueue#delivered) Message id: [1000122110401], State: [ok], Delivered in [0.087328889] seconds
However, it never actually created a task or queue any work. The key log messages that are missing are ones like this -
[----] I, [2018-12-21T08:48:53.711223 #8619:73d110] INFO -- : MIQ(MiqWidget#create_task) Created MiqTask ID: [1000000282542], Name: [Generate Widget: 'Vendor and Guest OS Chart'] for: [26] groups
[----] I, [2018-12-21T08:48:53.757794 #8619:73d110] INFO -- : MIQ(MiqWidget#queue_generate_content) Widget: [Vendor and Guest OS Chart] ID: [1000000000003] Queueing Content Generation
Those were emitted when I ran queue_generate_content on that widget manually.
Will continue to investigate