Bug 2112402

+++ This bug was initially created as a clone of Bug #2105144 +++

Description of problem:
>>
Scheduling a remote execution job through API does not honor -0400 in the scheduling call. It was working as expected till Satellite 6.9 and after upgrade to 6.10 the behavior is changed.

It works expected using hammer and WebUI.  However, when schedule a job through API call, it is changing it to UTC.

Version-Release number of selected component (if applicable):
Satellite 6.10.7

How reproducible:

---------------

- Schedule remote execution through API to compare 

- On Satellite 6.9

Set timezone for admin user from Administer >> Users >> admin >> Timezone >> (GMT -05:00) Eastern Time (US & Canada)
[root@sat69 ~]# date       <------Date on the server
Thu Jul  7 23:01:56 EDT 2022

[root@sat69 ~]# curl -k -X POST -H "Content-Type:application/json" -d '{"job_invocation" : {"job_template_id":"153","scheduling": {"start_at": "2022-07-07 23:04:16"}, "inputs": { "command": "uptime"},"search_query":"name ^ (station.example.com)", "targeting_type":"static_query"}}' -s -u admin:password  https://sat69.example.com/api/job_invocations

Time zone: America/New_York (EDT, -0400)

 Evaluated at: 2022-07-07 23:02:32 -0400
 Scheduled to start at: 2022-07-07 23:04:00 -0400        <------ On WebUI job shows that it will run in a minute.

-----
- On Satellite 6.10

Set timezone for admin user Administer >> Users >> admin >> Timezone >> (GMT -05:00) Eastern Time (US & Canada)

[root@sat610 ~]# date
Thu Jul  7 23:05:37 EDT 2022

[root@sat610 ~]# curl -k -X POST -H "Content-Type:application/json" -d '{"job_invocation" : {"job_template_id":"162","scheduling": {"start_at": "2022-07-07 23:09:16"}, "inputs": { "command": "uptime"},"search_query":"name ^ (station.example.com)", "targeting_type":"static_query"}}' -s -u admin:password  https://sat610.example.com/api/job_invocations

 Evaluated at: 2022-07-07 23:06:39 -0400
 Scheduled to start at: 2022-07-08 03:09:00 -0400    <------ On WebUI job shows that it will run after 4 hours.
---------------


Actual results:
>> Schedule call through API schedule  4 hours later than actual timezone.

Expected results:
It should not schedule call default to UTC and should honor -0400.

--- Additional comment from  on 2022-07-08T10:16:06Z 

This apparently did get broken by trying to fix https://bugzilla.redhat.com/show_bug.cgi?id=1914902 . For 6.11 it was also touched by https://bugzilla.redhat.com/show_bug.cgi?id=2052048

> It works expected using hammer

Does it? I would say it is more a case of several wrongs somewhat cancelling each other out. At least on my 6.10.7, if I pass --start-at '2022-07-07 23:09:16' to hammer, hammer assumes it is in UTC and adds +00:00 suffix to it before sending it to the server. On the server side, it gets processed and results into a task being scheduled for $TIME in server's time zone (2022-07-07 23:09:00 -0600), not the user's, which should be -0700.

--- Additional comment from  on 2022-07-08T12:05:07Z 

Upstream bug assigned to aruzicka

--- Additional comment from  on 2022-07-08T12:05:09Z 

Upstream bug assigned to aruzicka

--- Additional comment from  on 2022-07-08T13:29:25Z 

I ran some test and using this format "2022-07-08T09:20:00+00:00" the local timezone set for the user is considered (meaning, it will schedule the job to run at 09:20 at the timezone of the user making the request.

Example (time zone of the user is set to "Eastern Time (US & Canada)" which currently represents GMT-4 due to daylight saving)

~~~
# curl -k -X POST -H "Content-Type:application/json" -d '{"job_invocation" : {"job_template_id":"158","inputs": { "command": "uptime"}, "search_query":"name ^ (testserver)", "targeting_type":"static_query", "scheduling": { "start_at" : "2022-07-08T09:20:00+00:00" }}}' -s -u admin:password https://$(hostname -f)/api/job_invocations|json_reformat
{
    "id": 56,
    "description": "Run uptime",
    "job_category": "Commands",
    "targeting_id": 56,
    "status": 2,
    "start_at": "2022-07-08 09:20:00 -0400",
    "status_label": "queued",
    "dynflow_task": {
        "id": "432ed185-4520-4bd2-b50f-fb6254ab26d8",
        "state": "scheduled"
    },
    "succeeded": "N/A",
    "failed": "N/A",
    "pending": "N/A",
    "total": "N/A",
    "missing": 0,
    "targeting": {
        "bookmark_id": null,
        "search_query": "name ^ (testserver)",
        "targeting_type": "static_query",
        "user_id": 4,
        "randomized_ordering": null,
        "hosts": [

        ]
    },
    "task": {
        "id": "432ed185-4520-4bd2-b50f-fb6254ab26d8",
        "state": "scheduled"
    },
    "mode": "future",
    "scheduling": {
        "start_at": "2022-07-08 09:20:00 -0400",
        "start_before": null
    }
}
~~~

If the user doesn't have a specific timezone defined (set to "Browser timezone"), then same syntax will schedule jobs to run at UTC time:

~~~
# curl -k -X POST -H "Content-Type:application/json" -d '{"job_invocation" : {"job_template_id":"158","inputs": { "command": "uptime"}, "search_query":"name ^ (testserver)", "targeting_type":"static_query", "scheduling": { "start_at" : "2022-07-08T10:20:00+00:00" }}}' -s -u admin:redhat https://$(hostname -f)/api/job_invocations|json_reformat
{
    "id": 57,
    "description": "Run uptime",
    "job_category": "Commands",
    "targeting_id": 57,
    "status": 2,
    "start_at": "2022-07-08 10:20:00 UTC",
    "status_label": "queued",
    "dynflow_task": {
        "id": "360ae89a-f6ce-41a5-a37f-5da68e5838f6",
        "state": "scheduled"
    },
    "succeeded": "N/A",
    "failed": "N/A",
    "pending": "N/A",
    "total": "N/A",
    "missing": 0,
    "targeting": {
        "bookmark_id": null,
        "search_query": "name ^ (testserver)",
        "targeting_type": "static_query",
        "user_id": 4,
        "randomized_ordering": null,
        "hosts": [

        ]
    },
    "task": {
        "id": "360ae89a-f6ce-41a5-a37f-5da68e5838f6",
        "state": "scheduled"
    },
    "mode": "future",
    "scheduling": {
        "start_at": "2022-07-08 10:20:00 UTC",              <================== HERE
        "start_before": null
    }
}
~~~

Maybe this is not the expected behavior, but seems consistent.

--- Additional comment from  on 2022-07-28T00:05:10Z 

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35181 has been resolved.