Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1918039

Summary:	jobs scheduled in nova_api_cron, heat_api_cron and cinder_api_cron are not running as scheduled
Product:	Red Hat OpenStack	Reporter:	David Vallee Delisle <dvd>
Component:	openstack-tripleo-heat-templates	Assignee:	RHOS Maint <rhos-maint>
Status:	CLOSED NOTABUG	QA Contact:	Joe H. Rahme <jhakimra>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	16.1 (Train)	CC:	agreentr, dasmith, dvd, eglynn, jhakimra, kchamart, mburns, nova-maint, rhopp, sbauza, sgordon, vromanso
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1916956	Environment:
Last Closed:	2021-01-20 13:24:29 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1916956, 2203391
Bug Blocks:

Description David Vallee Delisle 2021-01-19 21:57:26 UTC

We can confirm that the jobs are not running because of no trace in the logs in the host.

We can confirm that syslog redirection from the containers to the host works because when we edit the crontabs [1], we can see the operations [2]

After adding a job every minute, we can see the events occuring in the host's journald

This is a follow up on bz#1916956 where the operators had a pretty large number of deleted instances, to the point where archiving was broken.

I confirmed that I don't see events from heat_api_cron and cinder_api_cron either.

When we reload the cron, the first time, it works and we started seeing the events in journald, but when we restart the containers, we end up with these messages [3].

I'll deploy a fresh lab and reproduce.

[1]
~~~
podman exec -u0 -ti nova_api_cron bash
crontab -e -u nova
~~~

[2]
~~~
Jan 19 21:21:42 ess1612-ctrl-1 crontab[966216]: (root) BEGIN EDIT (nova)
Jan 19 21:21:44 ess1612-ctrl-1 crontab[966216]: (root) END EDIT (nova)
~~~

[3]
~~~
Jan 19 21:22:01 ess1612-ctrl-1 CROND[967632]: (nova) CMD (echo "DVD writing from container")
Jan 19 21:22:02 ess1612-ctrl-1 CROND[967631]: (nova) CMDOUT (DVD writing from container)
~~~

[4]
~~~
Jan 19 21:54:01 ess1612-ctrl-2 crond[696397]: pam_loginuid(crond:session): Error writing /proc/self/loginuid: Operation not permitted
Jan 19 21:54:01 ess1612-ctrl-2 crond[696397]: pam_loginuid(crond:session): set_loginuid failed
Jan 19 21:54:01 ess1612-ctrl-2 crond[696397]: (heat) PAM ERROR (Cannot make/remove an entry for the specified session)
Jan 19 21:54:01 ess1612-ctrl-2 crond[696397]: (heat) FAILED to open PAM security session (Cannot make/remove an entry for the specified session)
Jan 19 21:54:01 ess1612-ctrl-2 crond[696477]: pam_loginuid(crond:session): Error writing /proc/self/loginuid: Operation not permitted
Jan 19 21:54:01 ess1612-ctrl-2 crond[696476]: pam_loginuid(crond:session): Error writing /proc/self/loginuid: Operation not permitted
Jan 19 21:54:01 ess1612-ctrl-2 crond[696477]: pam_loginuid(crond:session): set_loginuid failed
Jan 19 21:54:01 ess1612-ctrl-2 crond[696476]: pam_loginuid(crond:session): set_loginuid failed
Jan 19 21:54:01 ess1612-ctrl-2 crond[696477]: (root) PAM ERROR (Cannot make/remove an entry for the specified session)
Jan 19 21:54:01 ess1612-ctrl-2 crond[696476]: (nova) PAM ERROR (Cannot make/remove an entry for the specified session)
Jan 19 21:54:01 ess1612-ctrl-2 crond[696477]: (root) FAILED to open PAM security session (Cannot make/remove an entry for the specified session)
Jan 19 21:54:01 ess1612-ctrl-2 crond[696476]: (nova) FAILED to open PAM security session (Cannot make/remove an entry for the specified session)
~~~