Bug 1670032
|
Description
Alexander Todorov
2019-01-28 13:14:15 UTC
Created attachment 1524211 [details]
/var/log/lorax-composer/composer.log
Created attachment 1524212 [details]
/var/log/lorax-composer/program.log
Created attachment 1524213 [details]
/var/log/lorax-composer/server.log
Created attachment 1524214 [details]
/var/log/lorax-composer/yum.log
Created attachment 1524215 [details]
/var/lib/lorax/composer/results/72e435fb-e7b9-47d6-abce-8866e886d08f/logs/anaconda/anaconda.log
Created attachment 1524217 [details]
/var/lib/lorax/composer/results/72e435fb-e7b9-47d6-abce-8866e886d08f/logs/anaconda/packaging.log
Created attachment 1524218 [details]
/var/lib/lorax/composer/results/72e435fb-e7b9-47d6-abce-8866e886d08f/logs/anaconda/program.log
Created attachment 1524219 [details]
/var/lib/lorax/composer/results/72e435fb-e7b9-47d6-abce-8866e886d08f/logs/anaconda/storage.log
More info from devel: (13,26,56) mkolman: atodorov: this is what seems to touch /etc/localtime from Anaconda: https://github.com/rhinstaller/anaconda/blob/rhel7-branch/pyanaconda/timezone.py#L90 (13,27,51) mkolman: atodorov: but it seems to be called correctly here: https://github.com/rhinstaller/anaconda/blob/rhel7-branch/pyanaconda/kickstart.py#L1888 (13,28,14) mkolman: if iutil.getSysroot() returned incorrect values, much more would be broken than just this (13,30,34) mkolman: atodorov: also, looks like we just symlink /etc/localtime to some tz file (13,30,57) mkolman: atodorov: so if the file is overwritten in place, it's something else doing that In lorax sources I see: share/runtime-cleanup.tmpl:removefrom glibc /etc/gai.conf /etc/ld.so.conf /etc/localtime /etc/rpc but I don't know ATM why and how this is used. Brian, does this lead you in the right direction? This one is a serious issue? runtime-cleanup isn't used by composer so that's not related. I'm trying to reproduce this in a 7.6 VM and am not seeing it happen. I changed my /etc/localtime to point to a zone in Asia and setup an audit to log changes to it, and it hasn't changed. Tried building ami, qcow2, and tar. Anaconda is what sets up the timezone from the kickstart, so I'm pretty confident that whatever the issue is, it's an anaconda bug. Brian, can you distil this to an anaconda command which I can use for reproduction ? Also there could be differences between the 7.6 host you are using and the one I am using (has updates applied to it and pulp repos enabled). (In reply to Alexander Todorov from comment #14) > Brian, > can you distil this to an anaconda command which I can use for reproduction ? > > Also there could be differences between the 7.6 host you are using and the > one I am using (has updates applied to it and pulp repos enabled). Well, since I can't reproduce it, I'm not 100% sure, but you should be able to take a final-kickstart.ks and run it like this: anaconda --kickstart final-kickstart.ks --cmdline --repo http://url-of-repo --dirinstall which will leave the files in /mnt/sysimage/ > if the time isn't close enough I know that some crypto protocols
> will fail, which is what seems to be happening here.
>
for the record: Amazon S3 will check timestamps when you try to upload files and if they differ with more than 15 minutes it will fail with RequestTimeTooSkewed. Unfortunately this is very common error and Google tells you to configure NTP to point to Amazon's servers to sync the time, which we already do.
More updates: I provisioned a slave manually, applied the ansible configuration and left it idle. 3+hrs and the clock is still in sync. /etc/localtime symlink hasn't been changed. Tomorrow I will try running a bash script from Jenkins that doesn't do anything but print on the terminal. (In reply to Alexander Todorov from comment #22) > More updates: > I provisioned a slave manually, applied the ansible configuration and left > it idle. 3+hrs and the clock is still in sync. /etc/localtime symlink hasn't > been changed. > After about 20hrs still no drift in the clock. I haven't seen any evidence of the original bug here in any of my testing -- as far as I can tell it is not modifying the host system. Can we close this? Or push it to 7.8? (In reply to Brian Lane from comment #29) > I haven't seen any evidence of the original bug here in any of my testing -- > as far as I can tell it is not modifying the host system. Can we close this? > Or push it to 7.8? We still see this in CI, even after the switch to Cockpit CI. You can push to 7.8 but the problem remains and I've not been able to make any progress towards figuring out what is happening. Track this with bug #1766785 Track this with bug #1766785 |