Bug 1661597

Summary: Under podman, python recompiles sources even if they are compiled in build time
Product: Red Hat Enterprise Linux 8 Reporter: Jan Pazdziora (Red Hat) <jpazdziora>
Component: podmanAssignee: Nalin Dahyabhai <nalin>
Status: CLOSED ERRATA QA Contact: atomic-bugs <atomic-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0CC: ddarrah, dwalsh, jligon, jpazdziora, lsm5, nalin, ypu
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-1.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-05 21:00:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jan Pazdziora (Red Hat) 2018-12-21 17:45:36 UTC
Description of problem:

In freeipa-container we use the patch python sources && python -m compileall approach to make needed changes to the python sources yet do all the changes in build time.

Under podman, that no longer works.

Version-Release number of selected component (if applicable):

podman-0.11.1.1-2.git594495d.module+el8+2237+43d1adf6.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have Dockerfile where a python source gets updated and compiled:

FROM registry.access.redhat.com/rhel7
RUN rm /usr/lib64/python2.7/gzip.py[co]
RUN touch /usr/lib64/python2.7/gzip.py && python -m compileall /usr/lib64/python2.7/gzip.py

2. Build the image: podman build -t diffit .
3. Run container: podman run --name diffit-c --rm -ti diffit bash
4. In the container, check the files: 
[root@305616dff5ba /]# stat /usr/lib64/python2.7/gzip.py*
  File: '/usr/lib64/python2.7/gzip.py'
  Size: 18694     	Blocks: 40         IO Block: 4096   regular file
Device: 37h/55d	Inode: 101856031   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-21 17:09:43.000000000 +0000
Modify: 2018-12-21 17:09:43.000000000 +0000
Change: 2018-12-21 17:09:42.638174210 +0000
 Birth: -
  File: '/usr/lib64/python2.7/gzip.pyc'
  Size: 15071     	Blocks: 32         IO Block: 4096   regular file
Device: 37h/55d	Inode: 101856032   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-21 17:09:43.000000000 +0000
Modify: 2018-12-21 17:09:43.000000000 +0000
Change: 2018-12-21 17:09:42.638174210 +0000
 Birth: -

5. Note that the timestamps on both .py and .pyc are the same.
6. In the container, use the python module:
[root@305616dff5ba /]# python -c 'import gzip'

7. In the container, stat the files again: stat /usr/lib64/python2.7/gzip.py*
8. In different terminal, use podman diff: podman diff diffit-c

Actual results:

  File: '/usr/lib64/python2.7/gzip.py'
  Size: 18694     	Blocks: 40         IO Block: 4096   regular file
Device: 37h/55d	Inode: 101856031   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-21 17:09:43.000000000 +0000
Modify: 2018-12-21 17:09:43.000000000 +0000
Change: 2018-12-21 17:09:42.638174210 +0000
 Birth: -
  File: '/usr/lib64/python2.7/gzip.pyc'
  Size: 15071     	Blocks: 32         IO Block: 4096   regular file
Device: 37h/55d	Inode: 68517034    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-21 17:43:02.338619328 +0000
Modify: 2018-12-21 17:43:02.338619328 +0000
Change: 2018-12-21 17:43:02.338619328 +0000
 Birth: -

and

C /usr/lib64/python2.7
C /usr/lib64/python2.7/gzip.pyc

Expected results:

Stat should still show the same timestamp for both .py and .pyc, podman diff should not show anything.

Additional info:

Comment 1 Daniel Walsh 2018-12-21 20:23:32 UTC
I know that Nalin was playing around with some timestamp issues, that might fix this problem.

Could you by any change check this out with podman 0.12?

Comment 2 Jan Pazdziora (Red Hat) 2018-12-22 20:54:37 UTC
With podman-0.12.1.2-4.git9551f6b.module+el8+2594+933f97c7.x86_64, the behaviour is the same:

host# podman run --name diffit-c --rm -ti diffit bash
[root@b9e3cc619f91 /]# stat /usr/lib64/python2.7/gzip.py*
  File: '/usr/lib64/python2.7/gzip.py'
  Size: 18694     	Blocks: 40         IO Block: 4096   regular file
Device: 2ch/44d	Inode: 33937730    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-22 20:52:34.000000000 +0000
Modify: 2018-12-22 20:52:34.000000000 +0000
Change: 2018-12-22 20:52:33.869019692 +0000
 Birth: -
  File: '/usr/lib64/python2.7/gzip.pyc'
  Size: 15071     	Blocks: 32         IO Block: 4096   regular file
Device: 2ch/44d	Inode: 33937731    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-22 20:52:34.000000000 +0000
Modify: 2018-12-22 20:52:34.000000000 +0000
Change: 2018-12-22 20:52:33.870019707 +0000
 Birth: -
[root@b9e3cc619f91 /]# python -c 'import gzip'
[root@b9e3cc619f91 /]# stat /usr/lib64/python2.7/gzip.py*
  File: '/usr/lib64/python2.7/gzip.py'
  Size: 18694     	Blocks: 40         IO Block: 4096   regular file
Device: 2ch/44d	Inode: 33937730    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-22 20:52:34.000000000 +0000
Modify: 2018-12-22 20:52:34.000000000 +0000
Change: 2018-12-22 20:52:33.869019692 +0000
 Birth: -
  File: '/usr/lib64/python2.7/gzip.pyc'
  Size: 15071     	Blocks: 32         IO Block: 4096   regular file
Device: 2ch/44d	Inode: 68104574    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2018-12-22 20:53:08.261520518 +0000
Modify: 2018-12-22 20:53:08.262520533 +0000
Change: 2018-12-22 20:53:08.262520533 +0000
 Birth: -

host# podman diff diffit-c
C /usr/lib64/python2.7
C /usr/lib64/python2.7/gzip.pyc

Comment 3 Nalin Dahyabhai 2019-01-10 22:11:25 UTC
Yes, this looks like it's another result of golang 1.10's archive/tar package rounding timestamps instead of truncating them, this time tripping up the python interpreter, which decides whether or not a .py file needs to be byte compiled by comparing the mtime on the .py file with a 32-bit value that was stored in the .pyc file, which I expect it's arriving at by truncating a value.

We added a workaround in containers/storage#252 to force truncation when creating layer diffs, but libpod v0.12.1.2 was built using a version of a containers/storage from just before we merged that workaround, so it's affected.  libpod was updated to a sufficiently new version in libpod#2071, so its next release should avoid creating images that cause this to happen.

Comment 4 Jan Pazdziora (Red Hat) 2019-01-11 09:49:08 UTC
I confirm that I no longer see the podman diff issue in Travis CI: https://travis-ci.org/adelton/freeipa-container/builds/477234614, so at least upstream seems to be fixed now.

Comment 5 Daniel Walsh 2019-01-11 14:32:19 UTC
Fixed in podman 1.0.

Comment 8 Joy Pu 2019-09-27 06:43:06 UTC
Can reproduce with podman-0.10.1.3-5.gitdb08685.el8+2131+7e3e9e07.x86_64. Test with podman-1.4.2-5.module+el8.1.0+4240+893c1ab8.x86_64. It works as expect and podman diff will not show anything now. So set this to verified.

Comment 10 errata-xmlrpc 2019-11-05 21:00:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:3403