Bug 1273565 - docker exec behavior differs from run
docker exec behavior differs from run
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: docker (Show other bugs)
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Mrunal Patel
: Extras
Depends On:
  Show dependency treegraph
Reported: 2015-10-20 13:24 EDT by Jim Perrin
Modified: 2017-10-03 21:30 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Jim Perrin 2015-10-20 13:24:06 EDT
Description of problem:

Disclaimer: I know this isn't considered proper use for docker, but this still smells like a bug to me. 

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. in terminal 1, 'docker run --rm -it centos:latest'
2. in terminal 2, docker exec -it <hash from terminal1> /bin/bash'
3. in the resulting shell, run 'curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.rpm.sh | bash'

Actual results:
This script configures a yum repository that puts its gpg key behind https, rather than locally in /etc/pki/rpm-gpg. This seems to trigger python/gpg errors.

Downloading repository file: https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/config_file.repo?os=centos&dist=7&source=script
Installing pygpgme to verify GPG signatures...
Loaded plugins: fastestmirror
base                                                                                           | 3.6 kB  00:00:00     
extras                                                                                         | 3.4 kB  00:00:00     
runner_gitlab-ci-multi-runner-source/signature                                                 |  836 B  00:00:00     
Retrieving key from https://packages.gitlab.com/gpg.key
Importing GPG key 0xE15E78F4:
 Userid     : "GitLab B.V. (package repository signing key) <packages@gitlab.com>"
 Fingerprint: 1a4c 919d b987 d435 9396 38b9 1421 9a96 e15e 78f4
 From       : https://packages.gitlab.com/gpg.key
runner_gitlab-ci-multi-runner-source/signature                                                 |  951 B  00:00:00 !!! 
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 367, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 174, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 571, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 440, in doCommand
    return base.installPkgs(extcmds, basecmd=basecmd)
  File "/usr/share/yum-cli/cli.py", line 967, in installPkgs
    txmbrs = self.install(pattern=arg)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4764, in install
    mypkgs = self.pkgSack.returnPackages(patterns=pats,
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1079, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 784, in _getSacks
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 344, in populateSack
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
    dl = repo._async and repo._commonLoadRepoXML(repo)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1456, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1234, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 448, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 425, in _mirror_try
    return func_ref( *(fullurl,), opts=opts, **kw )
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1172, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1064, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1166, in retryfunc
    _run_callback(opts.checkfunc, obj)
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1032, in _run_callback
    return cb(obj, *arg, **karg)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1700, in _checkRepoXML
    self.gpg_import_func(self, self.confirm_func)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 6352, in getKeyForRepo
    self._getAnyKeyForRepo(repo, repo.gpgdir, repo.gpgkey, is_cakey=False, callback=callback)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 6314, in _getAnyKeyForRepo
    result = misc.import_key_to_pubring(info['raw_key'], info['hexkeyid'], gpgdir=destdir)
  File "/usr/lib/python2.7/site-packages/yum/misc.py", line 560, in import_key_to_pubring
gpgme.GpgmeError: (7, 32870, u'Inappropriate ioctl for device')

If a repository like epel is used which imports the gpg key from a local file, then yum proceeds normally.

Expected results:

Yum fetches and validates the remote gpg key. 

Additional info:

If this command is passed into the shell via 'docker run --rm -it centos:latest /bin/bash'

then it works as expected and functions normally. 

While I know this is an outside (and probably ill-advised) use case, I do believe that docker exec and docker run should function somewhat similarly.
Comment 2 Antonio Murdaca 2015-10-30 08:46:24 EDT
This is actually an issue with docker exec not having a tty when exec'ing;

docker exec -ti CONTAINER tty
not at tty

docker run -ti centos:latest tty

The issue is actually tracked upstream here: https://github.com/docker/docker/issues/8755

A workarund for your script is to exec and run script:

docker exec -ti 603 script -qc '/bin/bash'

This way the resulting exec does have a tty (/dev/pts/0) and your script runs fine. I'm gonna see if this is fixable now since there seems to be some issues in the way Docker itself mounts /dev/console in the container's mount ns.
Comment 3 Daniel Walsh 2015-10-30 11:24:30 EDT
Nice find Antonio.  Mrunal is also looking at this issue.  Strange that the issue above went dead half way through the summer.
Comment 4 Mrunal Patel 2015-11-05 19:46:07 EST
I spent some time looking into this today. The main issue is that the exec process parent outside needs to be able to copy data back/forth from the container's controlling terminal, so we leak a fd which becomes the tty in the container. However, this fd  for e.g. /dev/pts/11 is on the host which isn't visible in the exec container's namespace as soon as we join the mount namespace.

I will try out a couple of ideas I have and update here.

Using script is a good workaround for now.

Comment 5 Daniel Walsh 2015-12-01 17:22:47 EST
Mrunal were you able to find any solution to this?
Comment 6 Mrunal Patel 2015-12-03 13:51:07 EST
I tried a couple of things that didn't work out. One more idea I have is bind mounting in the devtmpfs from outside so the exec could use the same. I will try that out.
Comment 7 Daniel Walsh 2016-01-06 14:46:30 EST
Mrunal any update?
Comment 8 Mrunal Patel 2016-01-25 12:22:40 EST
No update yet. Higher priority work is taking precedence. I will get back to it.
Comment 11 Mrunal Patel 2016-06-03 15:10:03 EDT
We have discussed a possible fix for this in runc. It requires some rework but I expect it to be fixed in the coming weeks.
Comment 12 Mrunal Patel 2016-06-03 15:17:09 EDT
To add more details, we are planning to create the psuedoterminal pair in the container instead of on the host and then pass back the master to the parent process for copying data back and forth.

Today the pair is created outside on the host so the console doesn't truly belong to the container.
Comment 13 hmeng 2016-06-03 15:19:40 EDT
I will try to understand the problem this week, we can discuss it together next week.
Comment 14 Mrunal Patel 2016-08-22 13:18:51 EDT
Work is being done upstream to overhaul the console handling code which will hopefully fix this before runc 1.0.
Comment 15 Daniel Walsh 2016-10-18 11:01:36 EDT
Work is continuing upstream on this.
Comment 16 Daniel Walsh 2017-06-30 11:36:00 EDT
Mrunal has this been fixed?
Comment 17 Mrunal Patel 2017-06-30 11:41:27 EDT
Yeah, this is fixed but will come out as part of docker 17.06.
Comment 18 Daniel Walsh 2017-07-02 06:13:31 EDT
Is this something we should consider back porting or just say it is in a future version of docker?

Note You need to log in before you can comment on or make changes to this bug.