Bug 1372234

Summary: mock --new-chroot can't shutdown mongo running inside
Product: [Fedora] Fedora Reporter: Vít Ondruch <vondruch>
Component: mockAssignee: Miroslav Suchý <msuchy>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 26CC: Frodox, jdisnard, mebrown, msimacek, msuchy, praiskup, williams
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mock-1.3.4-1.fc25 mock-1.3.4-1.fc24 mock-1.3.4-1.el7 mock-1.3.5-1.el6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-08 13:28:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vít Ondruch 2016-09-01 08:27:45 UTC
Description of problem:
I am trying to update rubygem-mongo and in this package, the MongoDB service is started to provide the database backend for the test suite [1]. After the test suite is executed, the MongoDB is shut down [2]. This works just fine with --old-chroot, but with --new-chroot it actually gets stick at the point, when MongoDB should be shut down:

```
+ mongod --shutdown --dbpath data
killing process with pid: 2500
```

Version-Release number of selected component (if applicable):
$ rpm -q mock
mock-1.2.20-1.fc26.noarch


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:



[1] http://pkgs.fedoraproject.org/cgit/rpms/rubygem-mongo.git/tree/rubygem-mongo.spec?id=07ccdc22bad0a63943b82c9de4e11c643ea68570#n68
[2] http://pkgs.fedoraproject.org/cgit/rpms/rubygem-mongo.git/tree/rubygem-mongo.spec?id=07ccdc22bad0a63943b82c9de4e11c643ea68570#n83

Comment 1 Vít Ondruch 2016-09-01 08:33:13 UTC
This should be the nspawn command used to execute the build:

Executing command: ['/usr/bin/systemd-nspawn', '-q', '-M', 'cd998437cdff44219744d978b17b999f', '-D', '/var/lib/mock/fedora-rawhide-x86_64/root', '--setenv=PS1=<mock-chroot> \\s-\\v\\$ ', '--setenv=LANG=cs_CZ.utf8', '--setenv=PROMPT_COMMAND=printf "\\033]0;<mock-chroot>\\007"', '--setenv=HOSTNAME=mock', '--setenv=HOME=/builddir', '--setenv=TERM=vt100', '--setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin', '--setenv=SHELL=/bin/bash', '-u', 'mockbuild', '/usr/bin/rpmbuild', '-bb', '--target', 'x86_64', '--nodeps', '/builddir/build/SPECS/rubygem-mongo.spec'] with env {'PS1': '<mock-chroot> \\s-\\v\\$ ', 'LANG': 'cs_CZ.utf8', 'PROMPT_COMMAND': 'printf "\\033]0;<mock-chroot>\\007"', 'HOSTNAME': 'mock', 'HOME': '/builddir', 'TERM': 'vt100', 'PATH': '/usr/bin:/bin:/usr/sbin:/sbin', 'SHELL': '/bin/bash'} and shell False

Comment 2 Vít Ondruch 2016-09-01 10:20:56 UTC
So far it looks like /proc is not correctly mounted. Let me explain.

To debug this issue, I executed the "mongod --shutdown --dbpath data" with strace, here is the snippet of output:

```
stat("/proc/2500", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
nanosleep({1, 0}, 0x7fff02417b80)       = 0
```

So it tries to access the process directory, but overall the output is not very helpful.

As a next step, I tried to debug what is going on with the process, so I followed this guide [1] how to debug the container. I my case, it was something like:

```
$ machinectl 
MACHINE                          CLASS     SERVICE       
213731c4a85d47f38e2e83765d201807 container systemd-nspawn

1 machines listed.

$ machinectl status 213731c4a85d47f38e2e83765d201807
213731c4a85d47f38e2e83765d201807(224dc752107d419d9c071eb3c7e745b8)
           Since: Čt 2016-09-01 11:01:38 CEST; 17s ago
          Leader: 21334 (rpmbuild)
         Service: systemd-nspawn; class container
            Root: /var/lib/mock/fedora-rawhide-x86_64/root
              OS: Fedora 26 (Rawhide)
            Unit: machine-213731c4a85d47f38e2e83765d201807.scope
                  ├─21334 /usr/bin/rpmbuild -bb --target x86_64 --nodeps /builddir/build/SPECS/rubygem-mongo.spec
                  ├─23913 /bin/sh -e /var/tmp/rpm-tmp.d8iLLi
                  ├─23935 strace mongod --shutdown --dbpath data
                  └─23937 mongod --shutdown --dbpath data

zář 01 11:01:38 unused-4-132.brq.redhat.com systemd[1]: Started Container 213731c4a85d47f38e2e83765d201807.

$ sudo nsenter --target 23937 --mount --uts --ipc --net
```

But as soon as I tried to list the running processes, I got this error:

```
# ps -ef
Error, do this: mount -t proc proc /proc
```

As soon as I executed the mount command suggest by the ps error output, suddenly, the ps started to work, but not only that, also the mongodb immediately shuts down.

My question now is what is wrong with the /proc directory, since it seems to be mounted but it is probably not properly accessible for some reason. Is this mock issue or nspawn issue?

[1] https://seanmcgary.com/posts/nsenter-a-systemd-nspawn-container

Comment 3 Vít Ondruch 2016-09-01 11:03:30 UTC
I should probably also mention SystemD version I am using:

$ rpm -qf `which systemd-nspawn`
systemd-container-231-3.fc26.x86_64

Comment 4 Vít Ondruch 2016-12-16 09:47:33 UTC
Ping?

Comment 5 Vít Ondruch 2017-01-27 16:08:12 UTC
Now I leardned, that there is this options of systemd-nspawn:

  -a --as-pid2              Maintain a stub init as PID1, invoke binary as PID2

Trying the command above with this option, it seems the build passes. Nevertheless, I have no idea if this is the right medicine ...

Comment 7 Miroslav Suchý 2017-02-21 16:22:04 UTC
PR in #6 has been merged.

Comment 8 Fedora Update System 2017-02-27 17:55:46 UTC
mock-1.3.4-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2017-1c433a692a

Comment 9 Fedora Update System 2017-02-27 17:56:08 UTC
mock-1.3.4-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2c7e8574bf

Comment 10 Fedora Update System 2017-02-27 17:56:30 UTC
mock-1.3.4-1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-3cb1bb5aa5

Comment 11 Fedora Update System 2017-02-27 17:56:52 UTC
mock-1.3.4-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-89e17056c0

Comment 12 Fedora End Of Life 2017-02-28 10:11:15 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle.
Changing version to '26'.

Comment 13 Fedora Update System 2017-03-01 01:48:46 UTC
mock-1.3.4-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-89e17056c0

Comment 14 Fedora Update System 2017-03-01 02:50:42 UTC
mock-1.3.4-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-1c433a692a

Comment 15 Fedora Update System 2017-03-01 02:53:34 UTC
mock-1.3.4-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-3cb1bb5aa5

Comment 16 Fedora Update System 2017-03-01 17:17:42 UTC
mock-1.3.4-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2c7e8574bf

Comment 17 Fedora Update System 2017-03-02 10:21:44 UTC
mock-1.3.5-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2c7e8574bf

Comment 18 Fedora Update System 2017-03-03 03:54:59 UTC
mock-1.3.4-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2017-03-03 04:17:17 UTC
mock-1.3.5-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-2c7e8574bf

Comment 20 Fedora Update System 2017-03-08 13:28:36 UTC
mock-1.3.4-1.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Comment 21 Fedora Update System 2017-03-15 13:48:20 UTC
mock-1.3.4-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.

Comment 22 Fedora Update System 2017-04-13 00:16:43 UTC
mock-1.3.5-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.