+++ This bug was initially created as a clone of Bug #1668836 +++

Description of problem:

When nfs-utils package is installed in Fedora container image, running systemd in the container work fine (apart from bug 1615101) but then the container cannot be removed.

Version-Release number of selected component (if applicable):

nfs-utils-2.3.3-3.rc2.fc30.x86_64

How reproducible:

Deterministic.

Steps to Reproduce:
1. Have Dockerfile

FROM registry.fedoraproject.org/fedora:rawhide
RUN dnf install -y nfs-utils
ENV container oci
ENTRYPOINT [ "/usr/sbin/init" ]

2. Build container image from it:
   docker build -t nfs-convert .
3. Run container from the image:
   docker run --name nfs-convert --rm nfs-convert
4. In different terminal, check that nfs-convert finished fine:
   docker exec nfs-convert systemctl status nfs-convert
5. In that other terminal, try to remove the container:
   docker rm -f nfs-convert

Actual results:

$ docker exec nfs-convert systemctl status nfs-convert
● nfs-convert.service - Preprocess NFS configuration convertion
   Loaded: loaded (/usr/lib/systemd/system/nfs-convert.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Wed 2019-01-23 16:51:44 UTC; 4s ago
  Process: 21 ExecStart=/usr/libexec/nfs-utils/nfsconvert.sh (code=exited, status=0/SUCCESS)
 Main PID: 21 (code=exited, status=0/SUCCESS)

Jan 23 16:51:44 b3a8bcd328f5 systemd[1]: nfs-convert.service: Succeeded.
Jan 23 16:51:44 b3a8bcd328f5 systemd[1]: Started Preprocess NFS configuration convertion.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

$ docker rm -f nfs-convert
Error response from daemon: Driver overlay2 failed to remove root filesystem b3a8bcd328f59b49954e6c8a294d6643206f6b30232de737c88b31c6a3251b4a: remove /var/lib/docker/overlay2/f882e8b503c40a81f94f5d379bcbb375e891500f8a8ad5643f28e690ceb5fe04/diff/etc/sysconfig/nfs: operation not permitted

and the docker run in the first terminal is still running.

Expected results:

No error, the docker run stops.

Additional info:

--- Additional comment from Jan Pazdziora on 2019-01-23 16:54:39 UTC ---

Running

chattr -i /var/lib/docker/overlay2/f882e8b503c40a81f94f5d379bcbb375e891500f8a8ad5643f28e690ceb5fe04/diff/etc/sysconfig/nfs

makes the  subsequent docker rm -f nfs-convert pass without error and remove the container.

--- Additional comment from Jan Pazdziora on 2019-01-23 16:56:49 UTC ---

At the end of /usr/libexec/nfs-utils/nfsconvert.sh, there is now

#
# Do the conversion 
#
/usr/sbin/nfsconvert

#
# If successful, make the file immutable.
# This is to ensure that configuration management 
# software gets an error trying to modify it. 
#
# Run `chattr -i /etc/sysconfig/nfs` as root 
# to make it mutable again.
#
if [ $? -eq 0 ]; then
	chattr +i /etc/sysconfig/nfs
fi

--- Additional comment from Jan Pazdziora on 2019-01-23 16:58:10 UTC ---

Either the nfs-convert.service should not be enabled by default at all, or the nfsconvert.sh should at least check if the $container environment variable is set and not do that chattr. Otherwise in Kubernetes and OpenShift there will be bunch of unremovable directories and containers around.

--- Additional comment from Jan Pazdziora on 2019-01-23 16:58:38 UTC ---

In the real scenario, the nfs-utils package gets installed into my image via some dependency, it's not like I'm installing NFS-related stuff and then complain that it does not work.

--- Additional comment from Steve Dickson on 2019-01-29 20:56:21 UTC ---

I'm thinking what needs to happen is make sysconfig/nfs mutable when the package is remove
so sysconfig/nfs can also be removed or saved. 

Can you please try this scratch build of nfs-utils that does that.

https://koji.fedoraproject.org/koji/taskinfo?taskID=32335172

--- Additional comment from Jan Pazdziora on 2019-01-30 08:10:05 UTC ---

That build does not change anything.

Note that stopping and removing a container is just stopping and removing process -- it has nothing to do with removing package via rpm.

--- Additional comment from Steve Dickson on 2019-01-30 17:00:59 UTC ---

(In reply to Jan Pazdziora from comment #6)
> That build does not change anything.
> 
> Note that stopping and removing a container is just stopping and removing
> process -- it has nothing to do with removing package via rpm.

So I guess I don't understand why making a file immutable would 
cause the container to hang....

--- Additional comment from Jan Pazdziora on 2019-01-31 07:53:24 UTC ---

Removal of the container hangs because docker cannot remove that file:

Error response from daemon: Driver overlay2 failed to remove root filesystem b3a8bcd328f59b49954e6c8a294d6643206f6b30232de737c88b31c6a3251b4a: remove /var/lib/docker/overlay2/f882e8b503c40a81f94f5d379bcbb375e891500f8a8ad5643f28e690ceb5fe04/diff/etc/sysconfig/nfs: operation not permitted

--- Additional comment from Jan Pazdziora on 2019-01-31 07:54:27 UTC ---

I mean, removing container is of course removal of the process and the underlying storage, from under /var/lib/docker/overlay2/.

But it's not running "dnf remove".

--- Additional comment from Steve Dickson on 2019-02-04 20:10:45 UTC ---

(In reply to Jan Pazdziora from comment #0)
> Description of problem:
> 
> When nfs-utils package is installed in Fedora container image, running
> systemd in the container work fine (apart from bug 1615101) but then the
> container cannot be removed.
> 
> Version-Release number of selected component (if applicable):
> 
> nfs-utils-2.3.3-3.rc2.fc30.x86_64
> 
> How reproducible:
> 
> Deterministic.
> 
> Steps to Reproduce:
> 1. Have Dockerfile
> 
> FROM registry.fedoraproject.org/fedora:rawhide
> RUN dnf install -y nfs-utils
> ENV container oci
> ENTRYPOINT [ "/usr/sbin/init" ]
> 
> 2. Build container image from it:
>    docker build -t nfs-convert .
> 3. Run container from the image:
>    docker run --name nfs-convert --rm nfs-convert
> 4. In different terminal, check that nfs-convert finished fine:
>    docker exec nfs-convert systemctl status nfs-convert
> 5. In that other terminal, try to remove the container:
>    docker rm -f nfs-convert

I'm doing something wrong...

fedora# cat Dockerfile 
FROM registry.fedoraproject.org/fedora:rawhide
RUN dnf install -y nfs-utils
ENV container oci
ENTRYPOINT [ "/usr/sbin/init" ]

fedora# docker build -t nfs-convert .
Sending build context to Docker daemon 18.43 MB
Step 1/4 : FROM registry.fedoraproject.org/fedora:rawhide
 ---> fcecb29385cc
Step 2/4 : RUN dnf install -y nfs-utils
 ---> Using cache
 ---> 94107a902e65
Step 3/4 : ENV container oci
 ---> Using cache
 ---> 83bd0a1b7f79
Step 4/4 : ENTRYPOINT /usr/sbin/init
 ---> Using cache
 ---> eb0776652d97
Successfully built eb0776652d97

fedora# docker exec nfs-convert systemctl status nfs-convert
Error response from daemon: No such container: nfs-convert

any ideas?

--- Additional comment from Jan Pazdziora on 2019-02-05 07:39:04 UTC ---

You need to run the container before exec-ing into it:

docker run --name nfs-convert --rm nfs-convert

You might also need the container_manage_cgroup SELinux boolean set to true.

--- Additional comment from Steve Dickson on 2019-02-05 15:12:55 UTC ---

(In reply to Jan Pazdziora from comment #11)
> You need to run the container before exec-ing into it:
> 
> docker run --name nfs-convert --rm nfs-convert
my bad.... I did do this 
fedora# docker run --name nfs-convert --rm nfs-convert

and still got

fedora# docker exec nfs-convert systemctl status nfs-convert
Error response from daemon: No such container: nfs-convert

> 
> You might also need the container_manage_cgroup SELinux boolean set to true.
Where do I set this, since SELinux is enabled

tia!

--- Additional comment from Jan Pazdziora on 2019-02-05 15:21:18 UTC ---

setsebool container_manage_cgroup 1

--- Additional comment from Steve Dickson on 2019-02-05 16:57:12 UTC ---

(In reply to Jan Pazdziora from comment #13)
> setsebool container_manage_cgroup 1

It was a SELinux thing... 

fedora# docker run --name nfs-convert --rm nfs-convert
now does not return immediately and the container is found. Thanks!

But... I do have a question about this is even being done: 
'docker exec nfs-convert systemctl status nfs-convert'

This service is only used by the NFS server what it is started
so it should not be called directly. It should be called via
systemctl start nfs-server.

Note when I do docker 'exec nfs-convert systemctl start nfs-server'
it fails with 

systemd[1]: Dependency failed for NFS server and services.
systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.

Which I don't understand because when I created the container all the dependent
packages got installed. 

Also note, the docker rm -r nfs-convert works for me;
fedora# docker rm -f nfs-convert
nfs-convert

The nfs-utils that got install was nfs-utils-1:2.3.3-4.rc2.fc30

--- Additional comment from Jan Pazdziora on 2019-02-05 17:40:59 UTC ---

(In reply to Steve Dickson from comment #14)
> 
> But... I do have a question about this is even being done: 
> 'docker exec nfs-convert systemctl status nfs-convert'

Well, I only call that to see that nfs-convert.service in fact was run. In my case it seems to caused by rpc-gssd.service and rpc-statd-notify.service:

# docker exec nfs-convert systemctl list-dependencies --all | grep -1 nfs-convert
● │ │ ├─rpc-gssd.service
● │ │ │ ├─nfs-convert.service
● │ │ │ │ └─system.slice
--
● │ ├─rpc-statd-notify.service
● │ │ ├─nfs-convert.service
● │ │ │ └─system.slice
--
●     │ ├─rpc-gssd.service
●     │ │ ├─nfs-convert.service
●     │ │ │ └─system.slice
--
●     ├─rpc-statd-notify.service
●     │ ├─nfs-convert.service
●     │ │ └─system.slice

> This service is only used by the NFS server what it is started
> so it should not be called directly.

In my reproducer case in comment 0, I don't start nfs-convert.service directly.

> It should be called via
> systemctl start nfs-server.
> 
> Note when I do docker 'exec nfs-convert systemctl start nfs-server'
> it fails with 
> 
> systemd[1]: Dependency failed for NFS server and services.
> systemd[1]: nfs-server.service: Job nfs-server.service/start failed with
> result 'dependency'.
> 
> Which I don't understand because when I created the container all the
> dependent
> packages got installed. 

Well, running nfs-server in the container is complete different story, probably out of scope of this bugzilla. But running

# docker exec nfs-convert journalctl -l

in my case shows

Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Listening on RPCbind Server Activation Socket.
Feb 05 17:35:17 9e9cad10a0e3 mount[139]: mount: /var/lib/nfs/rpc_pipefs: permission denied.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Starting Preprocess NFS configuration convertion...
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Reached target RPC Port Mapper.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Reached target Network.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Mounting NFSD configuration filesystem...
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Condition check resulted in Kernel Module supporting RPCSEC_GSS being skipped.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: var-lib-nfs-rpc_pipefs.mount: Mount process exited, code=exited, status=32/n/a
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: var-lib-nfs-rpc_pipefs.mount: Failed with result 'exit-code'.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Failed to mount RPC Pipe File System.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Dependency failed for rpc_pipefs.target.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Dependency failed for RPC security service for NFS client and server.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: rpc-gssd.service: Job rpc-gssd.service/start failed with result 'dependency'.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Dependency failed for NFSv4 ID-name mapping service.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: nfs-idmapd.service: Job nfs-idmapd.service/start failed with result 'dependency'.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: rpc_pipefs.target: Job rpc_pipefs.target/start failed with result 'dependency'.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: nfs-convert.service: Succeeded.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Started Preprocess NFS configuration convertion.
Feb 05 17:35:17 9e9cad10a0e3 mount[141]: mount: /proc/fs/nfsd: cannot mount nfsd read-only.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Starting NFS status monitor for NFSv2/3 locking....
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: proc-fs-nfsd.mount: Mount process exited, code=exited, status=32/n/a
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: proc-fs-nfsd.mount: Failed with result 'exit-code'.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Failed to mount NFSD configuration filesystem.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Dependency failed for NFS server and services.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: Dependency failed for NFS Mount Daemon.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start failed with result 'dependency'.
Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.

-- so that failure is caused by the container not being able to mount things, which is sort of expected. It's a systemd runtime dependency, not package dependency.

Anyway, here we probably want to focus on the situation when nfs-server is not explicitly started.

> Also note, the docker rm -r nfs-convert works for me;
> fedora# docker rm -f nfs-convert
> nfs-convert
> 
> The nfs-utils that got install was nfs-utils-1:2.3.3-4.rc2.fc30

Interesting. What does

# cat /etc/sysconfig/docker-storage

return on your host? Could it be something specific to overlay2 that you don't use? My host is Fedora 29 with

docker-1.13.1-62.git9cb56fd.fc29.x86_64

--- Additional comment from Steve Dickson on 2019-02-05 18:21:20 UTC ---

(In reply to Jan Pazdziora from comment #15)
> (In reply to Steve Dickson from comment #14)
> > 
> > But... I do have a question about this is even being done: 
> > 'docker exec nfs-convert systemctl status nfs-convert'
> 
> Well, I only call that to see that nfs-convert.service in fact was run. In
> my case it seems to caused by rpc-gssd.service and rpc-statd-notify.service:
hmm... I guess it can be running by different services.

> 
> # docker exec nfs-convert systemctl list-dependencies --all | grep -1
This gives me a grep usage error... but
docker exec nfs-convert systemctl list-dependencies --all | egrep "nfs|rpc"

> nfs-convert
> ● │ │ ├─rpc-gssd.service
> ● │ │ │ ├─nfs-convert.service
> ● │ │ │ │ └─system.slice
> --
> ● │ ├─rpc-statd-notify.service
> ● │ │ ├─nfs-convert.service
> ● │ │ │ └─system.slice
> --
> ●     │ ├─rpc-gssd.service
> ●     │ │ ├─nfs-convert.service
> ●     │ │ │ └─system.slice
> --
> ●     ├─rpc-statd-notify.service
> ●     │ ├─nfs-convert.service
> ●     │ │ └─system.slice
> 
fedora# docker exec nfs-convert systemctl list-dependencies --all | egrep "nfs|rpc"
● ├─nfs-client.target
● │ ├─auth-rpcgss-module.service
● │ │ ├─rpc-gssd.service
● │ │ │ ├─nfs-convert.service
● │ │ │ └─rpc_pipefs.target
● │ │ │   └─var-lib-nfs-rpc_pipefs.mount
● │ │ ├─rpc-svcgssd.service
● │ ├─rpc-statd-notify.service
● │ │ ├─nfs-convert.service
●   └─nfs-client.target
●     ├─auth-rpcgss-module.service
●     │ ├─rpc-gssd.service
●     │ │ ├─nfs-convert.service
●     │ │ └─rpc_pipefs.target
●     │ │   └─var-lib-nfs-rpc_pipefs.mount
●     │ ├─rpc-svcgssd.service
●     ├─rpc-statd-notify.service
●     │ ├─nfs-convert.service

which is slightly different 

> > This service is only used by the NFS server what it is started
> > so it should not be called directly.
> 
> In my reproducer case in comment 0, I don't start nfs-convert.service
> directly.
right my bad... 

> 
> > It should be called via
> > systemctl start nfs-server.
> > 
> > Note when I do docker 'exec nfs-convert systemctl start nfs-server'
> > it fails with 
> > 
> > systemd[1]: Dependency failed for NFS server and services.
> > systemd[1]: nfs-server.service: Job nfs-server.service/start failed with
> > result 'dependency'.
> > 
> > Which I don't understand because when I created the container all the
> > dependent
> > packages got installed. 
> 
> Well, running nfs-server in the container is complete different story,
> probably out of scope of this bugzilla. But running
I agree... I'm just trying to get nfs-convert to run... 
> 
> # docker exec nfs-convert journalctl -l
>
> Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: nfs-mountd.service: Job
> nfs-mountd.service/start failed with result 'dependency'.
> Feb 05 17:35:17 9e9cad10a0e3 systemd[1]: nfs-server.service: Job
> nfs-server.service/start failed with result 'dependency'.
> 
> -- so that failure is caused by the container not being able to mount
> things, which is sort of expected. It's a systemd runtime dependency, not
> package dependency.
The problem is this:
fedora# docker exec nfs-convert  mount -t nfsd nfsd /proc/fs/nfsd
mount: /proc/fs/nfsd: permission denied.

So I have to wonder how anything is service, that would run nfs-convert,
is getting started... 

> 
> Anyway, here we probably want to focus on the situation when nfs-server is
> not explicitly started.
> 
> > Also note, the docker rm -r nfs-convert works for me;
> > fedora# docker rm -f nfs-convert
> > nfs-convert
> > 
> > The nfs-utils that got install was nfs-utils-1:2.3.3-4.rc2.fc30
> 
> Interesting. What does
> 
> # cat /etc/sysconfig/docker-storage
fedora# cat /etc/sysconfig/docker-storage
DOCKER_STORAGE_OPTIONS="--storage-driver overlay2 "

> 
> return on your host? Could it be something specific to overlay2 that you
> don't use? My host is Fedora 29 with
> 
> docker-1.13.1-62.git9cb56fd.fc29.x86_64

Ah.. I'm using rawhide.... let me switch

docker-1.13.1-63.git1185cfd.fc30.x86_64

--- Additional comment from Steve Dickson on 2019-02-05 20:49:48 UTC ---

ok... I'm now able to see this with both f29 and rawhide....

Please tell me about the $container env variable.

--- Additional comment from Jan Pazdziora on 2019-02-06 07:02:03 UTC ---

(In reply to Steve Dickson from comment #17)
> ok... I'm now able to see this with both f29 and rawhide....
> 
> Please tell me about the $container env variable.

It tells systemd to act slightly differently in containerized environment. The ConditionVirtualization= directive (man systemd.unit(5)) can be used to test for execution of services in containerized environments. But if you are looking for the best way to fix this via systemd directives, systemd folks might be of better help -- I'm merely observing the failure.

--- Additional comment from Steve Dickson on 2019-02-07 16:38:05 UTC ---


Note: if nfs-utils package is removed
     docker exec nfs-convert dnf -y remove nfs-utils

then the docker rm -f nfs-convert will succeed.

Just out curiosity, what are the step to bring up
an NFS server in a container? Is it even possible?
If not then why are we worrying about this?

tia,

--- Additional comment from Jan Pazdziora on 2019-02-07 16:51:31 UTC ---

(In reply to Steve Dickson from comment #19)
> Note: if nfs-utils package is removed
>      docker exec nfs-convert dnf -y remove nfs-utils
> 
> then the docker rm -f nfs-convert will succeed.

Sure but no many people will want to run dnf remove in the container to be able to stop and remove the container.

> Just out curiosity, what are the step to bring up
> an NFS server in a container? Is it even possible?

No idea.

> If not then why are we worrying about this?

Because when nfs-utils gets installed to a container systemd, even unintentionally like via rpm dependencies, it will break smooth operation of said container because nfs-convert will execute and mess up the attribute. Which is exactly what happens when we install FreeIPA in container. As mentioned in comment 4: In the real scenario, the nfs-utils package gets installed into my image via some dependency, it's not like I'm installing NFS-related stuff and then complain that it does not work.

--- Additional comment from Steve Dickson on 2019-02-07 17:07:10 UTC ---

(In reply to Jan Pazdziora from comment #20)
> (In reply to Steve Dickson from comment #19)
> > Note: if nfs-utils package is removed
> >      docker exec nfs-convert dnf -y remove nfs-utils
> > 
> > then the docker rm -f nfs-convert will succeed.
> 
> Sure but no many people will want to run dnf remove in the container to be
> able to stop and remove the container.Yes
Yes... it is awkward... but a lot of things are awkward
in a  virtualized environments, as I'm learning... ;-)

Would it be possible to document this awkwardness? 

> 
> > Just out curiosity, what are the step to bring up
> > an NFS server in a container? Is it even possible?
> 
> No idea.
LOL... Thanks for your honesty. :-) 

I've been looking into but I can't even do
a local mount 
   'mount -t nfsd nfsd /proc/fs/nfsd' 
    mount: /proc/fs/nfsd: permission denied.
> 
> > If not then why are we worrying about this?
> 
> Because when nfs-utils gets installed to a container systemd, even
> unintentionally like via rpm dependencies, it will break smooth operation of
> said container because nfs-convert will execute and mess up the attribute.
> Which is exactly what happens when we install FreeIPA in container. As
> mentioned in comment 4: In the real scenario, the nfs-utils package gets
> installed into my image via some dependency, it's not like I'm installing
> NFS-related stuff and then complain that it does not work.
Ok...

--- Additional comment from Jan Pazdziora on 2019-02-07 17:34:00 UTC ---

(In reply to Steve Dickson from comment #21)
> 
> Would it be possible to document this awkwardness? 

Well, I strongly believe that the chattr +i part of /usr/libexec/nfs-utils/nfsconvert.sh should be reverted and different approach of what it tries to achieve found. To me, immutable bit is something that admins use as a last resort hotfix mechanism, knowingly. Not something that some service does without asking, to stop some other service (config management) from doing its business. We shouldn't need such drastic measures in our packages.

--- Additional comment from Steve Dickson on 2019-02-07 18:06:42 UTC ---

(In reply to Jan Pazdziora from comment #22)
> (In reply to Steve Dickson from comment #21)
> > 
> > Would it be possible to document this awkwardness? 
> 
> Well, I strongly believe that the chattr +i part of
> /usr/libexec/nfs-utils/nfsconvert.sh should be reverted and different
> approach of what it tries to achieve found. To me, immutable bit is
> something that admins use as a last resort hotfix mechanism, knowingly. Not
> something that some service does without asking, to stop some other service
> (config management) from doing its business. We shouldn't need such drastic
> measures in our packages.

We need to let config management solutions know that sysconfig/nfs is no
longer the place to configure NFS... Stephen and I will be meeting
this afternoon to talk about it.... 

The problem here is, as I see it, nfs-convert is being run when it shouldn't be.
So I think a possible solutions, for fresh installs, is no longer
install /etc/sysconfig/nfs and only run nfs-convert when the file
does exist via a ConditionPathExist in the systemd script. 

I'm assuming all I'll I need to do is point the Docker file
to a local nfs-utils version to get that installed...
something similar to:

RUN dnf install -y /path/nfs-utils

--- Additional comment from Steve Dickson on 2019-02-07 18:18:06 UTC ---

(In reply to Steve Dickson from comment #23)
> (In reply to Jan Pazdziora from comment #22)
> 
> I'm assuming all I'll I need to do is point the Docker file
> to a local nfs-utils version to get that installed...
> something similar to:
> 
> RUN dnf install -y /path/nfs-utils

I guess my assumption was wrong

# docker build -t nfs-convert .
Sending build context to Docker daemon 32.03 MB
Step 1/4 : FROM registry.fedoraproject.org/fedora:rawhide
 ---> fcecb29385cc
Step 2/4 : RUN dnf install -y /tmp/nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm
 ---> Running in c7728a15449d
Fedora - Modular Rawhide - Developmental packag 149 kB/s | 2.2 MB     00:14    
Fedora - Rawhide - Developmental packages for t 594 kB/s |  61 MB     01:45    
Last metadata expiration check: 0:00:03 ago on Thu Feb  7 18:13:07 2019.
Can not load RPM file: /tmp/nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm.
Could not open: /tmp/nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm
The command '/bin/sh -c dnf install -y /tmp/nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm' returned a non-zero code: 1

fedora# ll /tmp/nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm
150294 -rw-r--r--. 1 steved steved 390652 Feb  7 13:11 /tmp/nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm

How do I load a local version of nfs-utils?

--- Additional comment from Jan Pazdziora on 2019-02-07 18:29:03 UTC ---

You need to have it copied from the current directory (where Dockerfile is) to the container first:

COPY nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm /
RUN dnf install -y /nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm

The container (in which the build runs) does not have access to your host's /tmp.

--- Additional comment from Steve Dickson on 2019-02-07 18:46:35 UTC ---

(In reply to Jan Pazdziora from comment #25)
> You need to have it copied from the current directory (where Dockerfile is)
> to the container first:
> 
> COPY nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm /
> RUN dnf install -y /nfs-utils-2.3.3-5.rc2.fc30.x86_64.rpm
> 
> The container (in which the build runs) does not have access to your host's
> /tmp.

Thank you!

We will be removing the immutable of /etc/sysconfig/nfs, as requested.