Bug 1443662 - installation problems with repos accessed via NFS
Summary: installation problems with repos accessed via NFS
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 28
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Anaconda Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-19 16:20 UTC by René Genz
Modified: 2019-05-28 19:46 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-28 19:46:41 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
kickstart file (3.26 KB, text/plain)
2017-06-20 00:37 UTC, René Genz
no flags Details

Description René Genz 2017-04-19 16:20:45 UTC
I am not sure against which component the bug report has to be filed.

Description of problem:
From my point of view there are two sub-problems. I think they are closely related, hence I describe them together. I will create an additional bug report with only one of the problems, if you want me to.

(problem 1)
In graphical installation mode local can be accessed via NFS and installation succeeds.
However, the '--opts=nfsvers=3' has to be removed to access repos with NFS.

(problem 2)
Using NFS repositories in manual text mode installation and automated kickstart installation fails.
The KS file is loaded but the communication with the NFS server does not work.
Even the anaconda-ks.cfg file created by the GUI-based installation fails.


Version-Release number of selected component (if applicable):
not sure; as in Fedora-Workstation-netinst-x86_64-26-Alpha-1.7.iso

How reproducible:
100% and easy

Steps to Reproduce:
0. download Fedora-Workstation-netinst-x86_64-26-Alpha-1.7.iso
1. start virtual machine (problem is reproducible on hardware) with ISO file and start graphical installation
2. configure local repos:
- protocol: NFS
- server: <IPv4-address>/dist/f26-x86_64
- NFS mount option: --opts=nfsvers=3
- [Done]
3. repos cannot be loaded
4. set NFS mount option to "" ; click [Done] ; repos can be loaded
### => this is problem 1

5. use 'minimal environment' for installation environment
6. after installation put '/root/anaconda-ks.cfg' on local HTTP server and make it available for reinstallation
7. reinstall virtual machine with same ISO file; kernel boot parameter is:
vmlinuz initrd=initrd.img ks=http://<IPv4-address>/kickstart/anaconda-ks.cfg
8. installation does not start
### => this is problem 2



Actual results:
(only for problem 2)
press Ctrl+Alt+F4 to see the log:
- 1 minute after start of installation it displays at end:
ERR systemd-logind: Failed to start autovt: Connection timed out
- 3 minutes after start of installation it adds another line:
NOTICE kernel:nfs: server <IPv4-address> not responding, still trying
- installation is stuck and never starts

Expected results:
(problem 1)
'--opts=nfsvers=3' should still work out.

(problem 2)
installation should start



Additional info:
The workaround is to access the repos with a different protocol, f.e. HTTP:
Line in kickstart file is:
url --url="http://<IPv4-address>/dist/f26-x86_64"


In Fedora 24 using NFS repos was no problem with GUI and kickstart installation.
The line from kickstart file:
nfs --server <IPv4-address> --dir=/install/dist/f24-x86_64 

In Fedora 26 this does not work.
Our environment for Fedora 24 installation is identical to the environment for Fedora 26. Hence, I assume it is a problem with Fedora 26.

Comment 1 René Genz 2017-04-19 16:25:47 UTC
for the 'additional info' paragraph:
I accidentally cut the '--opts=nfsvers=3' from the end of the line 'nfs server ...'

Comment 2 René Genz 2017-06-16 16:21:55 UTC
retested with Fedora 26 Beta using:
Fedora-Workstation-netinst-x86_64-26_Beta-1.4.iso

"problem 1" is fixed and was apparently caused by me.
In the 'NFS mount options:" input box instead of writing:
--opts=nfsvers=3

I should have written:
nfsvers=3

This works as well.



"problem" 2 is still present.
Maybe this is a related bug 1444335?

Unattended installation using NFS repo is not possible.
To me this violates:
- http://fedoraproject.org/wiki/Fedora_26_Beta_Release_Criteria#Remote_package_sources
(too late for that)
and

- http://fedoraproject.org/wiki/Fedora_26_Final_Release_Criteria#Package_and_installer_sources


I will propose "problem 2" as a blocker bug starting at:
https://qa.fedoraproject.org/blockerbugs

Comment 3 Fedora Blocker Bugs Application 2017-06-16 16:30:32 UTC
Proposed as a Blocker for 26-final by Fedora user sobek using the blocker tracking app because:

 Unattended kickstart installation with accessing repos by NFS is not possible.
The installer loads the kickstart file but cannot download files from the repos.

To me this violates:
http://fedoraproject.org/wiki/Fedora_26_Beta_Release_Criteria#Remote_package_sources
"When using the dedicated installer images, the installer must be able to use HTTP, FTP and NFS repositories as package sources."


We are past 26-beta. Hence using 'f26-final' as milestone and violation is:
http://fedoraproject.org/wiki/Fedora_26_Final_Release_Criteria#Package_and_installer_sources
"The installer must be able to use all supported local and remote package and installer sources."

Comment 4 Adam Williamson 2017-06-19 18:18:52 UTC
Discussed at 2017-06-19 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-06-19/f26-blocker-review.2017-06-19-16.00.html . We agreed that we could do with some more information and independent re-testing of this to be sure.

René, could you clarify exactly what your configuration is and exactly what is not working now? Could you attach the specific kickstart that does not work for you, and expand a bit on the claim that using NFS interactively with the text installer does not work?

Is there some reason why you need to specify 'nfsvers=3' at all when doing the GUI install? Does the install work if you do not specify it at all?

Thanks.

Comment 5 René Genz 2017-06-20 00:37:44 UTC
Created attachment 1289362 [details]
kickstart file

After some more testing the summary from my point of view is:
1st problem (covered by 1443662):
Installation is not working if you do an unattended kickstart-based installation using static IP address configuration and NFS-based repo.


2nd problem (covered by 1444335):
In an unattended kickstart-based installation using static IP address configuration and HTTP-based repo in the %post section you cannot mount NFSv3 shares. mount.nfs tries only NFSv4.2 ; you have to use 'nfsvers=3' mount option to force mounting of NFSv3, but it fails with 'mount.nfs: mount(2): No such device'.





Now answering your questions for this bug.

> René, could you clarify exactly what your configuration is and exactly what is not working now?

With `rsync` I am downloading Fedora repo 'http://.../releases/24/Everything/x86_64/os/' to a local server, SERVER1.
SERVER1 offers the repo via a NFSv3 share.
For testing purpose, in addition SERVER2 mounts the NFSv3 share and offers it via HTTP.

My Fedora clients download packages during installation of Fedora from the NFSv3 share of SERVER1.
If NFS does not work for some reason I can use HTTP.

I am using a virtual machine to test Fedora 26 installation.
I start the VM and load the file:
Fedora-Workstation-netinst-x86_64-26_Beta-1.4.iso

I perform unattended kickstart-based TUI installation like this:
- after loading the ISO select 'Install Fedora 26' and press tab key
- delete with backspace till only 'vmlinuz initrd=initrd.img ' is present
- type 'ks=http://<IPv4 address>/kickstart/anaconda-ks.cfg'
- press enter



>  Could you attach the specific kickstart that does not work for you, and expand a bit on the claim that using NFS interactively with the text installer does not work?

I am sorry for causing confusion.
For me the installation is not working if you do an unattended kickstart-based installation using static IP address configuration and NFS-based repo.

The combination is important.
Installation will work if you modify the above like:
-DHCP and NFS-based repo
-static IP and HTTP-based repo
-DHCP and HTTP-based repo


When the installation does not work it looks like this in the tmux window:
---8<---
Starting installer, one moment...
anaconda 26.21.9-1 ...
 * ...
 * ...
 * ...
 * ...
---8<---

The text 'Starting automated install' does not appear like it should.


The attached kickstart file helps to reproduce both problems. It has been redacted. Placeholders are: SERVERNAME1 SERVERNAME2 NAMECLIENT1 IPv4ADDRESS1 IPv4ADDRESS2 IPv4ADDRESS3 NAMECLIENT1 ROOTPWCRYPTED





> Is there some reason why you need to specify 'nfsvers=3' at all when doing the GUI install? Does the install work if you do not specify it at all?

It was a habit from previous Fedora installation environments.
It is not necessary with Fedora 26 Beta in GUI, TUI, and unattended kickstart installation for the repo. Leaving the NFS mount options blank works fine.

Comment 6 Adam Williamson 2017-06-26 18:56:18 UTC
So I'm working on reproducing this. So far I have not; I used a kickstart based on Rene's, but shortened, the salient bits being:

network --bootproto static --ip 192.168.1.55 --netmask 255.255.255.0
nfs --server 192.168.1.5 --dir /mnt/temp

and it worked fine. However, the NFS server I'm using is a Fedora 26 box, and so obviously is an NFSv4 server. I'll try and set up an NFSv3-only server, somehow, and test again.

Comment 7 Adam Williamson 2017-06-27 00:00:29 UTC
Nope, just tried with my NFS server configured to only serve v3 (via /etc/nfs.conf) and still cannot reproduce this.

Comment 8 Adam Williamson 2017-06-27 00:31:35 UTC
Discussed at 2017-06-26 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-06-26/f26-blocker-review.2017-06-26-16.03.html . We agreed to delay decision on this again pending attempts to reproduce it independently and fully identify the circumstances under which the bug happens.

Comment 9 Matthew Miller 2017-06-28 23:12:21 UTC
I'm -1 to this as a blocker. Installation over NFS is important, but it seems to be only happening in a narrow case and Adam can demonstrate it working in a similar situation -- and René, you have a workaround with by changing to either DHCP or HTTP, right?

(What happens if you use your DHCP server to allocate fixed addresses?)

Comment 10 Adam Williamson 2017-06-29 18:59:44 UTC
Discussed at 2017-06-29 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-06-29/f26-blocker-review.2017-06-29-16.00.html . Rejected as a blocker: the information we have so far indicates this is some kind of corner case, not a general problem that's likely to affect many people, and the reporter has several workarounds they can use.

Comment 11 René Genz 2017-07-03 17:43:48 UTC
Matthew, I agree. It seems like I am hitting a special code path.

Today, I found a workaround:
- use NFSv3-based repos with static IPs for OS installation and
- wait 17 to 20 minutes till 'Starting automated install' is shown and installation starts
With this workaround mounting NFSv3 shares in %post works!

The delay is due to 'e2fsck'. But there is no hard disk or CPU activity.
The delay occurs only with the above combination. For all other combinations 'e2fsck' is done within a second.
I did not find the workaround earlier, because I killed the installations after 5 to 10 minutes.



For reference the other combination's results and why I cannot use them.
HTTP-based repos and (static IPs OR DHCP) for OS installation starts installation without delay.
But it breaks NFSv3 share mounting in %post. This is a requirement for me.
The '--noIPv6' option for static IPs makes no difference.
I have not tested '--noIPv6' option with DHCP.


NFS-based repos and DHCP-served dynamic IPs for OS installation start installation without delay.
Mounting NFSv3 shares in %post works. But our policy is to assign locally static IPs for Fedora-computers, no DHCP server involved.
I have not tested '--noIPv6' option with DHCP.


I tested DHCP-served static IP with HTTP-based repos for OS installation.
It starts without delay.
Mounting NFSv3 shares in %post works.
I have not tested '--noIPv6' option with DHCP.
Unfortunately, the specified DHCP-served static IP is not used. A DHCP-served dynamic IP is used. Maybe I am doing something wrong.
If you are interested, I can try again. However, I would rather get my F26 installation setup into shape. :)

Comment 12 René Genz 2018-03-04 01:14:59 UTC
I tested Fedora 28 with Fedora-Workstation-netinst-x86_64-28-20180302.n.0.iso
and packages from current date of posting with automated kickstart installation on a virtual machine.
My 2x2 test matrix consists of:
IPv4 set to : static <> DHCP
repository for installation served via : NFS <> HTTP

Result:
- mounting NFSv3 share in %post works for all use cases
- use cases DHCP/NFS, DHCP/HTTP, and static/HTTP have no delay before start of installation
- use case static/NFS' delay before start of installation has been reduced from 17 to 4 minutes



During the delay you can switch to the 'program-log' pane with Alt+Tab; in it's last line you can read:
<time> INF program: Running [6] e2fsck -V ...

After 4 minutes have passed these lines are appended:
<time> INF program: stdout[6]:
<time> INF program: stderr[6]: e2fsck 1.43.8 (1-Jan-2018)
  Using EXT2FS Library version 1.43.8, 1-Jan-2018

<time> INF program: ...done [6] (exit code: 0)

This is when in the 'main' pane the installation starts.

Comment 13 René Genz 2018-03-17 22:42:44 UTC
With Fedora-Workstation-netinst-x86_64-28-20180315.n.0.iso and Fedora-Workstation-netinst-x86_64-28-20180316.n.1.iso the problem of "failing to mount NFSv3 shares in %post with static IP and NFSv3 repo in kickstart file" is happening again.
I have not tested the other cases.

Executing the command in %post (NFSSERVER, NFSSHARE, MOUNTPOINT, IPV4-1, IPv4-2 are dummies):
mount -v -t nfs -o nolock ${NFSSERVER}:${NFSSHARE} "${MOUNTPOINT}"
fails and creates the output:
Running the command without  results in output:
mount.nfs: mount(2): Protocol not supported
mount.nfs: mount(2): Protocol not supported
mount.nfs: mount(2): Protocol not supported
mount.nfs: Protocol not supported
mount.nfs: timeout set for Sat Mar 17 21:33:38 2018
mount.nfs: trying text-based options 'nolock,vers=4.2,addr=${IPv4-1},clientaddr=${IPv4-2}'
mount.nfs: trying text-based options 'nolock,vers=4,minorversion=1,addr=${IPv4-1},clientaddr=${IPv4-2}'
mount.nfs: trying text-based options 'nolock,vers=4,addr=${IPv4-1},clientaddr=${IPv4-2}'


The workaround is to use the 'nfsvers=3' option:
mount -v -t nfs -o nolock,nfsvers=3 ${NFSSERVER}:${NFSSHARE} "${MOUNTPOINT}"
the output is:
mount.nfs: trying ${IPv4-1} prog 100003 vers 3 prot TCP port 2049
mount.nfs: trying ${IPv4-1} prog 100005 vers 3 prot UDP port 635
mount.nfs: timeout set for Sat Mar 17 22:12:24 2018
mount.nfs: trying text-based options 'nolock,nfsvers=3,addr=${IPv4-1}'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: prog 100005, trying vers=3, prot=17

Comment 14 Ben Cotton 2019-05-02 21:59:30 UTC
This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 15 Ben Cotton 2019-05-28 19:46:41 UTC
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.