Bug 1224048

Created attachment 1028550 [details]
journal

Created attachment 1028551 [details]
anaconda.log

Created attachment 1028552 [details]
dnf.log

Created attachment 1028553 [details]
dnf.rpm.log

Created attachment 1028554 [details]
hawkey.log

Created attachment 1028555 [details]
ifcfg.log

Created attachment 1028556 [details]
packaging.log

Created attachment 1028557 [details]
program.log

Created attachment 1028558 [details]
storage.log

Created attachment 1028559 [details]
syslog

Created attachment 1028560 [details]
X.log

This seems to be the error:

09:18:36,144 INFO packaging: Downloading packages.
09:18:36,153 INFO packaging: Downloading packages finished.
09:18:37,573 ERR packaging: The transaction process has ended abruptly
09:18:37,574 INFO packaging: Transaction check error:
  installing package authconfig-6.2.10-6.fc22.x86_64 needs 3MB on the / filesystem
  installing package passwd-0.79-5.fc22.x86_64 needs 4MB on the / filesystem
  installing package bash-completion-1:2.1-6.20141110git52d8316.fc22.noarch needs 7MB on the / filesystem
  installing package sudo-1.8.12-1.fc22.x86_64 needs 10MB on the / filesystem
  installing package grubby-8.40-1.fc22.x86_64 needs 10MB on the / filesystem
  installing package e2fsprogs-1.42.12-4.fc22.x86_64 needs 15MB on the / filesystem
  installing package xfsprogs-3.2.2-1.fc22.x86_64 needs 19MB on the / filesystem
  installing package uboot-tools-2015.01-3.fc22.x86_64 needs 20MB on the / filesystem
  installing package rootfiles-8.1-17.fc21.noarch needs 20MB on the / filesystem

Error Summary
-------------
Disk Requirements:
  At least 20MB more space needed on the / filesystem.


Even though anaconda claims earlier:

09:18:29,672 INFO anaconda: fs space: 819.2 MiB  needed: 728.24 MiB


I guess anaconda did not check the real disk free space, but just the partition size. The filesystem metadata will eat some of that, so the real free space is smaller, and it did not fit.

So, I tested with the Workstation package set, using netinst. Anaconda log claims it needs 4.39 GiB of free space, and that's the partition size it checks. However, it completely neglects that the files need to be downloaded as well, and they are saved on the root partition. For the current Workstation package set, that's 1.18 GiB. So in total, we need at least 5.57 GiB of space.

I have verified that it still crashes with 5.5 GiB partition. However, it also crashes with 5.6 GiB partition. I guess that's the filesystem metadata which are also not counted in. With 5.7 GiB partition, it finally works.

So, this bug will happen only if the user sets the partition size above the minimum size allowed by anaconda but below the real size needed by the chosen package set. If using netinst, that sizes can differ substantially (one or more GiB), based on the chosen package set. And some little more overhead is needed for filesystem metadata.

The reproducer is very simple:
1. boot netinst, if you want to hit this very easily, or DVD, if you want to make it harder
2. set a default partitioning and pick some package set
3. see anaconda log for "fs space: xxxx  needed: xxxx" line
4. go to custom partitioning (you might need to hit autopart button again), and set root partition size to be above what's needed, but just by a slight margin (or a bigger margin, if using netinst and having lots of packages to download).
5. start installation
6. while downloading packages, you can see how much space the downloaded packages occupy, you can use it in your next attempt
7. see anaconda hard reboot once it is starting package installation phase

I guess this should get discussed during the blocker review. The violated criteria potentially include:
"When using the custom partitioning flow, the installer must be able to: 
 Reject or disallow invalid disk and volume configurations without crashing. "
https://fedoraproject.org/wiki/Fedora_22_Beta_Release_Criteria#Custom_partitioning
where having a too small partition should be rejected, but it isn't, and
"The installer must be able to create and install to any workable partition layout using any file system and/or container format combination offered in a default installer configuration. "
https://fedoraproject.org/wiki/Fedora_22_Final_Release_Criteria#Disk_layouts
where the explanation note says:
"This is a huge catch-all criterion and it's subject to a lot of on-the-fly interpretation. Broadly what it's 'meant to mean' is that you should be able to do anything sane that the Installation Destination spoke attempts to let you do, without the installer exploding or failing."


I have tested this in F21 (using netinst to install Workstation, using 5000 MiB for 4437 MiB Workstation package set), and the problem is not there - at least not the package download problem, there still might be the slight issue with filesystem metadata. The difference is that in F21 yum was used, and it did not pre-download all packages in advance. It downloaded just a single RPM into ramdisk (/tmp/yum.cache/<reponame>/packages), installed it, and then removed it and went to the next one. Therefore it did not exhaust the root partition disk space. F22 with dnf, otoh, pre-downloads all RPMs in advance onto the root partition and therefore substantially decreases the free space size.


So it is a regression from F21. OTOH, this issue is very hard to trigger using DVD (and not possible to trigger at all using Live, I guess), only netinst is affected. And the user must be trying to be *very* conservative with the assigned disk space (which might happen if somebody is trying to just test Fedora, instead of installing it as a production system).

So far I have reproduced this only in manual partitioning, but since the computations are not correct, I assume it will be also possible to find a way to trigger this also in the guided partitioning - by having just the right amount of free space, the root partition might get created with just not large enough size.

Discussed at today's go/no-go meeting [1].

This bug was rejected as F22 Final Blocker, but accepted as F23 Final Blocker -  This bug does violate the criteria, but the cases where a user can hit this are slim. Rejected as a blocker for F22, please document in Common Bugs. This has been accepted as a blocker for F23 so it doesn't get forgotten.

[1] http://meetbot.fedoraproject.org/fedora-meeting-2/2015-05-22/f22_final_gono-go_meeting_-_2.2015-05-22-17.01.log.txt

FWIW, I think I have seen this or something similar twice with VM installation from netinst (the result both times was apparent hang with black screen). But I didn't investigate or try to reproduce.

This fix will go to both anaconda and blivet:

Blivet PR: https://github.com/rhinstaller/blivet/pull/217
Anaconda PR: https://github.com/rhinstaller/anaconda/pull/358

python-blivet-1.12.5-1.fc23 anaconda-23.19.5-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-fe57978cc8

anaconda-23.19.5-1.fc23, python-blivet-1.12.5-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update python-blivet anaconda'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-fe57978cc8

Should be fixed in Final TC1, can you confirm? Thanks!

anaconda-23.19.5-1.fc23, python-blivet-1.12.5-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

I'm still having some issues with F23 TC1. There were definitely big improvements, and the download size now seems to be included in the minimum partition size computation for netinst. However, if I set the partition size very close to the minimum threshold (either I lower the size step by step until anaconda complains, and then increase it a bit; or I get very unlucky and just pick a value close to the limit), anaconda still crashes for me. I wasn't able to reproduce it with Workstation netinst, but I was able to do it with Server netinst.

With Server netinst, minimal installation crashes for me when I set root partition (using LVM and XFS, the defaults) to have 850 MiB. Server package set installation crashes when I set root partition to have 1.42 GiB.

There is some weirdness in logs. I use `grep -r 'fs space' /tmp` to monitor how much size anaconda wants for different package sets. And for some reason, it changes in time:
./anaconda.log:16:19:00,216 INFO anaconda: fs space: 8.25 GiB  needed: 1.83 GiB
./anaconda.log:16:19:04,101 INFO anaconda: fs space: 8.25 GiB  needed: 1.07 GiB
./anaconda.log:16:19:04,115 INFO anaconda: fs space: 8.25 GiB  needed: 1.07 GiB
./anaconda.log:16:19:04,129 INFO anaconda: fs space: 8.25 GiB  needed: 1.07 GiB
./anaconda.log:16:19:43,749 INFO anaconda: fs space: 822.56 MiB  needed: 778.89 MiB
./anaconda.log:16:19:43,770 INFO anaconda: fs space: 822.56 MiB  needed: 778.89 MiB
./anaconda.log:16:19:45,162 INFO anaconda: fs space: 822.56 MiB  needed: 778.89 MiB
./anaconda.log:16:19:45,179 INFO anaconda: fs space: 822.56 MiB  needed: 778.89 MiB

The first number, 1.83 GiB, is the default Server package set anaconda chooses by default. The second number, 1.07 GiB, is when I switch to minimal set. But the third number, 778.89 MiB, is when I go to manual partitioning and set root partition to be 850 MiB (it shows up in the log as 822.56 MiB, probably excluding fs metadata size?). I'm very confused how it is possible that for the same package set (minimal), it first needs 1.07 GiB, but then it needs just 778.89 MiB. Maybe it's a bug somewhere?

Anyway, it crashes with 850 MiB partition. But it does not crash with 900 MiB partition, which is still under the original 1.07 GiB requirement. So the crash does not occur any time you go under the original requested value, but only when you get very close to the limit.

Similarly with Server install:
./anaconda.log:16:52:21,018 INFO anaconda: fs space: 8.25 GiB  needed: 1.83 GiB
./anaconda.log:16:53:03,721 INFO anaconda: fs space: 993.28 MiB  needed: 1.37 GiB
<snip>
./anaconda.log:16:54:22,183 INFO anaconda: fs space: 1.38 GiB  needed: 1.37 GiB

Originally it wanted 1.83 GiB. After I selected a smaller partition, it started to require just 1.37 GiB. When I selected 1.42 GiB (which showed up as 1.38 GiB in the log), the installation proceeded and then crashed. With somewhat higher root partition sizes (e.g. 1.5 GiB), it does not crash.

Interestingly enough, when I try the same approach with Workstation netinst, using Workstation package set, it requires 5.63 GiB all the time, no matter what root partition size I set, and the threshold *does not decrease*.

Created attachment 1081004 [details]
minimal - anaconda.log

Created attachment 1081006 [details]
minimal - packaging.log

Created attachment 1081007 [details]
minimal - program.log

Created attachment 1081008 [details]
minimal - storage.log

Created attachment 1081009 [details]
minimal - syslog

Created attachment 1081010 [details]
server - anaconda.log

Created attachment 1081011 [details]
server - packaging.log

Created attachment 1081012 [details]
server - program.log

Created attachment 1081013 [details]
server - storage.log

Created attachment 1081014 [details]
server - syslog

Marking for blocker bug (or freeze exception) re-evaluation. Since a large portion of the issue was resolved (the downloaded size is now included in the minimum partition size computation), the risk for our users has been reduced greatly. You either need to get very unlucky when picking partition size, or you need to try to downsize the partition to the bone, not leaving 100MiB+ extra (which is a valid approach, of course, but hopefully not that common). You can still end up with previous disk layout destroyed and no new system installed, but the risk is considerably lower and I think that deserves a new discussion.

Discussed during the 2015-10-12 blocker review meeting [1]:

This bug was rejected as a blocker for F23 final because after recent changtes, it's no longer common enough or easy enough to hit to justify blocking the release of F23 final. That being said, a tested fix would be considered after freeze.

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-10-12/f23-blocker-review.2015-10-12-16.01.html

At the same meeting, this was accepted as a freeze exception.

PR: https://github.com/rhinstaller/anaconda/pull/423

It should be much more better than now it is.

Also this patch is related to this problem but I don't think it's now time to do so big change to F23. What do you think Kamil?
https://github.com/rhinstaller/anaconda/pull/424

It depends how risky you consider this change to be. It's already accepted as a freeze exception, but that doesn't mean we have to take it. We eliminated this problem to be a very corner case, so unless you consider this to be "very safe", I'd rather not push an anaconda patch into our current RC composes.

OK then, I will use only the main patch and the second one will go to master only.

When I just tried install F23 i686 to 30 GB ext4 fs (+ /boot and /home on other partitions), kickstart w. Server-netinst installation from NFS repos (main + updates + both rpmfusion repos), I ran into the same problem.
With nokill I saw in anaconda.log message:
"INFO anaconda: fs space: 25,49 GiB  needed: 2,79 GiB".

After a severe reduction in the number of installed packages (from ~2520 to ~1880) I was able to complete installation. What was interesting, root fs was only 16% full, there was 25 GB of free space. And when I manually installed the remaining packages which was in original kickstart, then root fs was 34% full and there was 19 GB of free space!
IMO anaconda calculates the necessary space completely wrong.

Hello Frantisek,

I think there will be another problem. When you are using kickstart automatic installation the space requirement checks are disabled so anaconda didn't calculate nothing for you or it shouldn't.

To help you I need logs from the installation '/tmp' mainly packaging.log, anaconda.log, storage.log and program.log. It would nice to have dnf.log too.

Thank you for your bug report.

(In reply to Frantisek Hanzlik from comment #39)
> With nokill I saw in anaconda.log message:
> "INFO anaconda: fs space: 25,49 GiB  needed: 2,79 GiB".

Unless you also see a message like this:
" At least 20MB more space needed on the / filesystem. "
then it's likely to be a different problem.

(In reply to Kamil Páral from comment #41)
> (In reply to Frantisek Hanzlik from comment #39)
> > With nokill I saw in anaconda.log message:
> > "INFO anaconda: fs space: 25,49 GiB  needed: 2,79 GiB".
> 
> Unless you also see a message like this:
> " At least 20MB more space needed on the / filesystem. "
> then it's likely to be a different problem.

Hello Jiri a Kamil,

you are right, sorry for noise. After inspecting log files again, I found the probable cause of the anaconda crash in 'packaging.log', which ends with:
**********************
...
22:52:47,201 INFO packaging: selected group: electronic-lab
...
22:52:53,331 INFO packaging: selected group: system-tools
22:52:53,339 INFO packaging: selected group: text-internet
22:52:53,484 INFO packaging: selected group: web-server
22:52:53,491 INFO packaging: selected group: window-managers
22:52:53,498 INFO packaging: selected package: 'xorg-x11-resutils'
22:52:53,505 INFO packaging: selected package: 'wine'
22:52:53,512 INFO packaging: selected package: 'chrony'
22:52:53,520 INFO packaging: selected package: 'xorg-x11-drivers'
22:52:53,527 INFO packaging: kernel: selected kernel-PAE
22:52:54,030 INFO packaging: Sufficient mountpoints found: {'/mnt/sysimage/home': Size('376,45 GiB'), '/mnt/sysimage': Size('27,83 GiB')}
22:52:54,031 INFO packaging: 2681 packages selected totalling 11,72 GiB
22:52:54,074 INFO packaging: Sufficient mountpoints found: {'/mnt/sysimage/home': Size('376,45 GiB'), '/mnt/sysimage': Size('27,83 GiB')}
22:52:54,326 INFO packaging: Downloading packages.
22:52:54,368 INFO packaging: Downloading packages finished.
22:53:12,573 ERR packaging: The transaction process has ended abruptly
22:53:12,573 INFO packaging: Kontrola transakce selhala:
  file /usr/bin/admsCheck conflicts between attempted installs of mot-adms-2.3.4-1.fc23.i686 and qucs-0.0.18-4.fc23.i686
  file /usr/bin/admsXml conflicts between attempted installs of mot-adms-2.3.4-1.fc23.i686 and qucs-0.0.18-4.fc23.i686

Přehled chyb
------------
**********************
Thus anaconda was evidently crashed due to conflicts between attempted packages.

Frantisek, that problem seems to being resolved here:
https://bodhi.fedoraproject.org/updates/FEDORA-2016-f2ae801867

Once that update is pushed stable and the mirrors pull it, you should be able to install mot-adms and qucs simultaneously. Until then, you need to perform your installation without one of these packages in the installation set. Sorry for troubles.

(In reply to Kamil Páral from comment #43)
> ...
> Until then, you need to perform your installation without one of these
> packages in the installation set.

Exactly what I did.

But, another suggestion: it seems that the F23 installer had/has enormous number of problems (I personally came across several in kickstart install: nonworking %include, python-blivet bugs and others).
Are somewhere, or will be released correcting netinst images? I think it would be very good - would not have to wait for another round of tests and repairs for F24, and lot of Fedora users could use them now (IMO lot of Fedora users, me include, still wait for F23 rpmfusion repo release, and will be install F23 later).
Thanks for consideration.

You can find something here, but it's completely unofficial, they are not well tested, it's only Lives, and I don't know who creates them:
https://alt.fedoraproject.org/pub/alt/live-respins/

No official support for respins is coming in the nearest future, sorry.

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.