Bug 1872056 - Booting Server DVD then selecting a mirrorlist repository does not work
Summary: Booting Server DVD then selecting a mirrorlist repository does not work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 33
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Jiri Konecny
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: openqa AcceptedBlocker
Depends On:
Blocks: BetaBlocker, F33BetaBlocker 1882367
TreeView+ depends on / blocked
 
Reported: 2020-08-24 22:36 UTC by Adam Williamson
Modified: 2020-09-24 11:32 UTC (History)
10 users (show)

Fixed In Version: anaconda-33.25.2-2 anaconda-33.25.2-3.fc33
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1882367 (view as bug list)
Environment:
Last Closed: 2020-09-11 20:35:34 UTC
Type: Bug


Attachments (Terms of Use)
anaconda.log (20.87 KB, text/plain)
2020-08-24 22:40 UTC, Adam Williamson
no flags Details
dbus.log (2.63 KB, text/plain)
2020-08-24 22:41 UTC, Adam Williamson
no flags Details
dnf.librepo.log (2.32 KB, text/plain)
2020-08-24 22:41 UTC, Adam Williamson
no flags Details
packaging.log (16.65 KB, text/plain)
2020-08-24 22:41 UTC, Adam Williamson
no flags Details
program.log (1.70 KB, text/plain)
2020-08-24 22:41 UTC, Adam Williamson
no flags Details
syslog (567.13 KB, text/plain)
2020-08-24 22:42 UTC, Adam Williamson
no flags Details

Description Adam Williamson 2020-08-24 22:36:05 UTC
Since Fedora-33-20200822.n.0 , the openQA test install_mirrorlist_graphical is failing. What this test does is boot the Server DVD, then go to INSTALLATION SOURCE, select "On the network:", enter a mirrorlist URL, then change "URL type:" to "mirrorlist" and press Done.

When it does this, the intaller returns to the hub, but the Installation Source entry now shows "Error setting up base repository".

This was working until Fedora-33-20200822.n.0 , which is when anaconda-33.25.2-1.fc33 landed. Previously 33 had anaconda-33.25-1.fc33 . So this seems to have broken between 33.25 and 33.25.2.

This doesn't seem to affect just entering a repository URL directly - that seems to work fine. Only mirrorlist URLs are affected for some reason.

Will attach logs from a recent failure.

Proposing as a Beta blocker as a violation of Basic criterion "When using a release-blocking dedicated installer image, the installer must be able to use either HTTP or FTP repositories (or both) as package sources. Release-blocking network install images must default to a valid publicly-accessible package source" , when the 'repository' in question is a mirrorlist and not a direct repo URL.

Comment 1 Adam Williamson 2020-08-24 22:40:54 UTC
Created attachment 1712443 [details]
anaconda.log

Comment 2 Adam Williamson 2020-08-24 22:41:07 UTC
Created attachment 1712444 [details]
dbus.log

Comment 3 Adam Williamson 2020-08-24 22:41:20 UTC
Created attachment 1712445 [details]
dnf.librepo.log

Comment 4 Adam Williamson 2020-08-24 22:41:34 UTC
Created attachment 1712446 [details]
packaging.log

Comment 5 Adam Williamson 2020-08-24 22:41:53 UTC
Created attachment 1712447 [details]
program.log

Comment 6 Adam Williamson 2020-08-24 22:42:15 UTC
Created attachment 1712448 [details]
syslog

Comment 7 Kamil Páral 2020-08-28 12:16:53 UTC
Accepted as a Beta Blocker:
https://pagure.io/fedora-qa/blocker-review/issue/35

> AGREED AcceptedBetaBlocker Accepted as a violation of Basic criterion "When using a release-blocking dedicated installer image, the installer must be able to use either HTTP or FTP repositories (or both) as package sources. Release-blocking network install images must default to a valid publicly-accessible package source" [1] , when the 'repository' in question is a mirrorlist and not a direct repo URL. [1] https://fedoraproject.org/wiki/Basic_Release_Criteria#Remote_package_sources

Comment 8 Adam Williamson 2020-08-31 22:02:21 UTC
This part of packaging.log looks like the problem to me:

09:22:47,488 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='anaconda-0', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='MIRRORLIST', url='https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64')
09:22:47,489 DBG packaging: getting release version from tree at  (33)
09:22:47,489 DBG packaging: using treeinfo release version of 33
09:22:47,489 DBG packaging: No base repository found in treeinfo file. Using installation tree root.
09:22:47,489 DBG packaging: releasever from  is 33
09:22:47,489 DBG packaging: Adding new treeinfo repository: Server enabled: True
09:22:47,490 INF packaging: added repo: 'Server' - file:///run/install/sources/mount-0000-cdrom

somewhere in DNFPayload._add_repo we're going from a mirrorlist repo with mirrorlist URL https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64 to what looks like trying to set up a local mount repo with a URI of file:///run/install/sources/mount-0000-cdrom . That doesn't look right at all.

I suspect the payload-related changes Jiri Konecny made between 33.25-1 and 33.25.2-1 are the problem here - probably one of:

a57be7d30897ecf301de673e41d1af975b4f593b - Fix issue that treeinfo repositories were never disabled
1c03016c3906a458a74262eaaffbc732d70d5281 - Move parts together in the DNF repo
0788d3fb8a696fa83a818148f08eb439123cc389 - Get rid of add_disable_repo

Jiri, can you please take a look?

Comment 9 Adam Williamson 2020-08-31 22:22:58 UTC
Here's the equivalent section from a test with anaconda-33.25-1.fc33:

15:21:13,175 INF packaging: Configuring the base repo
15:21:13,205 INF packaging: Enabled 'updates'
15:21:13,205 INF packaging: Disabled 'updates-testing'
15:21:13,208 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='anaconda-0', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='MIRRORLIST', url='https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64')
15:21:13,208 DBG packaging: getting release version from tree at  (rawhide)
15:21:13,208 DBG packaging: using treeinfo release version of rawhide
15:21:13,208 DBG packaging: No base repository found in treeinfo file. Using installation tree root.
15:21:13,208 DBG packaging: releasever from  is rawhide
15:21:13,209 INF packaging: added repo: 'anaconda' - https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64
15:21:13,209 DBG dnf: repo: downloading from remote: anaconda
15:21:17,057 INF packaging: enabled repo: 'anaconda' - https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64 and got repomd

Comment 10 Adam Williamson 2020-08-31 22:30:50 UTC
Ah - looking at those logs I actually think https://github.com/rhinstaller/anaconda/commit/6f27de8a38cc7900bb35a4fac4ec258f50207468 is the offending commit here. Things go wrong in 33.25.2 when we get to "Adding new treeinfo repository: Server enabled: True", which is in `_load_treeinfo_repositories` (previously `_add_treeinfo_repositories`), and that commit changes anaconda from doing that *only on the first payload reset* to doing it on *every* payload reset. `_load_treeinfo_repositories` is doing something wrong in this case. I do note this check:

        if base_repo_url is not None:

which is questionable here, because I believe we are passing in `base_repo_url` as an empty string on this route.

Comment 11 Adam Williamson 2020-08-31 23:51:13 UTC
Hmm, no, that check isn't the issue - I patched update_base_repo to pass `base_repo_url or None` but it doesn't help. I think what's going on here is that we're trying to reload the repo from the DVD, but it has been umounted in the meantime. There's some key lines in syslog that we don't see in packaging.log. Here's the whole flow around the *first* base repo setup:

syslog:

12:50:47,514 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... mount -t iso9660 -o ro /dev/sr0 /run/install/sources/mount-0000-cdrom
12:50:47,532 WARNING org.fedoraproject.Anaconda.Modules.Payloads:DEBUG:program:Return code: 0
12:50:47,532 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:anaconda.modules.payloads.source.cdrom.initialization:using CD-ROM device sr0 mounted at /run/install/sources/mount-0000-cdrom

packaging.log:

12:50:48,883 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='BASEURL', url='file:///run/install/sources/mount-0000-cdrom')
12:50:48,884 DBG packaging: retrieving treeinfo from file:///run/install/sources/mount-0000-cdrom (proxy: None ; ssl_verify: True)
12:50:48,885 INF packaging: Trying to download '.treeinfo'
12:50:48,887 DBG packaging: Retrieved '.treeinfo' from file:///run/install/sources/mount-0000-cdrom
12:50:48,891 DBG packaging: getting release version from tree at file:///run/install/sources/mount-0000-cdrom (33)
12:50:48,891 DBG packaging: using treeinfo release version of 33
12:50:48,892 DBG packaging: No base repository found in treeinfo file. Using installation tree root.
12:50:48,892 DBG packaging: releasever from file:///run/install/sources/mount-0000-cdrom is 33
12:50:48,893 INF packaging: added repo: 'anaconda' - file:///run/install/sources/mount-0000-cdrom
12:50:48,928 DBG dnf: repo: downloading from remote: anaconda
12:50:49,105 INF packaging: enabled repo: 'anaconda' - ['file:///run/install/sources/mount-0000-cdrom'] and got repomd
12:50:49,106 DBG dnf: repo: using cache for: anaconda
12:50:49,106 DBG packaging: repo anaconda: _sync_metadata success from ['file:///run/install/sources/mount-0000-cdrom']

syslog again:

12:51:42,537 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... umount /run/install/sources/mount-0000-cdrom

now back to packaging.log:

08:51:44,613 DBG packaging: Adding new treeinfo repository: Server enabled: True
08:51:44,614 INF packaging: added repo: 'Server' - file:///run/install/sources/mount-0000-cdrom
08:51:44,615 DBG dnf: repo: downloading from remote: Server
08:51:44,616 DBG dnf: error: Curl error (37): Couldn't read a file:// file for file:///run/install/sources/mount-0000-cdrom/repodata/repomd.xml [Couldn't open file /run/install/sources/mount-0000-cdrom/repodata/repomd.xml] (file:///run/install/sources/mount-0000-cdrom/repodata/repomd.xml).

so we mount the repo, set it up successfully, *then unmount it*, then `_load_treeinfo_repositories` runs again and tries to load it again, but without remounting it first.

Comment 12 Adam Williamson 2020-08-31 23:52:29 UTC
oh, the timezone gets set in the middle there or something, which is why we suddenly jump "back" four hours. But that is the order stuff actually happens in, I'm fairly sure.

Comment 13 Adam Williamson 2020-08-31 23:58:04 UTC
ah, and interestingly, `DNFPayload.update_base_repo` has a `checkmount` arg which looks like it makes sure something like the DVD repo is mounted if necessary, and pyanaconda/ui/gui/spokes/installation_source.py `SourceSpoke.apply` does this:

        if source_changed or repo_changed or self._error:
            payloadMgr.restart_thread(self.payload, checkmount=False)

...and `payloadMgr.restart_thread` winds up doing `payload.update_base_repo` with that `checkmount` value. So I bet that's where we're at here.

Comment 14 Adam Williamson 2020-09-01 00:02:42 UTC
yeah, I added a log line and confirmed that during the *first* call to `update_base_repo` (where it works) checkmount is True, but during the *second* call (where it fails) it's False.

Comment 15 Jiri Konecny 2020-09-03 13:12:52 UTC
PR: https://github.com/rhinstaller/anaconda/pull/2826

Comment 16 Jiri Konecny 2020-09-03 13:26:33 UTC
This is an old issue but it wasn't a problem before our recent change. The old logic has loaded all additional repositories from the treeinfo just once from the first repository. We have changed the logic to load it every time the repository has changed, however, I forget to change also the metadata loading so it was not cleared and old repository was used as the metadata source. Clear the metadata properly to fix this issue.

Comment 17 Fedora Update System 2020-09-07 13:12:37 UTC
FEDORA-2020-daac40f3d3 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

Comment 18 Fedora Update System 2020-09-07 21:45:03 UTC
FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-daac40f3d3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Adam Williamson 2020-09-08 16:02:49 UTC
We should be able to check the fix for this in the next Rawhide compose. Testing the F33 update itself is a bit tricky as it requires building a Server DVD...

Comment 20 Fedora Update System 2020-09-09 13:46:48 UTC
FEDORA-2020-daac40f3d3 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

Comment 21 Adam Williamson 2020-09-09 19:30:11 UTC
So Rawhide composes are failing, but they're getting far enough to produce a Server DVD. Here's the latest:
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200909.n.0/compose/Server/x86_64/iso/Fedora-Server-dvd-x86_64-Rawhide-20200909.n.0.iso

I'm downloading that to verify the fix now.

Comment 22 Adam Williamson 2020-09-09 20:48:04 UTC
Unfortunately that doesn't boot, so can't test. Will try again with the next one.

Comment 23 Fedora Update System 2020-09-10 15:47:33 UTC
FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-daac40f3d3`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 24 Adam Williamson 2020-09-11 06:39:32 UTC
Rawhide latest compose succeeded, and test passed: https://openqa.fedoraproject.org/tests/661773
so we can confirm the fix for this looks good.

Comment 25 Fedora Update System 2020-09-11 20:35:34 UTC
FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.