Hide Forgot
Since Fedora-33-20200822.n.0 , the openQA test install_mirrorlist_graphical is failing. What this test does is boot the Server DVD, then go to INSTALLATION SOURCE, select "On the network:", enter a mirrorlist URL, then change "URL type:" to "mirrorlist" and press Done. When it does this, the intaller returns to the hub, but the Installation Source entry now shows "Error setting up base repository". This was working until Fedora-33-20200822.n.0 , which is when anaconda-33.25.2-1.fc33 landed. Previously 33 had anaconda-33.25-1.fc33 . So this seems to have broken between 33.25 and 33.25.2. This doesn't seem to affect just entering a repository URL directly - that seems to work fine. Only mirrorlist URLs are affected for some reason. Will attach logs from a recent failure. Proposing as a Beta blocker as a violation of Basic criterion "When using a release-blocking dedicated installer image, the installer must be able to use either HTTP or FTP repositories (or both) as package sources. Release-blocking network install images must default to a valid publicly-accessible package source" , when the 'repository' in question is a mirrorlist and not a direct repo URL.
Created attachment 1712443 [details] anaconda.log
Created attachment 1712444 [details] dbus.log
Created attachment 1712445 [details] dnf.librepo.log
Created attachment 1712446 [details] packaging.log
Created attachment 1712447 [details] program.log
Created attachment 1712448 [details] syslog
Accepted as a Beta Blocker: https://pagure.io/fedora-qa/blocker-review/issue/35 > AGREED AcceptedBetaBlocker Accepted as a violation of Basic criterion "When using a release-blocking dedicated installer image, the installer must be able to use either HTTP or FTP repositories (or both) as package sources. Release-blocking network install images must default to a valid publicly-accessible package source" [1] , when the 'repository' in question is a mirrorlist and not a direct repo URL. [1] https://fedoraproject.org/wiki/Basic_Release_Criteria#Remote_package_sources
This part of packaging.log looks like the problem to me: 09:22:47,488 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='anaconda-0', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='MIRRORLIST', url='https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64') 09:22:47,489 DBG packaging: getting release version from tree at (33) 09:22:47,489 DBG packaging: using treeinfo release version of 33 09:22:47,489 DBG packaging: No base repository found in treeinfo file. Using installation tree root. 09:22:47,489 DBG packaging: releasever from is 33 09:22:47,489 DBG packaging: Adding new treeinfo repository: Server enabled: True 09:22:47,490 INF packaging: added repo: 'Server' - file:///run/install/sources/mount-0000-cdrom somewhere in DNFPayload._add_repo we're going from a mirrorlist repo with mirrorlist URL https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64 to what looks like trying to set up a local mount repo with a URI of file:///run/install/sources/mount-0000-cdrom . That doesn't look right at all. I suspect the payload-related changes Jiri Konecny made between 33.25-1 and 33.25.2-1 are the problem here - probably one of: a57be7d30897ecf301de673e41d1af975b4f593b - Fix issue that treeinfo repositories were never disabled 1c03016c3906a458a74262eaaffbc732d70d5281 - Move parts together in the DNF repo 0788d3fb8a696fa83a818148f08eb439123cc389 - Get rid of add_disable_repo Jiri, can you please take a look?
Here's the equivalent section from a test with anaconda-33.25-1.fc33: 15:21:13,175 INF packaging: Configuring the base repo 15:21:13,205 INF packaging: Enabled 'updates' 15:21:13,205 INF packaging: Disabled 'updates-testing' 15:21:13,208 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='anaconda-0', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='MIRRORLIST', url='https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64') 15:21:13,208 DBG packaging: getting release version from tree at (rawhide) 15:21:13,208 DBG packaging: using treeinfo release version of rawhide 15:21:13,208 DBG packaging: No base repository found in treeinfo file. Using installation tree root. 15:21:13,208 DBG packaging: releasever from is rawhide 15:21:13,209 INF packaging: added repo: 'anaconda' - https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64 15:21:13,209 DBG dnf: repo: downloading from remote: anaconda 15:21:17,057 INF packaging: enabled repo: 'anaconda' - https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-33&arch=x86_64 and got repomd
Ah - looking at those logs I actually think https://github.com/rhinstaller/anaconda/commit/6f27de8a38cc7900bb35a4fac4ec258f50207468 is the offending commit here. Things go wrong in 33.25.2 when we get to "Adding new treeinfo repository: Server enabled: True", which is in `_load_treeinfo_repositories` (previously `_add_treeinfo_repositories`), and that commit changes anaconda from doing that *only on the first payload reset* to doing it on *every* payload reset. `_load_treeinfo_repositories` is doing something wrong in this case. I do note this check: if base_repo_url is not None: which is questionable here, because I believe we are passing in `base_repo_url` as an empty string on this route.
Hmm, no, that check isn't the issue - I patched update_base_repo to pass `base_repo_url or None` but it doesn't help. I think what's going on here is that we're trying to reload the repo from the DVD, but it has been umounted in the meantime. There's some key lines in syslog that we don't see in packaging.log. Here's the whole flow around the *first* base repo setup: syslog: 12:50:47,514 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... mount -t iso9660 -o ro /dev/sr0 /run/install/sources/mount-0000-cdrom 12:50:47,532 WARNING org.fedoraproject.Anaconda.Modules.Payloads:DEBUG:program:Return code: 0 12:50:47,532 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:anaconda.modules.payloads.source.cdrom.initialization:using CD-ROM device sr0 mounted at /run/install/sources/mount-0000-cdrom packaging.log: 12:50:48,883 DBG packaging: Using the repo configuration: RepoConfigurationData(cost=1000, excluded_packages=[], included_packages=[], name='', proxy='', ssl_configuration=SSLConfigurationData(ca_cert_path='', client_cert_path='', client_key_path=''), ssl_verification_enabled=True, type='BASEURL', url='file:///run/install/sources/mount-0000-cdrom') 12:50:48,884 DBG packaging: retrieving treeinfo from file:///run/install/sources/mount-0000-cdrom (proxy: None ; ssl_verify: True) 12:50:48,885 INF packaging: Trying to download '.treeinfo' 12:50:48,887 DBG packaging: Retrieved '.treeinfo' from file:///run/install/sources/mount-0000-cdrom 12:50:48,891 DBG packaging: getting release version from tree at file:///run/install/sources/mount-0000-cdrom (33) 12:50:48,891 DBG packaging: using treeinfo release version of 33 12:50:48,892 DBG packaging: No base repository found in treeinfo file. Using installation tree root. 12:50:48,892 DBG packaging: releasever from file:///run/install/sources/mount-0000-cdrom is 33 12:50:48,893 INF packaging: added repo: 'anaconda' - file:///run/install/sources/mount-0000-cdrom 12:50:48,928 DBG dnf: repo: downloading from remote: anaconda 12:50:49,105 INF packaging: enabled repo: 'anaconda' - ['file:///run/install/sources/mount-0000-cdrom'] and got repomd 12:50:49,106 DBG dnf: repo: using cache for: anaconda 12:50:49,106 DBG packaging: repo anaconda: _sync_metadata success from ['file:///run/install/sources/mount-0000-cdrom'] syslog again: 12:51:42,537 WARNING org.fedoraproject.Anaconda.Modules.Payloads:INFO:program:Running... umount /run/install/sources/mount-0000-cdrom now back to packaging.log: 08:51:44,613 DBG packaging: Adding new treeinfo repository: Server enabled: True 08:51:44,614 INF packaging: added repo: 'Server' - file:///run/install/sources/mount-0000-cdrom 08:51:44,615 DBG dnf: repo: downloading from remote: Server 08:51:44,616 DBG dnf: error: Curl error (37): Couldn't read a file:// file for file:///run/install/sources/mount-0000-cdrom/repodata/repomd.xml [Couldn't open file /run/install/sources/mount-0000-cdrom/repodata/repomd.xml] (file:///run/install/sources/mount-0000-cdrom/repodata/repomd.xml). so we mount the repo, set it up successfully, *then unmount it*, then `_load_treeinfo_repositories` runs again and tries to load it again, but without remounting it first.
oh, the timezone gets set in the middle there or something, which is why we suddenly jump "back" four hours. But that is the order stuff actually happens in, I'm fairly sure.
ah, and interestingly, `DNFPayload.update_base_repo` has a `checkmount` arg which looks like it makes sure something like the DVD repo is mounted if necessary, and pyanaconda/ui/gui/spokes/installation_source.py `SourceSpoke.apply` does this: if source_changed or repo_changed or self._error: payloadMgr.restart_thread(self.payload, checkmount=False) ...and `payloadMgr.restart_thread` winds up doing `payload.update_base_repo` with that `checkmount` value. So I bet that's where we're at here.
yeah, I added a log line and confirmed that during the *first* call to `update_base_repo` (where it works) checkmount is True, but during the *second* call (where it fails) it's False.
PR: https://github.com/rhinstaller/anaconda/pull/2826
This is an old issue but it wasn't a problem before our recent change. The old logic has loaded all additional repositories from the treeinfo just once from the first repository. We have changed the logic to load it every time the repository has changed, however, I forget to change also the metadata loading so it was not cleared and old repository was used as the metadata source. Clear the metadata properly to fix this issue.
FEDORA-2020-daac40f3d3 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3
FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-daac40f3d3` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-daac40f3d3 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
We should be able to check the fix for this in the next Rawhide compose. Testing the F33 update itself is a bit tricky as it requires building a Server DVD...
So Rawhide composes are failing, but they're getting far enough to produce a Server DVD. Here's the latest: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20200909.n.0/compose/Server/x86_64/iso/Fedora-Server-dvd-x86_64-Rawhide-20200909.n.0.iso I'm downloading that to verify the fix now.
Unfortunately that doesn't boot, so can't test. Will try again with the next one.
Rawhide latest compose succeeded, and test passed: https://openqa.fedoraproject.org/tests/661773 so we can confirm the fix for this looks good.
FEDORA-2020-daac40f3d3 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2020-384ff75a01 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-384ff75a01
FEDORA-2020-384ff75a01 has been pushed to the Fedora 33 testing repository. In short time you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-384ff75a01` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-384ff75a01 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2020-384ff75a01 has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.