Bug 1607077

Summary: Today after upgrade system stop working GUI
Product: [Fedora] Fedora Reporter: Mikhail <mikhail.v.gavrilov>
Component: mutterAssignee: Florian Müllner <fmuellner>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: awilliam, fmuellner, mikhail.v.gavrilov, otaylor, robatino, walters
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-23 16:57:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1517011, 1517012    
Attachments:
Description Flags
system log
none
dnf log none

Description Mikhail 2018-07-21 14:24:00 UTC
Created attachment 1469674 [details]
system log

Description of problem:
Today after upgraate system stop working GUI

I suspect this is is occurred due this error:
Jul 21 18:04:59 localhost.localdomain org.gnome.Shell.desktop[1243]: /usr/bin/gnome-shell: symbol lookup error: /lib64/libmutter-2.so.0: undefined symbol: pw_stream_peek_buffer


Most annoying thing that "mutter" package couldn't be downgraded.

Comment 1 Fedora Blocker Bugs Application 2018-07-22 08:53:40 UTC
Proposed as a Blocker and Freeze Exception for 29-beta by Fedora user mikhail using the blocker tracking app because:

 Fully unworkable Workstation build

Comment 2 Mikhail 2018-07-22 10:53:50 UTC
Bug was introduced in mutter.x86_64 3.29.2-3.fc29 and fixed in mutter.x86_64 3.29.4-1.fc29

Comment 3 Adam Williamson 2018-07-23 16:57:53 UTC
The 20180723.n.0 compose included the new mutter build and the Workstation live image openQA tests passed (indicating this is fixed), so closing.

Comment 4 Mikhail 2018-07-25 07:56:47 UTC
Adam, I am more worried about impossibility downgrade package. I am stay on Rawhide and if some package will broke system again I can't quickly restore system. It look like I am sitting on a bomb.

Comment 5 Adam Williamson 2018-07-25 17:24:41 UTC
You didn't provide any useful information about that, so it's sort of hard to debug. What do you mean by 'impossibility' exactly?

Comment 6 Mikhail 2018-07-25 19:49:16 UTC
(In reply to Adam Williamson from comment #5)
> You didn't provide any useful information about that, so it's sort of hard
> to debug. What do you mean by 'impossibility' exactly?

# rpm -q mutter
mutter-3.29.4-1.fc29.x86_64
# dnf downgrade mutter
Last metadata expiration check: 0:58:34 ago on Wed 25 Jul 2018 11:43:17 PM +05.
Package mutter of lowest version already installed, cannot downgrade it.
Error: No packages marked for downgrade.

Same output I have when try downgrade mutter.x86_64 3.29.2-3.fc29.

Also
# ​dnf history undo 212
not helps.

2018-07-21T13:13:02Z INFO --- logging initialized ---
2018-07-21T13:13:02Z DDEBUG timer: config: 3 ms
2018-07-21T13:13:02Z DEBUG Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repograph, repomanage, reposync
2018-07-21T13:13:02Z DEBUG DNF version: 2.7.5
2018-07-21T13:13:02Z DDEBUG Command: dnf history undo 212 
2018-07-21T13:13:02Z DDEBUG Installroot: /
2018-07-21T13:13:02Z DDEBUG Releasever: 29
2018-07-21T13:13:02Z DEBUG cachedir: /var/cache/dnf
2018-07-21T13:13:02Z DDEBUG Base command: history
2018-07-21T13:13:02Z DDEBUG Extra commands: ['history', 'undo', '212']
2018-07-21T13:13:02Z DEBUG repo: using cache for: rawhide
2018-07-21T13:13:02Z DEBUG not found deltainfo for: Fedora - Rawhide - Developmental packages for the next Fedora release
2018-07-21T13:13:02Z DEBUG not found updateinfo for: Fedora - Rawhide - Developmental packages for the next Fedora release
2018-07-21T13:13:03Z DEBUG rawhide: using metadata from Thu 19 Jul 2018 02:31:05 PM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: rawhide-debuginfo
2018-07-21T13:13:03Z DEBUG not found deltainfo for: Fedora - Rawhide - Debug
2018-07-21T13:13:03Z DEBUG not found updateinfo for: Fedora - Rawhide - Debug
2018-07-21T13:13:03Z DEBUG rawhide-debuginfo: using metadata from Thu 19 Jul 2018 02:30:10 PM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: google-chrome-unstable
2018-07-21T13:13:03Z DEBUG not found deltainfo for: google-chrome-unstable
2018-07-21T13:13:03Z DEBUG not found updateinfo for: google-chrome-unstable
2018-07-21T13:13:03Z DEBUG google-chrome-unstable: using metadata from Wed 18 Jul 2018 11:47:43 PM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: local-repo
2018-07-21T13:13:03Z DEBUG not found deltainfo for: local repo
2018-07-21T13:13:03Z DEBUG not found updateinfo for: local repo
2018-07-21T13:13:03Z DEBUG local-repo: using metadata from Fri 13 Jul 2018 08:37:55 AM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: opera
2018-07-21T13:13:03Z DEBUG not found deltainfo for: Opera packages
2018-07-21T13:13:03Z DEBUG not found updateinfo for: Opera packages
2018-07-21T13:13:03Z DEBUG opera: using metadata from Fri 20 Jul 2018 11:48:54 AM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: rpmfusion-free-rawhide
2018-07-21T13:13:03Z DEBUG not found deltainfo for: RPM Fusion for Fedora Rawhide - Free
2018-07-21T13:13:03Z DEBUG not found updateinfo for: RPM Fusion for Fedora Rawhide - Free
2018-07-21T13:13:03Z DEBUG rpmfusion-free-rawhide: using metadata from Sat 21 Jul 2018 03:51:56 PM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: rpmfusion-free-rawhide-debuginfo
2018-07-21T13:13:03Z DEBUG not found deltainfo for: RPM Fusion for Fedora Rawhide - Free - Debug
2018-07-21T13:13:03Z DEBUG not found updateinfo for: RPM Fusion for Fedora Rawhide - Free - Debug
2018-07-21T13:13:03Z DEBUG rpmfusion-free-rawhide-debuginfo: using metadata from Sat 21 Jul 2018 03:48:48 PM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: rpmfusion-nonfree-rawhide
2018-07-21T13:13:03Z DEBUG not found deltainfo for: RPM Fusion for Fedora Rawhide - Nonfree
2018-07-21T13:13:03Z DEBUG not found updateinfo for: RPM Fusion for Fedora Rawhide - Nonfree
2018-07-21T13:13:03Z DEBUG rpmfusion-nonfree-rawhide: using metadata from Thu 19 Jul 2018 02:36:45 PM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: rpmfusion-nonfree-rawhide-debuginfo
2018-07-21T13:13:03Z DEBUG not found deltainfo for: RPM Fusion for Fedora Rawhide - Nonfree - Debug
2018-07-21T13:13:03Z DEBUG not found updateinfo for: RPM Fusion for Fedora Rawhide - Nonfree - Debug
2018-07-21T13:13:03Z DEBUG rpmfusion-nonfree-rawhide-debuginfo: using metadata from Sat 21 Jul 2018 04:13:58 PM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: sublime-text
2018-07-21T13:13:03Z DEBUG not found deltainfo for: Sublime Text - x86_64 - Stable
2018-07-21T13:13:03Z DEBUG not found updateinfo for: Sublime Text - x86_64 - Stable
2018-07-21T13:13:03Z DEBUG sublime-text: using metadata from Mon 14 May 2018 05:05:07 AM +05.
2018-07-21T13:13:03Z DEBUG repo: using cache for: code
2018-07-21T13:13:03Z DEBUG not found deltainfo for: Visual Studio Code
2018-07-21T13:13:03Z DEBUG not found updateinfo for: Visual Studio Code
2018-07-21T13:13:03Z DEBUG code: using metadata from Fri 13 Jul 2018 02:22:35 AM +05.
2018-07-21T13:13:03Z INFO Last metadata expiration check: 0:01:21 ago on Sat 21 Jul 2018 06:11:41 PM +05.
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rawhide-2d95c80a1fa0a67d
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rawhide-debuginfo-b0297fcd3e236bd4
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/google-chrome-unstable-eb0d6f10ccbdafba
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/local-repo-f13aca0339b74221
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/opera-4e999c6fc4466cf1
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rpmfusion-free-rawhide-11a4e834e18e3a11
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rpmfusion-free-rawhide-debuginfo-61cabae612735d71
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rpmfusion-nonfree-rawhide-978ef37952204a2d
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/rpmfusion-nonfree-rawhide-debuginfo-86c496277c656274
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/sublime-text-97de60ea8b05c154
2018-07-21T13:13:03Z DEBUG Missing file *modules.yaml in metadata cache dir: /var/cache/dnf/code-1a27fa61fdd77ffb
2018-07-21T13:13:03Z DDEBUG timer: sack setup: 901 ms
2018-07-21T13:13:03Z INFO Undoing transaction 212, from Sat 21 Jul 2018 05:06:48 PM +05
2018-07-21T13:13:27Z INFO No package [1mPackageKit-debuginfo-0:1.1.10-2.fc29.x86_64[m available.
2018-07-21T13:13:27Z DDEBUG Cleaning up.
2018-07-21T13:13:27Z SUBDEBUG 
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 99, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 115, in cli_run
    cli.run()
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 1016, in run
    return self.command.run()
  File "/usr/lib/python3.6/site-packages/dnf/cli/commands/__init__.py", line 990, in run
    raise dnf.exceptions.Error(strs[0])
dnf.exceptions.Error: An operation cannot be undone
2018-07-21T13:13:27Z CRITICAL Error: An operation cannot be undone

Comment 7 Mikhail 2018-07-25 19:50:10 UTC
Created attachment 1470548 [details]
dnf log

Comment 8 Adam Williamson 2018-07-25 21:02:14 UTC
well, that's sort of an inevitable limitation: dnf can only downgrade if it can *find a package to downgrade to*. There isn't a default repo where you can find previous builds of Rawhide packages, that's just...not a thing.

If you want to downgrade to a previous Rawhide build you have to just go grab the relevant rpms manually from Koji, or set up a repo pointing to one of the earlier composes at https://kojipkgs.fedoraproject.org/compose/rawhide/ .

(similarly with dnf history undo; it's not magic, non-Atomic systems don't store every state the system's ever been in somewhere, somehow. To undo an upgrade from foo-1.0-1 to foo-2.0-1, dnf needs to have foo-1.0-1.rpm actually accessible to it somehow).

Comment 9 Mikhail 2018-07-25 21:33:45 UTC
(In reply to Adam Williamson from comment #8)
> well, that's sort of an inevitable limitation: dnf can only downgrade if it
> can *find a package to downgrade to*. There isn't a default repo where you
> can find previous builds of Rawhide packages, that's just...not a thing.

Why fedora maintainers remove the previous version from the repository in Rawhide? Is there really not enough disk space? In any case, I think keeping the three previous versions of the package would not be costly.

> If you want to downgrade to a previous Rawhide build you have to just go
> grab the relevant rpms manually from Koji, or set up a repo pointing to one
> of the earlier composes at
> https://kojipkgs.fedoraproject.org/compose/rawhide/ .

It will be problematic to run a web browser (for search needed package in koji) if the GUI breaks again.

> (similarly with dnf history undo; it's not magic, non-Atomic systems don't
> store every state the system's ever been in somewhere, somehow. To undo an
> upgrade from foo-1.0-1 to foo-2.0-1, dnf needs to have foo-1.0-1.rpm
> actually accessible to it somehow).

I understand it. But I would be save system worked if previous version of the package would available.

Comment 10 Adam Williamson 2018-07-25 21:39:10 UTC
It's not a case of 'removing' it, exactly. That's just...not how things work. The repositories are *re-generated* with each compose, and only the current version of each package included. Successful composes are synced into the location the repository definitions point to.

If you're running Rawhide...sometimes stuff breaks. We do try and avoid this, but still, it *is* the case, and as the Rawhide instructions in the wiki say, you should be prepared to deal with this. I usually keep a system running a stable release handy, for instance.