Bug 1907228 - Silverblue: Podman complains kernel not supporting overlay fs (backing file system is unsupported for this graph driver)
Summary: Silverblue: Podman complains kernel not supporting overlay fs (backing file s...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: podman
Version: 33
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Lokesh Mandvekar
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-13 19:14 UTC by rugk
Modified: 2020-12-17 15:20 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-17 15:20:15 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description rugk 2020-12-13 19:14:58 UTC
Description of problem:
Podman is completly broken on Silverblue. I did not do anything, but just tried to use podman.
It throws me the error shown below or is just stuck and does not continue.

Version-Release number of selected component (if applicable):
podman version 2.1.1

ostree://fedora:fedora/33/x86_64/silverblue
Version: 33.20201207.0 (2020-12-07T01:04:03Z)

How reproducible:
Always.

Steps to Reproduce:
Run podman login or basically any other command.

Actual results:
$ podman login …                   
ERRO[0000] User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve

Expected results:
No error?

Additional info:

I did not change anything. I've set `REGISTRY_AUTH_FILE`, but that should not affect the whole system.
Podman could be used a few months ago. I did not test it in the meantime, but now everything fails.

So I've obviously looked that up and found e.g. this:
https://github.com/containers/podman/issues/7501
https://github.com/containers/podman/issues/7396

Nothing helped. E.g. I could delete (move) ~/.local/share/containers, but of course I 1. don't want to loose all my container and b) it actually sometimes shows a different behaviour:
When I simply run a "podman ps" it shows me an empty terminal. It get's stuck. No output, nothing. It also does not exist though (unless I Ctrl+C it, of course.)

Also tried configuring ~/.config/containers/storage.conf, but I don't know what to set there. Just work as usual in Silverblue. I don't know or care what storage driver you use. :D

Even toolbox fails fundamentally:
toolbox list                  
Error: failed to get the Podman version

Unfortunately 
podman info --debug

is also broken. (Does not output anything/does not exist)
Only podman -v works.

Comment 1 rugk 2020-12-13 19:26:14 UTC
Reported upstream: https://github.com/containers/podman/issues/8705

Funnily even `podman version` does not work, while `podman -v` does.

Comment 2 rugk 2020-12-13 19:33:19 UTC
Also opened discussion in Fedora commmunity forum: https://discussion.fedoraproject.org/t/podman-broken-on-silverblue-podman-complains-about-overwritten-graph-driver-and-wants-libpod-to-be-deleted/25403?u=rugk

BTW, I can see it re-creates the $HOME/.local/share/containers dir when I start it though. Actually, only one file is created though: $HOME/.local/share/containers/storage/libpod/bolt_state.db

Comment 3 rugk 2020-12-13 20:56:06 UTC
Oh after a system upgrade and reboot, that included an update of podman:
```
podman 2:2.1.1-12.fc33 -> 2:2.2.1-1.fc33
podman-plugins 2:2.1.1-12.fc33 -> 2:2.2.1-1.fc33
```

I now get a slightly different error message:
```
$ podman version
ERRO[0000] 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay" 
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
```

Note this still seems to be the default option. 

Also funny, if I rerun this, I only get the error message (no `ERRO[0000]`  is displayed anymore (which is useless anyway and I don't even get what this tries to tell me TBH. Looks like the strange and useless error codes you know from Windows :stuck_out_tongue_winking_eye:):
```
$ podman version
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/rugk/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
```

Comment 4 rugk 2020-12-14 10:54:55 UTC
Changing the settings fixes that.
`~/.config/containers/storage.conf` needs to be like this:
```
[storage]
driver = "overlay"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"
```

However, something seems to be changed in the latest Silverblue version, as this error likely should not happen by default.
So maybe you want to change the default config?

Comment 5 Daniel Walsh 2020-12-15 10:38:30 UTC
That file should just not be there, which is the default.  The problem was having that file in the homedir in the first place and that file not declaring the driver

Comment 6 rugk 2020-12-17 11:01:23 UTC
The bug initially happened **without any file** IIRC. But I see now that if I remove that file, it does work too.
If I just comment **all lines** in the file with # it does show an error though, which is very unexpected.

Comment 7 Daniel Walsh 2020-12-17 15:20:15 UTC
Well the default behaviour is only executed if there is no storage.conf in the users homedir. The thought is, if the user creates 
storage.conf in their homedir, then they should create the storage.conf fully.  If they don't specify a driver, we will default
to "overlay" but we will not define any additional storage options, including mount_program.


Note You need to log in before you can comment on or make changes to this bug.