Maybe to clarify this more, the "offline" part seems relevant without it, I can create a container.

As user:

admin@fedora-coreos-127-0-0-2-2202:~$ podman run -ti localhost/test-alpine sh
/ #

Then exit container and as root

[root@fedora-coreos-127-0-0-2-2202 ~]# podman run -ti localhost/test-alpine sh
/ #

Then try to start a container as user:

admin@fedora-coreos-127-0-0-2-2202:~$ podman run -ti localhost/test-alpine sh
Error: pasta failed with exit code 1:
External interface not usable

The offline VM part is relevant, without it, it does not reproduce.

When it fails here is the `ip` output:

admin@fedora-coreos-127-0-0-2-2202:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: ens14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    altname enp0s14
    inet 172.27.0.15/24 brd 172.27.0.255 scope global dynamic noprefixroute ens14
       valid_lft 86354sec preferred_lft 86354sec
    inet6 fec0::106f:f38b:236a:327b/64 scope site dynamic noprefixroute
       valid_lft 86356sec preferred_lft 14356sec
    inet6 fe80::157d:c453:cc72:71f6/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: ens15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:01:00:00:02 brd ff:ff:ff:ff:ff:ff
    altname enp0s15
    inet6 fe80::5054:1ff:fe00:2/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
4: podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:cd:e9:48:37:60 brd ff:ff:ff:ff:ff:ff
    inet 10.88.0.1/16 brd 10.88.255.255 scope global podman0
       valid_lft forever preferred_lft forever
    inet6 fe80::38cd:e9ff:fe48:3760/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
5: veth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master podman0 state UP group default qlen 1000
    link/ether 02:8b:c1:fe:02:f8 brd ff:ff:ff:ff:ff:ff link-netns netns-7000f382-87c0-499b-1eda-8155f6db65be
    inet6 fe80::d8d6:bff:feb3:9f4c/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

I re-tested this on current Fedora 40 and CoreOS with podman-5.0.3-1.fc40.x86_64 and passt-0^20240510.g7288448-1.fc40.x86_64, and it still fails. In a freshly booted VM, there is only a single route:

$ ip route
172.27.0.0/24 dev ens14 proto kernel scope link src 172.27.0.15 metric 100 

and with that it still works.

> This seems to only occur when a system container was started before the user container tries to start.

Right, because that creates a "podman0" veth with an extra route:

  10.88.0.0/16 dev podman0 proto kernel scope link src 10.88.0.1 

which seems to confuse passt.

A hack which keeps the VM offline (i.e. without an actual default route) seems to be:

   nmcli con add type dummy con-name fake ifname fake0 ip4 1.2.3.4/24 gw4 1.2.3.1

This also started to fail on CentOS 9 stream (podman-5.0.2-1.el9.x86_64 and passt-0^20231204.gb86afe3-1.el9.x86_64). I suppose there's not much point in filing a jira for this until this gets fixed in Fedora?

(In reply to Martin Pitt from comment #2)
> This also started to fail on CentOS 9 stream (podman-5.0.2-1.el9.x86_64 and
> passt-0^20231204.gb86afe3-1.el9.x86_64). I suppose there's not much point in
> filing a jira for this until this gets fixed in Fedora?

To have it fixed in Fedora, I guess it should be fixed upstream first :) so passt's Bugzilla (https://bugs.passt.top) would actually be more appropriate. But we can also track this here as far as I'm concerned.

I'm not sure what's expected from pasta here, though. My understanding is that there are two upstream interfaces, podman0 and ens14, with one IPv4 address each, and without a default route.

So, there are two interfaces, none of which has a default route, and pasta doesn't know which interface it should use as template interface to copy addresses and routes from. This is reflected in pasta(1) as updated by https://passt.top/passt/commit/?id=f919dc7a4b1ced7e80d790a654900415e1d6250e:

       -i, --interface name
              Use host interface name to derive addresses and routes.  Default
              is to use the interfaces specified by --outbound-if4 and  --out‐
              bound-if6, for IPv4 and IPv6 addresses and routes, respectively.

              If no interfaces are given, the interface with the first default
              routes for each IP version is selected. If no default routes are
              available  and  there is just one interface with any route, that
              interface will be chosen instead.

...what do you think pasta should do, instead? If connectivity is not expected to work, would it rather make sense to pass '--net none' to Podman?

> there are two interfaces, none of which has a default route, and pasta doesn't know which interface it should use

One of them was created by podman itself ("podman0"), so not that one :-)

> If connectivity is not expected to work, would it rather make sense to pass '--net none' to Podman?

No, that's too strong. We *do* want to talk from the container to the host, just not from the container to "anywhere else on the internet".

> ...what do you think pasta should do, instead?

I don't know really -- this never affected slirp (it just picked 10.0.2.x and 10.0.2.2 was the host gateway), which worked quite nicely -- especially as the container and gateway had a predictable IP inside. Falling back to such addresses may work if there's no more specific one? TBH I still don't understand what the addresses/routes *inside* the container have to do with the host's default route -- they should really not know about each other.

Anyway -- if you say that pasta won't support hosts without a default route, feel free to wontfix this. We can adjust our tests: as I said, this doesn't block our testing (we have a workaround with a fake device/route). But it's an user-visible behaviour change/regression, so I don't want to close it myself as that's not for me to judge.

(In reply to Martin Pitt from comment #4)
> > there are two interfaces, none of which has a default route, and pasta doesn't know which interface it should use
> 
> One of them was created by podman itself ("podman0"), so not that one :-)

That was created by another instance of Podman, though, and they should be independent (especially given that one was started as root), so we can't really expect Podman to tell pasta to use another interface (strictly speaking, it could).

> > If connectivity is not expected to work, would it rather make sense to pass '--net none' to Podman?
> 
> No, that's too strong. We *do* want to talk from the container to the host,
> just not from the container to "anywhere else on the internet".

I see, so this is similar to the use cases described at https://github.com/containers/podman/discussions/22737#discussioncomment-9478727 and https://github.com/containers/podman/discussions/22570.

> > ...what do you think pasta should do, instead?
> 
> I don't know really -- this never affected slirp (it just picked 10.0.2.x
> and 10.0.2.2 was the host gateway), which worked quite nicely -- especially
> as the container and gateway had a predictable IP inside. Falling back to
> such addresses may work if there's no more specific one?

We don't really want to hardcode addresses, even just as default or fallback, because those might be in use by other hosts, so you might have a process that expects to connect to something in 10.0.2.0/24, and once it's moved to a container, it connects to localhost instead. Same for IPv6.

Note that you can do that using -a / --address and -g / --gateway. In this case, actually, selecting ens14 as template interface with e.g. --net=pasta,-i,ens14 would also do the trick.

> TBH I still don't
> understand what the addresses/routes *inside* the container have to do with
> the host's default route -- they should really not know about each other.

The container needs to have an address to communicate with any other host or namespace, and using an address that's already assigned to the host guarantees it's valid and usable. On top of that, this configuration avoids (implicit) NAT, which streamlines compatibility with a number of applications (e.g. service meshes) and protocols (anything that refers to Layer-3 addresses in upper layers), and also avoids all the downsides nicely described in Sections 6 and 9 of RFC 2993. See also:
  https://passt.top/#motivation
  https://github.com/containers/podman/issues/22771#issuecomment-2148105764

> Anyway -- if you say that pasta won't support hosts without a default route,
> feel free to wontfix this.

Again, pasta(1) *does* now support hosts without a default route since https://passt.top/passt/commit/?id=f919dc7a4b1ced7e80d790a654900415e1d6250e, which we introduced because of the use case you reported. The only problem is that if there are multiple interfaces, none of which with a default route, we really don't know which one we should pick (and in this specific case, Podman can't tell us).

Perhaps it would actually be fine if we just picked an interface, even in this case, for example the first one that was created (the kernel returns them in that order). I'll try that and report back (probably in a little while).

> We can adjust our tests: as I said, this doesn't
> block our testing (we have a workaround with a fake device/route). But it's
> an user-visible behaviour change/regression, so I don't want to close it
> myself as that's not for me to judge.

It's documented here: https://blog.podman.io/2024/03/podman-5-0-breaking-changes-in-detail/. On the other hand, if we can make it less breaking without loss of generality or security, then I guess it's worth trying.

(In reply to Stefano Brivio from comment #5)
> Perhaps it would actually be fine if we just picked an interface, even in
> this case, for example the first one that was created (the kernel returns
> them in that order). I'll try that and report back (probably in a little
> while).

Patch at:

    https://archives.passt.top/passt-dev/20240618171803.1924322-1-sbrivio@redhat.com/

Fixed in version 2024_06_24.1ee2eca (https://archives.passt.top/passt-user/20240624210651.61ce77af@elisabeth/), and matching (upcoming) Fedora 40 update (https://bodhi.fedoraproject.org/updates/FEDORA-2024-4632bfa865).