Bug 2227109
| Summary: | `podman login` against the container registry returns 500 intermittently | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | matt jia <mjia> |
| Component: | Container Management - Content | Assignee: | Ian Ballou <iballou> |
| Status: | CLOSED MIGRATED | QA Contact: | Vladimír Sedmík <vsedmik> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 6.13.0 | CC: | ahumbe, alsouza, iballou, kmarutha, rlavi, saydas, vsedmik |
| Target Milestone: | stream | Keywords: | MigratedToJIRA, Triaged |
| Target Release: | Unused | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | smart_proxy_container_gateway-3.0.0.gem | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-06-06 16:25:30 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
matt jia
2023-07-28 01:28:56 UTC
The strange part is why update_user_repositories is called when doing `podman login`? It clearly seems like a bug to me. When a user authenticates to the smart proxy container gateway, we need to update what repositories they have access to in case anything changed on the Foreman. That's why update_user_repositories is run. The bug here is that sometimes SQLite locks up when multiple requests happen at once. This popped up before: https://bugzilla.redhat.com/show_bug.cgi?id=2105349 (In reply to Ian Ballou from comment #2) > When a user authenticates to the smart proxy container gateway, we need to > update what repositories they have access to in case anything changed on the > Foreman. That's why update_user_repositories is run. > The bug here is that sometimes SQLite locks up when multiple requests happen > at once. This popped up before: > https://bugzilla.redhat.com/show_bug.cgi?id=2105349 Right. Do we have any workaround here as this is significantly impacting the customer's Openstack deployment. (In reply to matt jia from comment #3) > (In reply to Ian Ballou from comment #2) > > When a user authenticates to the smart proxy container gateway, we need to > > update what repositories they have access to in case anything changed on the > > Foreman. That's why update_user_repositories is run. > > The bug here is that sometimes SQLite locks up when multiple requests happen > > at once. This popped up before: > > https://bugzilla.redhat.com/show_bug.cgi?id=2105349 > > Right. Do we have any workaround here as this is significantly impacting the > customer's Openstack deployment. Tagging Ian to know if we have any workaround available. A likely workaround would be to trigger the container registry user data sync between the Satellite and the Capsule manually: ``` # foreman-rake console ::SmartProxy.find(<capsule id>).sync_container_gateway ``` This avoids the overhead of a capsule sync. Run the workaround above when the capsule is not receiving registry requests from clients. Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36771 has been resolved. Smart proxy container gateway code is fixed in smart_proxy_container_gateway-1.0.9.gem Packaging PR upstream: https://github.com/theforeman/foreman-packaging/pull/9836 An installer change goes with this as well so the added SQLite busy timeout is more easily configured: https://github.com/theforeman/puppet-foreman_proxy/pull/813 The bug is likely to be fixed directly by smart_proxy_container_gateway-1.0.9 because I also increased the default sqlite timeout from 5 seconds to 30 seconds. If the busy error is still occurring, the busy timeout can be increased via the new sqlite_timeout setting introduced here. *** Bug 2105349 has been marked as a duplicate of this bug. *** Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set. For some reason during testing on Satellite, the busy timeout didn't help with concurrent `podman login` requests. My first action will be to see why that was occurring. However, I think the solution could be further improved. SQLite has a different mode that has better concurrency: https://www.sqlite.org/wal.html We could try using this instead. On top of that, the container gateway should probably be handling the busy exception and retrying. With WAL mode however the busy exception should become much rarer. Upstream bug assigned to iballou Upstream bug assigned to ekohlvan Moving to POST since all upstream PRs are now merged. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "SAT-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |