Bug 2008986 - prometheus stopped working after update from 2.24.1-6.fc34.x86_64 to 2.30.0-1.fc34.x86_64
Summary: prometheus stopped working after update from 2.24.1-6.fc34.x86_64 to 2.30.0-1...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: golang-github-prometheus
Version: 34
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Robert-André Mauchin 🐧
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-29 15:57 UTC by Ivan Mironov
Modified: 2022-01-26 00:59 UTC (History)
2 users (show)

Fixed In Version: golang-github-prometheus-2.32.1-1.fc34 golang-github-prometheus-2.32.1-1.fc35 golang-github-prometheus-2.32.1-1.el8
Doc Type: ---
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-25 01:03:16 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Ivan Mironov 2021-09-29 15:57:52 UTC
Description of problem:
# grep prometheus /var/log/dnf.rpm.log
...
2021-09-29T20:13:43+0500 SUBDEBUG Upgrade: golang-github-prometheus-2.30.0-1.fc34.x86_64
2021-09-29T20:14:01+0500 SUBDEBUG Upgraded: golang-github-prometheus-2.24.1-6.fc34.x86_64

# journalctl -u prometheus.service
...
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[1771]: level=info ts=2021-09-29T15:14:20.803Z caller=main.go:815 msg="Notifier manager stopped"
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[1771]: level=info ts=2021-09-29T15:14:20.804Z caller=main.go:827 msg="See you next time!"
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: prometheus.service: Deactivated successfully.
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: Stopped Prometheus service monitoring system and time series database.
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: prometheus.service: Consumed 1h 17min 55.399s CPU time.
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: Started Prometheus service monitoring system and time series database.
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[188094]: Error parsing commandline arguments: unexpected
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[188094]: prometheus: error: unexpected
Sep 29 20:14:21 router225.ivan-mironov.net systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
Sep 29 20:14:21 router225.ivan-mironov.net systemd[1]: prometheus.service: Failed with result 'exit-code'.
Sep 29 20:14:21 router225.ivan-mironov.net systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 1.
...


Version-Release number of selected component (if applicable):
golang-github-prometheus-2.30.0-1.fc34.x86_64


How reproducible:
Always.


Steps to Reproduce:
1. Run `dnf upgrade` on a working f34 system running prometheus.
2. Observe failed restart of prometheus.service.


Actual results:
prometheus.service fails to start after update.


Expected results:
prometheus.service should start like before.


Additional info:
It looks like this happens because systemd expands empty ${ADDITIONAL_OPTIONS} as an empty string argument instead of not adding any command line argument at all.

This bug can be workarounded by setting ADDITIONAL_OPTIONS in /etc/sysconfig/prometheus to some redundand value, like `ADDITIONAL_OPTIONS=--log.format=logfmt`.

Comment 1 Fedora Update System 2022-01-16 21:57:42 UTC
FEDORA-2022-077084b1cb has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-077084b1cb

Comment 2 Fedora Update System 2022-01-16 21:57:53 UTC
FEDORA-2022-3f5f3fa356 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-3f5f3fa356

Comment 3 Fedora Update System 2022-01-17 01:33:05 UTC
FEDORA-EPEL-2022-d652ca9a79 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-d652ca9a79

Comment 4 Fedora Update System 2022-01-17 02:00:58 UTC
FEDORA-2022-3f5f3fa356 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-3f5f3fa356`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3f5f3fa356

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 5 Fedora Update System 2022-01-17 02:16:14 UTC
FEDORA-2022-077084b1cb has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-077084b1cb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-077084b1cb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 6 Fedora Update System 2022-01-18 01:54:20 UTC
FEDORA-EPEL-2022-d652ca9a79 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-d652ca9a79

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2022-01-25 01:03:16 UTC
FEDORA-2022-077084b1cb has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 8 Fedora Update System 2022-01-25 01:10:55 UTC
FEDORA-2022-3f5f3fa356 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 9 Fedora Update System 2022-01-26 00:59:46 UTC
FEDORA-EPEL-2022-d652ca9a79 has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.