Bug 2008986

Summary: prometheus stopped working after update from 2.24.1-6.fc34.x86_64 to 2.30.0-1.fc34.x86_64
Product: [Fedora] Fedora Reporter: Ivan Mironov <mironov.ivan>
Component: golang-github-prometheusAssignee: Robert-André Mauchin 🐧 <eclipseo>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 34CC: eclipseo, go-sig
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: golang-github-prometheus-2.32.1-1.fc34 golang-github-prometheus-2.32.1-1.fc35 golang-github-prometheus-2.32.1-1.el8 Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-25 01:03:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ivan Mironov 2021-09-29 15:57:52 UTC
Description of problem:
# grep prometheus /var/log/dnf.rpm.log
...
2021-09-29T20:13:43+0500 SUBDEBUG Upgrade: golang-github-prometheus-2.30.0-1.fc34.x86_64
2021-09-29T20:14:01+0500 SUBDEBUG Upgraded: golang-github-prometheus-2.24.1-6.fc34.x86_64

# journalctl -u prometheus.service
...
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[1771]: level=info ts=2021-09-29T15:14:20.803Z caller=main.go:815 msg="Notifier manager stopped"
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[1771]: level=info ts=2021-09-29T15:14:20.804Z caller=main.go:827 msg="See you next time!"
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: prometheus.service: Deactivated successfully.
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: Stopped Prometheus service monitoring system and time series database.
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: prometheus.service: Consumed 1h 17min 55.399s CPU time.
Sep 29 20:14:20 router225.ivan-mironov.net systemd[1]: Started Prometheus service monitoring system and time series database.
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[188094]: Error parsing commandline arguments: unexpected
Sep 29 20:14:20 router225.ivan-mironov.net prometheus[188094]: prometheus: error: unexpected
Sep 29 20:14:21 router225.ivan-mironov.net systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
Sep 29 20:14:21 router225.ivan-mironov.net systemd[1]: prometheus.service: Failed with result 'exit-code'.
Sep 29 20:14:21 router225.ivan-mironov.net systemd[1]: prometheus.service: Scheduled restart job, restart counter is at 1.
...


Version-Release number of selected component (if applicable):
golang-github-prometheus-2.30.0-1.fc34.x86_64


How reproducible:
Always.


Steps to Reproduce:
1. Run `dnf upgrade` on a working f34 system running prometheus.
2. Observe failed restart of prometheus.service.


Actual results:
prometheus.service fails to start after update.


Expected results:
prometheus.service should start like before.


Additional info:
It looks like this happens because systemd expands empty ${ADDITIONAL_OPTIONS} as an empty string argument instead of not adding any command line argument at all.

This bug can be workarounded by setting ADDITIONAL_OPTIONS in /etc/sysconfig/prometheus to some redundand value, like `ADDITIONAL_OPTIONS=--log.format=logfmt`.

Comment 1 Fedora Update System 2022-01-16 21:57:42 UTC
FEDORA-2022-077084b1cb has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-077084b1cb

Comment 2 Fedora Update System 2022-01-16 21:57:53 UTC
FEDORA-2022-3f5f3fa356 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-3f5f3fa356

Comment 3 Fedora Update System 2022-01-17 01:33:05 UTC
FEDORA-EPEL-2022-d652ca9a79 has been submitted as an update to Fedora EPEL 8. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-d652ca9a79

Comment 4 Fedora Update System 2022-01-17 02:00:58 UTC
FEDORA-2022-3f5f3fa356 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-3f5f3fa356`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-3f5f3fa356

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 5 Fedora Update System 2022-01-17 02:16:14 UTC
FEDORA-2022-077084b1cb has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-077084b1cb`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-077084b1cb

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 6 Fedora Update System 2022-01-18 01:54:20 UTC
FEDORA-EPEL-2022-d652ca9a79 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-d652ca9a79

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 7 Fedora Update System 2022-01-25 01:03:16 UTC
FEDORA-2022-077084b1cb has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 8 Fedora Update System 2022-01-25 01:10:55 UTC
FEDORA-2022-3f5f3fa356 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 9 Fedora Update System 2022-01-26 00:59:46 UTC
FEDORA-EPEL-2022-d652ca9a79 has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.