Bug 1692016

Summary: Default Pagure configs not integrated to gitolite3 and rest of the system
Product: [Fedora] Fedora Reporter: Avi Alkalay <avibrazil>
Component: pagureAssignee: Neal Gompa <ngompa13>
Status: CLOSED WORKSFORME QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 29CC: bruno, ngompa13, pingou, vivekanand1101
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: pagure-5.4-1.fc30 pagure-5.4-1.el7 pagure-5.4-1.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-09 13:13:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Patch/edits for /etc/httpd/conf.d/pagure.conf
none
Patch/edits for /etc/pagure/pagure.cfg
none
Patch/edits for /etc/pagure/alembic.ini none

Description Avi Alkalay 2019-03-23 11:13:22 UTC 
Description of problem:
Pagure won't run at all out of the box after installing with dnf on a plain and single Fedora 29 machine.

Version-Release number of selected component (if applicable):
Pagure 5.3
gitolite3 3.6.11


How reproducible:
Just install Pagure official RPMs and try to run it. Impossible.

All users and groups in systemd service files are wrong. They use "git" but gitolite3 creates a "gitolite3" user that should be used on Pagure's system files.

The repository paths in /etc/httpd/conf.d/pagure.conf and /etc/pagure/pagure.cfg point to "/srv/git" but gitolite3 is in more standard "/var/lib/gitolite3".

Nowhere in the documentation or error messages say other Pagure packages need to be installed and running somewhere for a more coherent run. I had to figure it out by myself and "systemctl enable" all of them, after fixing the wrong user in service files.

The /etc/httpd/conf.d/pagure.conf file is entirely commented and disabled, giving no clue what is optional and what is necessary. A minimum working file is expected, as all packages.

After a lot of fiddling, I got it partially working with MySQL.

I can log in, validate user through e-mail link, create repos and groups.

But a repo created will keep that "The permissions on this repository are being updated" forever.

A repository created with "Mirror from URL" will never clone nothing from that URL, even if all the following services are systemd enabled and started:
pagure_ci.service
pagure_ev.service
pagure_gitolite_worker.service
pagure_loadjson.service
pagure_logcom.service
pagure_milter.service
pagure_mirror.service
pagure_webhook.service
pagure_worker.service

Regular repos created through the UI present a nice page with basic git operations, but they can't be seen or accessed through a remote git command. For debugging purposes, if I make "ssh -l gitolite3 myserver.com", I get:

——————-
PTY allocation request failed on channel 0
hello aviram, this is gitolite3@student running gitolite3 3.6.11-1.fc29 on git 2.20.1

 R W	testing
 R W	gitolite-admin
Connection to myserver.com closed.
——————-

These are the default initial gitolite3 repos. Repos created trough Pagure UI won't show here unless I go to the terminal as gitolite3 user and run "pagure-admin refresh-gitolite —all". Even so, "The permissions on this repository are being updated" message won't disappear.

After getting the Pagure-created repos appear in gitolite3, I can upload my project with git commands and see it in Pagure UI. But this is also not straight forward and I'll assume I lacks basic git knowledge.

It is expected that Fedora's Pagure packages will be more well integrated with Fedora's gitolite3 and the rest of the Fedora system in such a way that it will run out of the box or with minimum documented configuration in "/usr/share/doc/pagure/fedora-README", which currently doesn't exist. All other documentation and examples available on the Internet are for much older versions of Pagure or for different setups, such as Docker. Useless for the latest version of Pagure.

Please fix default configuration files and provide updated documentation.
Comment 1 Avi Alkalay 2019-03-23 11:18:37 UTC 
Here is my step-by-step setup on Fedora 29:

dnf install pagure* mariadb-server redis*

# Configure gitolite:
UU=aviram
cp "`getent passwd $UU | cut -f6 -d:`/.ssh/authorized_keys" ~gitolite3/$UU.pub
chmod a+r ~gitolite3/$UU.pub
su - gitolite3 -c "gitolite setup -pk ~/$UU.pub"
mkdir -p ~gitolite3/remotes
chown -R gitolite3. ~gitolite3/remotes

setfacl -Rdm user:apache:rx ~gitolite3/
setfacl -Rm user:apache:rx ~gitolite3/

mkdir /var/www/releases
chown gitolite3. /var/www/releases


# Put correct user on service files:
sed -i 's/=git$/=gitolite3/g; s/=mirror$/=gitolite3/g' /usr/lib/systemd/system/pagure*



#### Patch or edit by hand /etc/pagure/alembic.ini /etc/pagure/pagure.cfg and /etc/httpd/conf.d/pagure.conf
...


# Setup database:
systemctl restart mysqld
mysqladmin -u root -p drop pagure
mysql -u root -p -e "CREATE DATABASE pagure; GRANT ALL PRIVILEGES ON pagure.* TO 'pagure'@'%' IDENTIFIED BY 'pagurepass'; FLUSH PRIVILEGES;"
PAGURE_CONFIG=/etc/pagure/pagure.cfg python3 /usr/share/pagure/pagure_createdb.py


# Enable services:
systemctl enable /usr/lib/systemd/system/pagure_*.service  /usr/lib/systemd/system/redis*
systemctl restart redis* pagure*


# Start Apache with Pagure:
systemctl enable httpd
systemctl restart httpd
Comment 2 Avi Alkalay 2019-03-23 11:28:49 UTC 
I also get this type of error message on /var/log/messages:


  31927 Mar 23 06:57:13 student celery-3[29454]: 2019-03-23 06:57:13,653 [ERROR] celery.app.trace: Task pagure.lib.tasks.generate_gitolite_acls[bdea0922-0a91-4856-8a8b-6b44f0c5bcab] raised unexpected: TypeError("a bytes-like object is required, not 'str'")
  31928 Mar 23 06:57:13 student celery-3[29454]: Traceback (most recent call last):
  31929 Mar 23 06:57:13 student celery-3[29454]:  File "/usr/lib/python3.7/site-packages/celery/app/trace.py", line 382, in trace_task
  31930 Mar 23 06:57:13 student celery-3[29454]:    R = retval = fun(*args, **kwargs)
  31931 Mar 23 06:57:13 student celery-3[29454]:  File "/usr/lib/python3.7/site-packages/celery/app/trace.py", line 641, in __protected_call__
  31932 Mar 23 06:57:13 student celery-3[29454]:    return self.run(*args, **kwargs)
  31933 Mar 23 06:57:13 student celery-3[29454]:  File "/usr/lib/python3.7/site-packages/pagure/lib/tasks_utils.py", line 36, in decorated_function
  31934 Mar 23 06:57:13 student celery-3[29454]:    return function(self, session, *args, **kwargs)
  31935 Mar 23 06:57:13 student celery-3[29454]:  File "/usr/lib/python3.7/site-packages/pagure/lib/tasks.py", line 119, in generate_gitolite_acls
  31936 Mar 23 06:57:13 student celery-3[29454]:    helper.generate_acls(project=project, group=group_obj)
  31937 Mar 23 06:57:13 student celery-3[29454]:  File "/usr/lib/python3.7/site-packages/pagure/lib/git_auth.py", line 795, in generate_acls
  31938 Mar 23 06:57:13 student celery-3[29454]:    f.write(repo)
  31939 Mar 23 06:57:13 student celery-3[29454]:  File "/usr/lib64/python3.7/tempfile.py", line 481, in func_wrapper
  31940 Mar 23 06:57:13 student celery-3[29454]:    return func(*args, **kwargs)
  31941 Mar 23 06:57:13 student celery-3[29454]: TypeError: a bytes-like object is required, not 'str'
Comment 3 Avi Alkalay 2019-03-23 11:33:22 UTC 
I get this type of error all the time on /var/log/httpd/error_log:


[Fri Mar 22 23:18:31.527411 2019] [wsgi:error] [pid 29965:tid 139915157030656] [remote fd:900d:900d:0:d88f:5bf2:20f5:f090:61175] /usr/lib64/python3.7/site-packages/sqlalchemy/engine/default.py:536: Warning: (1265, "Data truncated for column 'date' at row 1")
[Fri Mar 22 23:18:31.527449 2019] [wsgi:error] [pid 29965:tid 139915157030656] [remote fd:900d:900d:0:d88f:5bf2:20f5:f090:61175]   cursor.execute(statement, parameters)



And this:


[Sat Mar 23 07:24:10.920982 2019] [wsgi:error] [pid 2027:tid 140364017178368] [remote 2804:14c:183:9855:91f:3d8c:8ea2:46f7:50347] 2019-03-23 07:24:10,920 [INFO] pagure.lib.notify: Sending blinker signal to: pagure - topic: issue.comment.added
Exception ignored in: <function Connection.__del__ at 0x7fa909308d08>
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/redis/connection.py", line 470, in __del__
NameError: name 'Exception' is not defined
Exception ignored in: <function Connection.__del__ at 0x7fa909308d08>
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/redis/connection.py", line 470, in __del__
NameError: name 'Exception' is not defined


Even if Redis is running
Comment 4 Avi Alkalay 2019-03-23 11:42:04 UTC 
Created attachment 1547254 [details]
Patch/edits for /etc/httpd/conf.d/pagure.conf

Apply with:

patch -p0 /etc/httpd/conf.d/pagure.conf < pagure.httpd.conf.patch
Comment 5 Avi Alkalay 2019-03-23 11:43:13 UTC 
Created attachment 1547255 [details]
Patch/edits for /etc/pagure/pagure.cfg

Apply with:

patch -p0 /etc/pagure/pagure.cfg < pagure.cfg.patch
Comment 6 Avi Alkalay 2019-03-23 11:44:11 UTC 
Created attachment 1547256 [details]
Patch/edits for /etc/pagure/alembic.ini

Apply with:

patch -p0 /etc/pagure/alembic.ini < alembic.ini.patch
Comment 7 Neal Gompa 2019-03-23 15:49:28 UTC 
So in other distributions, there's a README file that describes post-install manual setup steps. Let me see if I can at least put together one for Fedora's package.
Comment 8 Avi Alkalay 2019-03-23 17:26:53 UTC 
(In reply to Neal Gompa from comment #7)
> So in other distributions, there's a README file that describes post-install
> manual setup steps. Let me see if I can at least put together one for
> Fedora's package.

Yes please add this post-install instructions.
One that is tested and proven to work.
You’ll see system files that were supposed to be read only, need changes to work.
Comment 9 Avi Alkalay 2019-03-28 20:37:29 UTC 
Any updates on this ?

Anybody is working on this?

Should I wait for a fix in the next days? Or should I move on to another solution ?

Thank you in advance
Comment 10 Neal Gompa 2019-03-28 20:46:27 UTC 
I'm working on this for the Pagure 5.4 update. I'll hopefully have that ready as an update in a day or two.
Comment 11 Avi Alkalay 2019-03-29 10:23:17 UTC 
Excellent!

Can you share conclusions so far ?

Please make sure there will be Fedora 29 packages for stable.

Thank you very much !
Comment 12 Fedora Update System 2019-03-29 13:43:40 UTC 
pagure-5.4-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-e11eb7fa02
Comment 13 Fedora Update System 2019-03-29 13:43:44 UTC 
pagure-5.4-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-7fc1f4f099
Comment 14 Fedora Update System 2019-03-29 13:43:53 UTC 
pagure-5.4-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-c904f00e0d
Comment 15 Fedora Update System 2019-03-29 19:09:01 UTC 
pagure-5.4-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2019-e11eb7fa02
Comment 16 Fedora Update System 2019-03-29 20:32:23 UTC 
pagure-5.4-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-c904f00e0d
Comment 17 Fedora Update System 2019-03-29 22:41:22 UTC 
pagure-5.4-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-7fc1f4f099
Comment 18 Fedora Update System 2019-04-01 00:01:19 UTC 
pagure-5.4-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
Comment 19 Fedora Update System 2019-04-01 00:38:08 UTC 
pagure-5.4-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
Comment 20 Fedora Update System 2019-04-01 03:25:16 UTC 
pagure-5.4-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.
Comment 21 Avi Alkalay 2019-04-01 19:03:36 UTC 
This still doesn't work.

I created an alias user ID and group ID "git" to be the same as gitolite3 so I can follow your instructions.

I can create repos through UI but they keep that "permissions on this repository are being updated" message forever.
Creating repos as a mirror of another repo doesn't clone it.

I also noticed that running as root "pagure-admin refresh-gitolite —all" messes some gitolite file permissions, specially ~/.ssh/authorized_keys, in a way that makes it stop working.

The more Pagure's default configuration files are far from the underlying and essential gitolite and other system components, the more fiddling is necessary and increases the chance for it not to work.
Comment 22 Neal Gompa 2019-04-09 13:13:08 UTC 
@Avi,

You're supposed to create a new user for it, and give it a new home directory. Gitolite is executed by Pagure under that user. You're breaking it by trying to use the gitolite package defaults.