Bug 1064219

Summary: The existing iptables port proxy rules are missing after server upgrade and gear migrate
Product: OpenShift Online Reporter: Meng Bo <bmeng>
Component: ContainersAssignee: Lokesh Mandvekar <lsm5>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: admiller, dmcphers, jhonce, jkeck, lsm5
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-03-12 03:07:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1070362    
Bug Blocks:    

Description Meng Bo 2014-02-12 09:10:27 UTC
Description of problem:
The iptables port proxy rules config files are moved from /etc/openshift/ to /var/lib/openshift/.httpd.d/. But after server upgrade and gear migrate, the existing configs are not imported to the new path.

This makes all the scalable app cannot work well.

# cat /var/lib/openshift/.httpd.d/iptables.filter.rules |wc -l
1
# cat /var/lib/openshift/.httpd.d/iptables.nat.rules |wc -l
1

# cat /etc/openshift/iptables.filter.rules.bak |wc -l
37
# cat /etc/openshift/iptables.nat.rules.bak |wc -l
37


Version-Release number of selected component (if applicable):
devenv-stage_689 to devenv_4364

How reproducible:
always

Steps to Reproduce:
1. Create scalable app with db add on stage ami
2. Do search upgrade and gear migrate by following the release ticket
3. Check the scalable apps after upgrade

Actual results:
The existing scalable app cannot work well since the iptables port proxy rules are missing.

Expected results:
The rules should be migrated to the new path.

Additional info:

Comment 1 Lokesh Mandvekar 2014-02-12 21:58:15 UTC
Hi,

Could you try this please:

1. cp /etc/openshift/iptables.filter.rules /var/lib/openshift/.httpd.d/
2. cp /etc/openshift/iptables.nat.rules /var/lib/openshift/.httpd.d/
3. cp /etc/openshift/iptables.filter.rules.bak /var/lib/openshift/.httpd.d/
4. cp /etc/openshift/iptables.nat.rules.bak /var/lib/openshift/.httpd.d/

5. Perform upgrade (since files are copied over in 1-4, the new files installed here will have '.rpmnew' appended to them.

6. Verify state of files in /etc/openshift and /var/lib/openshift/.httpd.d 

7. delete said files in /etc/openshift if everything looks good in /var/lib/openshift/.httpd.d

Comment 2 Lokesh Mandvekar 2014-02-12 22:17:55 UTC
(In reply to Lokesh Mandvekar from comment #1)
> Hi,
> 
> Could you try this please:
> 
> 1. cp /etc/openshift/iptables.filter.rules /var/lib/openshift/.httpd.d/
> 2. cp /etc/openshift/iptables.nat.rules /var/lib/openshift/.httpd.d/
> 3. cp /etc/openshift/iptables.filter.rules.bak /var/lib/openshift/.httpd.d/
> 4. cp /etc/openshift/iptables.nat.rules.bak /var/lib/openshift/.httpd.d/
> 
> 5. Perform upgrade (since files are copied over in 1-4, the new files
> installed here will have '.rpmnew' appended to them.

ohh btw, the upgrade step will delete /etc/openshift/iptables.*.rules (but will not delete the backups in there), but it shouldn't be any trouble since all those files have been copied over to /var/lib/openshift/.httpd.d in 1-4, just fyi

> 
> 6. Verify state of files in /etc/openshift and /var/lib/openshift/.httpd.d 
> 
> 7. delete said files in /etc/openshift if everything looks good in
> /var/lib/openshift/.httpd.d

Comment 3 Meng Bo 2014-02-13 07:32:47 UTC
Tested on devenv_4370, with comment#2. The scalable app working well after migration.

@lsm5
I think we should add this to the release ticket of this sprint. The issue is already affected the latest INT deploy.

Comment 4 Lokesh Mandvekar 2014-02-13 07:42:36 UTC
(In reply to Meng Bo from comment #3)
> Tested on devenv_4370, with comment#2. The scalable app working well after
> migration.
> 
> @lsm5
> I think we should add this to the release ticket of this sprint. The issue
> is already affected the latest INT deploy.

Release ticket updated. Let me know if Comment 12 addresses it https://engineering.redhat.com/trac/Libra/wiki/Releases/2.0.40 

Thanks,

Comment 5 openshift-github-bot 2014-02-13 19:59:37 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/f8458142512e8a0ee35076badffc8101bced966d
Bug 1064219 - handle iptables rules

Via rpm itself, copy over iptables rules configs and backups from
/etc/openshift to /var/lib/openshift/.httpd.d before installing
new files and delete old files after the upgrade.

also, do not delete *.rpmnew in %post

	modified:   node/rubygem-openshift-origin-node.spec

Comment 6 Lokesh Mandvekar 2014-02-14 00:02:45 UTC
@bmeng: could you please test this latest change as well?

This change should get rid of the need to manually move/copy the iptables files to the new dir before the yum upgrade step. Things should get handled via yum upgrade itself.

Let me know how that works out. Thanks.

Comment 7 Meng Bo 2014-02-14 06:03:17 UTC
Hi, I have tried on devenv_4375, the package version is:
rubygem-openshift-origin-node-1.20.5-1.git.0.03e17c8.el6.noarch

After yum update, the existing iptables rules were not contained in the iptables.*.rules files but in the iptables.*.rules.bak.
That means the rules are not proper imported.

Assign this bug back to review the pacakge.



[root@ip-10-181-213-90 ~]# cat /var/lib/openshift/.httpd.d/iptables.nat.rules |wc -l
1
[root@ip-10-181-213-90 ~]# cat /var/lib/openshift/.httpd.d/iptables.filter.rules |wc -l
1
[root@ip-10-181-213-90 ~]# cat /var/lib/openshift/.httpd.d/iptables.filter.rules.bak |wc -l
13
[root@ip-10-181-213-90 ~]# cat /var/lib/openshift/.httpd.d/iptables.nat.rules.bak |wc -l
13

Comment 8 openshift-github-bot 2014-02-14 23:30:41 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/97beabc89e19b63515d99003a6d99fda3b9266f9
Bug 1064219 - revert iptables location change

migration wasn't clean enough, so reverting back until there's a cleaner
solution

	modified:   common/bin/oo-diagnostics
	modified:   node/misc/bin/oo-iptables-port-proxy
	modified:   node/misc/sbin/oo-admin-ctl-iptables-port-proxy
	modified:   node/rubygem-openshift-origin-node.spec

Comment 9 Dan McPherson 2014-02-14 23:56:46 UTC
https://github.com/openshift/origin-server/pull/4770

Comment 10 Meng Bo 2014-02-17 10:35:59 UTC
Checked upgrade from devenv-stage_689 to latest candidate repo.

The change has been reverted. And the scalable app works well now.

Move bug to verified.