Bug 1018343 - [deploy] Make sure existing Jenkins build scripts are forwards-compatible with new deploy code
[deploy] Make sure existing Jenkins build scripts are forwards-compatible wit...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Andy Goldstein
libra bugs
:
: 1020838 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-11 13:56 EDT by Andy Goldstein
Modified: 2015-05-14 19:29 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-29 19:48:53 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andy Goldstein 2013-10-11 13:56:28 EDT
With the new deployment code, we need to make sure that existing Jenkins jobs continue to function after the upgrade happens. For example:

- create app, enable jenkins
- make sure the build works
- upgrade system to new deploy code
- push a change to the app
- make sure the build continues to work
Comment 1 Dan McPherson 2013-10-18 16:39:03 EDT
*** Bug 1020838 has been marked as a duplicate of this bug. ***
Comment 2 Liang Xia 2013-10-25 03:11:15 EDT
This can be reproduced on INT(devenv_3942).

Description of problem:
When pushing changes to existing app after server migration, met following errors:
remote: An error occurred executing 'gear postreceive' (exit code: 1)
remote: Error message: Failed to execute: 'control post-receive' for /var/lib/openshift/5264c51403ef642922000319/jenkins-client

Version-Release number of selected component (if applicable):
INT(devenv_3942)

How reproducible:
always

Steps to Reproduce:
1.create an jbossews-2.0 app with jenkins-client.
2.migrate the server.
3.make some change in the local repo and git push.

Actual results:
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins-lxiamigrate.int.rhcloud.com/job/jbossews2-build
remote: 
remote: Waiting for build to schedule.....................................................Done
remote: Waiting for job to complete....................................................Done
remote: FAILED
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
remote: An error occurred executing 'gear postreceive' (exit code: 1)
remote: Error message: Failed to execute: 'control post-receive' for /var/lib/openshift/5264c51403ef642922000319/jenkins-client
remote: 
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://5264c51403ef642922000319@jbossews2-lxiamigrate.int.rhcloud.com/~/git/jbossews2.git/
   c5bba71..5454799  master -> master

Expected results:
Changes deployed successfully without errors.
Comment 3 Andy Goldstein 2013-10-25 15:33:41 EDT
https://github.com/openshift/origin-server/pull/4003
Comment 4 Qiushui Zhang 2013-10-28 01:48:30 EDT
Current INT is devenv_3948.
We will check it when next INT ready.
Comment 5 Qiushui Zhang 2013-10-28 03:36:39 EDT
Jenkins build after migration with current devenv_3953 still fails.

[openshift@dhcp-10-143 jbossews1]$ vi src/main/webapp/index.html 
[openshift@dhcp-10-143 jbossews1]$ gitpush 
[master 45b178e] this is a comment
 Committer: openshift <openshift@dhcp-10-143.nay.redhat.com>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly:

    git config --global user.name "Your Name"
    git config --global user.email you@example.com

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

 1 file changed, 1 insertion(+), 1 deletion(-)
warning: push.default is unset; its implicit value is changing in
Git 2.0 from 'matching' to 'simple'. To squelch this message
and maintain the current behavior after the default changes, use:

  git config --global push.default matching

To squelch this message and adopt the new behavior now, use:

  git config --global push.default simple

See 'git help config' and search for 'push.default' for further information.
(the 'simple' mode was introduced in Git 1.7.11. Use the similar mode
'current' instead of 'simple' if you sometimes use older versions of Git)

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for jbossews1-migration1.dev.rhcloud.com has changed,
and the key for the corresponding IP address 174.129.73.153
is unchanged. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /home/openshift/.ssh/known_hosts:199
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
0b:9d:8d:1e:a5:30:cc:6c:9c:39:80:0f:51:34:cf:a5.
Please contact your system administrator.
Add correct host key in /home/openshift/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/openshift/.ssh/known_hosts:47
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
Counting objects: 11, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 470 bytes | 0 bytes/s, done.
Total 6 (delta 4), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins1-migration1.dev.rhcloud.com/job/jbossews1-build
remote: 
remote: Waiting for build to schedule......................................Done
remote: Waiting for job to complete....................................................................................Done
remote: FAILED
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
remote: An error occurred executing 'gear postreceive' (exit code: 1)
remote: Error message: Failed to execute: 'control post-receive' for /var/lib/openshift/526dc7e5cab0818f3500017f/jenkins-client
remote: 
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://526dc7e5cab0818f3500017f@jbossews1-migration1.dev.rhcloud.com/~/git/jbossews1.git/
   9bb1dfa..45b178e  master -> master

Mark the bug as assigned.
Comment 6 Meng Bo 2013-10-29 02:30:14 EDT
Checked on latest INT (devenv_3957), found the following error in log for scaling app.



Stopping gear...

HAProxy instance is stopped

Stopping jbosseap cartridge
Sending SIGTERM to jboss:32113 ...

+ rsync --delete-after -azO -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/526f4f0303ef646f000002a2//.m2/ '5264d33003ef64b3a8000174@springs-bmengmint.int.rhcloud.com:~/.m2/'

+ rsync --delete-after -azO -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/526f4f0303ef646f000002a2/app-root/runtime/repo/deployments/ '5264d33003ef64b3a8000174@springs-bmengmint.int.rhcloud.com:${OPENSHIFT_REPO_DIR}deployments/'

+ rsync --delete-after -azO -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/526f4f0303ef646f000002a2/app-root/runtime/repo/.openshift/ '5264d33003ef64b3a8000174@springs-bmengmint.int.rhcloud.com:${OPENSHIFT_REPO_DIR}.openshift/'

+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 5264d33003ef64b3a8000174@springs-bmengmint.int.rhcloud.com 'gear remotedeploy'

Preparing build for deployment
deployment_datetime is required
Build step 'Execute shell' marked build as failure

Archiving artifacts

Finished: FAILURE
Comment 7 Andy Goldstein 2013-10-29 12:30:03 EDT
I have tested PHP and JBoss EWS applications and the Jenkins builds work fine using the previous jenkins_shell_command. What I do is this:

1) create a new application in the latest devenv
2) add the jenkins-client cartridge
3) log in to Jenkins
4) edit the Jenkins job and replace its contents with those from e.g. https://github.com/openshift/origin-server/blob/af3621a263675b16eb4781caa0f44c28054992df/cartridges/openshift-origin-cartridge-jbossews/metadata/jenkins_shell_command.erb (but make sure you keep the upstream_ssh line from the job so it has the right URL)
5) perform a build

At this point, it should work.

If you see this:

Preparing build for deployment
deployment_datetime is required <------

it means that the upgraded gear is either missing the app-deployments/current symlink or it is a broken symlink. I looked at 5264d33003ef64b3a8000174@springs-bmengmint.int.rhcloud.com and found that the "current" symlink is in fact broken. I also noticed that the app is a few days old (from around 10/24) and would recommend retesting with a new app that gets upgraded because we've fixed several upgrade bugs since then.

I believe that the Jenkins builds will work fine after an upgrade once all upgrade issues have been fixed.
Comment 8 Meng Bo 2013-10-30 04:24:39 EDT
Checked on devenv_3966 which upgraded from devenv-stage_528,

Jenkins build for both scale and non-scale apps are working fine now.

Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.