Robots are taking over

Since some time ago Jenkins, our continuous integration system, already reports back to pull requests the job build status, just like other popular CI systems, there’s some documentation on how to trigger jenkins builds on your pull requests.

On selected packages Jenkins is also reporting to GitHub if there are any code analysis errors, see the list of packages and how to run code analysis on your own.

mr.roboto is a pyramid app that works as a webservice providing integration between our plone/collective GitHub organization and Jenkins.

Some of you might have noticed that since last week is adding status updates on pull requests to check if the authors of the pull request have signed the Plone Contributors Agreement.

Since today it is also checking if pull requests are modifying the changelog entry file (namely CHANGES.rst), this way, no change will go unnoticed.

Next step is to warn about which pull requests jobs need to be run for a given pull request. With the current three major releases (4.3, 5.0 and 5.1) being tested is quite a challenge to know which major versions a pull request should be tested against.

As usual, please report any problem on jenkins.plone.org issue tracker.

Happy hacking!

How to test pull requests on jenkins.plone.org

Although for me it looks clear and straightforward, for some it may not be the case, so I decided to add a brief document explaining it. It should show up in Read The Docs here: http://jenkinsploneorg.readthedocs.org/en/latest/run-pull-request-jobs.html

If Read The Docs still hasn’t catch up the source in GitHub is easily readable as well: https://github.com/plone/jenkins.plone.org/blob/master/docs/source/run-pull-request-jobs.rst

Bonus point: I made my first screencast demoing it! Watch it in all its glory.

Numbers

0 to 1k

Yesterday marked the day that Jenkins Job “Pull Request 5.0” hit the 1000 job (right now running the 1016!).

It’s been a long journey to get it to its current status1 but IMHO since the introduction of it our three main Jenkins Jobs for both 4.3 and 5.0 have been far more stable.

Thanks to everyone that reported feedback and is using it!

100k to 0?

Jenkins is not only about tests, code analysis and all other kind of jobs are running on http://jenkins.plone.org. Two of the latest additions are:

  • per package code analysis jobs that report back to github: packages can opt-in2 and are checked with flake8 and other tools3
  • a global code analysis job: exactly the same as the per package jobs mentioned on the previous point, but running code analysis on ALL packages

The 100k is the flake8 error count for the global job, will we be able to bring that down to zero? :-)

Side note: Zope2 and CMFCore are the two (by far) with more code analysis errors, some other packages are probably going to be deprecated so there is no need to clean up them.

Best of both things?

Anyone (you!?) can grab a package clean it up, and run a pull request job to ensure nothing is broken after the clean up and happily merge it.

Happy hacking!

  1. be able to run 4.3 or 5.0 pull requests, allow multiple pull requests per job, report before and after to github just like Travis, allow external forks to be tested… []
  2. fill an issue! []
  3. more about it []

Testing multiple pull requests at once

It probably happened to you, dear reader, every now an then: the new feature you are working on has changes across more than one git repository, so how can this be tested to make sure nothing is broken before merging those nth separate pull requests?

That’s what #126 was about and finally, thanks to WPOD and the company I work for to allowing me take part of it, it’s finally fixed.

Let us know if it does not work as expected!

Testing pull requests and multi-repository changes

At Plone we use Continuous Integration (with Jenkins) to keep us aware of any change made on any of our +200 of packages that break the tests.

Thus making it feasible to spot where the problem was introduced, find the changes that were made and report back to the developer that made the changes to warn him/her about it.

A more elaborate step by step description is on our CI rules, be sure to read them!

At the same time though, we use GitHub pull requests to make code reviews easy and have a way to provide feedback and context to let everyone give their opinion/comment on changes that can be non-trivial.

Sadly, pull requests and multi-repository changes can not be tested directly with Jenkins, yes, there is a plugin for that, but our CI setup is a bit (note the emphasis) more complex than that…

Fortunately we came up with two solutions (it’s Plone at the end, we can not have only one solution :D)

Single pull requests

If you have a pull request on a core package that you want to test follow these steps:

  1. Get the pull request URL
  2. Go to http://jenkins.plone.org and login with your GitHub user
  3. Go to pull-request job: http://jenkins.plone.org/job/pull-request (you can see it always at the front page of jenkins.plone.org)
  4. Click on the Build with Parameters link on the left column
  5. Paste the pull request URL from #1 step
  6. Click on Build

Once it runs you will get an email with the result of the build. If everything is green you can add a comment on the pull request to let everyone know that tests pass.

Note: it’s your responsibility to run that job with your pull request and that changes made on other packages after tests started running can still make your pull request fail later on, so even if the pull-request job is green, be sure to keep an eye on the main jenkins jobs as soon as you merge your pull request.

Example: Remove Products.CMFDefault from Products.CMFPlone (by @tomgross)

Pull request: https://github.com/plone/Products.CMFPlone/pull/438

Jenkins job: http://jenkins.plone.org/job/pull-request/80

Multi-repository changes

When the changes, like massive renamings for example, are spread over more than one repository the approach taken before doesn’t work, as the pull-request Jenkins job is only able to change one single repository.

But we, the CI/testing team, have another ace on our sleeves: create a buildout configuration in the plips folder on buildout.coredev (branch 5.0) that lists all your repositories and which branch should be used, see some examples.

Once you have that cfg file, you can politely ask the CI team to create a Jenkins job for you. They will make a lot of clever things to make that work on jenkins (3 lines change plus following some instructions) and sooner or later a new Jenkins job will show up on the PLIPs tab on jenkins.plone.org.

Rinse and repeat!

Extra bonus and caveats

All Jenkins jobs, be it the pull-request, PLIPs and of course the core jobs, are configured to send an email to the one that triggered the job, so don’t worry about how long do they take to run, once they are finished you will get notified.

The caveat is that the above is only valid for changes targeting Plone 5. We didn’t put the extra effort to make it sure it also works for pull requests (either single or multi-repository) aimed at Plone 4.3. It’s quite trivial to add it for multi-repositories, a bit more work to make it run on single pull requests, still feasible to do if there’s enough people asking for it.

Hopefully the amount of pull requests for Plone 4.3 will decrease more and more as Plone 5 is getting closer and closer one pull request at a time :)

Now there’s no excuse on pushing changes to master without having tested them before on jenkins.plone.org!

Proposals on improvements and suggestions are always welcome on the issue tracker for jenkins.plone.org GitHub repository. Help on handling all those issues are, of course, also welcomed!

Happy testing!