Faster tests and python 3

2x faster

Thanks to the Plone Foundation that is sponsoring a new dedicated server for our jenkins nodes (the machines that run our test suite on every change),
the Plone community is starting to enjoy faster builds (twice as fast!).

If your pull request or jenkins job runs on Node 5 or Node 6 you will notice it :-)

Please report any misbehavior on jenkins.plone.org github project if you happen to notice something not working as expected.

Happy testing!

Python 3

As the Zope community is getting closer and closer to make a Zope release Python 3 compatible, us, the Plone community have to step up and do the same.
For that, we are working on, guess what, a new Jenkins job that will only run the test suite of all packages that are known to work on Python 3 already.

The initial list isn’t that big, roughly 10 packages so far, but as more and more Zope packages are updated, the more Plone packages that can be made compatible as well.

The upcoming Alpine Sprint will be dedicated towards this: getting a Plone version compatible with the current Zope versions, which will eventually lead to this Zope on python 3 target (aimed to be released by the end of next year).

Happy hacking!

Easily extract commits from a messed up branch

Once in a while you encounter branches that are have been over-used, i.e. multiple persons added commits there for unrelated issues.

How to get around/solve that? I’m glad you ask :-)

Today I was facing something similar at work, where the messed up branch in question had some commits already merged into master but there were some important other parts to be extracted from it.

So I used three tools:

  • git log --graph --decorate --pretty=oneline --abbrev-commit --all (with an alias git fulllog)
  • a git graphical visualizer (gitg for instance)
  • a simple plain text editor (gedit for instance)

Workflow

Fire up git fulllog in a terminal and copy & paste as much information as you need into the text editor.

Fire up the git graphical visualizer and check each relevant commit what’s doing.

Once you know what it is about annotate each commit so you know which commits relate together, i.e. from

ae0ee04 My commit message

to

ae0ee04 11111 My commit message

This way once everything is annotated is really simple to just grab all the 11111 commits and git cherry-pick them in a new branch :-)

Pro tip: as you are editing a plain text file you can keep removing lines and adjusting the indentation, suddenly you realize how things keep fitting together!

Done!

Plone testing team meeting

After a Plone Conference there’s always a lot of energy and willingness to move Plone one step forward.

The testing and CI team in Plone is no different, and for that I created a Doodle to see who is interested and what would be the best time to meet during next week.

If you are interested in the team, or know anyone who would be, please share that link and join the meeting!

Plone on python 3 status

2020 is approaching fast, one day at a time, and besides being a nice catchy year, it will be also the year where Python 2.7 will no longer get any security updates.

What that means for Plone? We should hurry up and get a python 3 porting of the whole stack, yes, including Zope, ZTK, ZODB and all the tooling around (zc.buildout, etc etc).

Fortunately quite some tooling and Zope/ZTK/ZODB is already updated and there’s ongoing effort on porting the remaining parts.

The big elephant on the room blocking any porting effort of Plone to python 3 was RestrictedPython, a python distribution that, quoting itself: provides a restricted execution environment for Python.

Note the past on the previous sentence.

Since RestrictedPython is being worked, now it’s high  time for the other python distributions from Plone to be also made compatible with Python 3. Stay tuned for the Plone Conference 2016 sprint report!

Copy&pasting&adapting a set of scripts to track the progress of the porting for the Zope foundation github organization, results here, I made the same but tracking what Plone 5.1 (including our testing environment) looks like on Python 3:

http://jenkins.plone.org/py3/

The code is on collective, so feel free to update the package list.

During the Plone Conference 2016  there is quite some work put on either reducing the amount of dependencies, or updating our stack to use newer (already python 3 compatible) part of the underlying stack.

The clock is ticking and Plonistas all over the world are working hard on it!

python 3 support one (of the many) step closer

During the Barcelona sprint (report from Paul and Ramon) there was quite some work to bring dexterity related packages to work both in python 2.7 and python 3.5.

That work is mostly still pending to be merged, because we had a blocker: our current CI infrastructure (namely jenkins master and its nodes) run on a version of Ubuntu1 that does not provide python 3.5 by default.

So it would be a bit irresponsible to merge those changes without a way to ensure that the effort that was put during that sprint is not long forgotten and had to be done again from scratch some months later.

As reports about the newer Ubuntu version regarding python and plone are not that encouraging, and also due to other reasons, I decided to take the longest but long-term best approach: enters gforcada.compile-python!

An ansible role

So I decided to create an ansible role, given that our jenkins CI setup is already using some, and Plone community is also favoring it, to install all the system dependencies to compile Python 2.6, 2.7 and 3.5.

As extras I added:

  • install virtualenv on python 2.6 and 2.7 (on python 3.5 is already available)
  • install system dependencies for Pillow and lxml

See its README for all the details and more.

With that and vagrant I was able to test that a buildout.coredev (branch 5.1) runs all its tests without a problem :-)

I was tempted to add pypy as well, but I was too lazy/busy for that, if anyone feels like it, pull requests are always welcome!

I hope you find it useful and happy hacking!

  1. 14.04 for the curious []

Berlin 2016 sprint update

This week some Plonistas, unfortunately fewer than expected, meet at IN-Berlin to work towards a brighter future for Plone.

Some of us resumed our work already started in Innsbruck sprint early this year while some other topics grew out of discussions and trying things out.

I will never get to write as good as Paul reporting on Barcelona’s sprint, but hey, I will do my best:

In no special order:

Just by trying out the newly shiny plone.org website (congrats plone.org team!!), some bugs where discovered and reported, work done by Stephan Klinger.

The PLIP about assimilating collective.indexing to Plone core finally got all its tests green and is about to be ready for review, work done by Maik Derstappen and Gil Forcada.

Python 3 support was a hot topic during the sprint, so with that in mind Florian Pilz and Michael Howitz created a new tool, zodb.py3migrate, that allows you to convert existing ZODB databases to python 3! Best of all, its great documentation! Be aware that the migration is done in place!

Our Jens2 team was not quadratic this time around, but Jens Vagelpohl continued trying to tame Pluggable Authentication Service (aka PAS).

Did I say that Python 3 was a hot topic? Thomas Lotze with googling support from Gil Forcada took the problem head on and decided that, as Thomas had not that much time (unfortunately he was around only 3 days), work on cracking hard nuts: C extensions. And indeed he did! AccessControl and DocumentTemplate at least compile on python 3.5 (throwing quite some warnings, but hey, have you ever seen Archetypes being installed?). Best of all is that Python 2.6 and 2.7 already have quite some macros for forward compatibility, so the bits that are compatible were pull requested and they are already merged!

Unfortunately that work, C extensions, came to a halt as we hit with RestrictedPython. After long discussions between Thomas, Gil and Maik we decided to make a video conference with the Barcelona sprint, Eric Bréhault, as well as Alexander Loechel, who couldn’t join is in Berlin, but that he already had worked on the topic while in Insbruck early this year (read about his findings). The discussion went really well and Alexander offered himself to continue working on it, stay tuned!

Maik and Stephan meanwhile where not happy with the constellation of form packages that we currently have (z3c.form, plone.autoform, plone.supermodel, plone.z3cform, plone.app.z3cform…) and made yet another package1… just kidding! No, instead they worked on improving the forms related documentation: getting rid of grok2 and making sure that the documentation is consistent3. For that Stephan created the linked above package and Maik was working on moving the remaining useful bits of plone.directives.form to plone.autoform. Stephan started a discussion on the topic on community.plone.org.

Lastly, I put my jenkins/mr.roboto hat on and added some more functionality to mr.roboto: warn which pull request jenkins jobs need to ran to know if a pull request can be safely merged and auto update buildout.coredev’s checkouts.cfg whenever a pull request gets merged. The logic is fairly trivial, but gathering all the pieces together to drive the control is far from it, and testing is yet a complete other matter (thanks mock!!).

All in all we had fun, lots of things got done or discussed and, as with every sprint, everyone is looking for the next one!

Happy hacking!

 

  1. will be soon moved to collective []
  2. and hidden grok dependencies lying in plone.directives.form and plone.directives.dexterity []
  3. at the beginning there are some fields, and two pages later, magically, fields are different! []

Robots are taking over

Since some time ago Jenkins, our continuous integration system, already reports back to pull requests the job build status, just like other popular CI systems, there’s some documentation on how to trigger jenkins builds on your pull requests.

On selected packages Jenkins is also reporting to GitHub if there are any code analysis errors, see the list of packages and how to run code analysis on your own.

mr.roboto is a pyramid app that works as a webservice providing integration between our plone/collective GitHub organization and Jenkins.

Some of you might have noticed that since last week is adding status updates on pull requests to check if the authors of the pull request have signed the Plone Contributors Agreement.

Since today it is also checking if pull requests are modifying the changelog entry file (namely CHANGES.rst), this way, no change will go unnoticed.

Next step is to warn about which pull requests jobs need to be run for a given pull request. With the current three major releases (4.3, 5.0 and 5.1) being tested is quite a challenge to know which major versions a pull request should be tested against.

As usual, please report any problem on jenkins.plone.org issue tracker.

Happy hacking!

Ready, set, sprint!

In less than a week Plonistas will be meeting in Berlin to sprint on Plone.

The topic this time is to continue the long endeavor to cleanup and slim down our stack, Zope included.

Thanks to IN-Berlin we will sprint at their offices. All information on how to get there is on the sprint page.

The main communication channels with the sprints will be the usual #sprint IRC channel (on freenode) and we will try to maintain a hangout channel open as well (we will provide the link later on).

On social networks you can try to keep an eye on #berlinsprint hashtag.

Happy hacking!

Berlin strategic sprint 2016 update

In nearly a month the Berlin strategic sprint 2016 will start, and finally we can announce where it will be hosted: IN-Berlin offices!

You can find all information in the link above.

There are still some spots left, the venue is big enough to accommodate around 15 to 20 persons, so if you want to join the sprint, now is the time to sign up and register. As it’s an strategic sprint the Plone Foundation could help with the travel costs if that’s an issue for you to attend, contact them!

And best of all: spring has already arrived in Berlin, so we can expect good weather1 and a full city to enjoy!

See you all in Berlin!

  1. hopefully not famous last words []

How to test pull requests on jenkins.plone.org

Although for me it looks clear and straightforward, for some it may not be the case, so I decided to add a brief document explaining it. It should show up in Read The Docs here: http://jenkinsploneorg.readthedocs.org/en/latest/run-pull-request-jobs.html

If Read The Docs still hasn’t catch up the source in GitHub is easily readable as well: https://github.com/plone/jenkins.plone.org/blob/master/docs/source/run-pull-request-jobs.rst

Bonus point: I made my first screencast demoing it! Watch it in all its glory.