the long path of unittesting JS in Plone (in Jenkins of course)

One of the main goals of Mockup (the new JavaScript story for Plone) is to unittested.

Before Mockup there was no systematic way to unittest JavaScript, you could only do some really slow integration tests via Robot Framework (with its Plone integration plone.app.robotframework).

Now with Mockup he have the other side of the coin as well: we can not only check how the code  integrates within Plone but we can also check its logic with fast unittests.

There’s already Travis integration for Mockup and Mockup-core (a building block for Mockup), but as Plone is using Jenkins it makes a lot of sense to run them there as well.

So, how that’s done? I’m glad you ask, exactly like this:

  1. Add Nodejs on Jenkins nodes (plone.jenkins_node)
  2. Configure Jenkins so that tests will find where Nodejs is (plone.jenkins_server)
  3. Configure Grunt (a JavaScript test runner) for Jenkins (mockup-core)
  4. Add a make target for the new Grunt configuration (mockup)
  5. Create a Jenkins job that runs it (jenkins.plone.org)
  6. Profit!

All the code is there, the last three pull requests are pending to be merged.

Once that’s done jenkins.plone.org will be able to run our JavaScript unittests!!

Happy hacking!

P.D: remember that tomorrow is WPOD!

WPOD is approaching, are you ready for it?

This Friday Mai 29th we are celebrating the 2nd edition of WPOD!

Look around your city, meet with friends, make new ones and contribute to Plone!

As usual IRC will be full of contributors willing to help smooth your path into Plone, so don’t be shy and ask ;)

For Berliners: we will meet again at der Freitag offices, please RSVP at meetup.

See you  there/in IRC!

Fast clones for faster CI

Did you ever got tired waiting a repository (managed by mr.developer) to be cloned?

Wait no more! mr.developer 1.32 comes with two new options to make your life easier:

  • a global git-clone-depth option that allows to fetch only the amount of history specified on this option (i.e. git-clone-depth = 1 for only one single commit)
  • a per-source depth option that allows to specify the same as git-clone-depth but only for that specific source

What’s the benefit of this? See it for yourself:

git clone https://github.com/plone/buildout.coredev.git test-full
du -sh test-full
36M test-full

git clone https://github.com/plone/buildout.coredev.git --depth=1 test-shallow
du -sh test-shallow
1,5M    test-shallow

That’s a 24x saving.

So think about your CI environments where they are downloading over and over the whole repository information while you only care about the very last changes.

Think about some remote environments (mobile connection while you are on a train?)

Update: it seems that is completely broken, sorry for that, working on a fix.

Update 2: mr.developer 1.33 released, thanks @fschulze for the new release!

zc.buildout tricks

Maybe everyone is already aware of them, but just in case…

zc.buildout is THE building block that assembles Plone together.

It’s been around for quite a while (+10 years) and it has plenty of features.

Two of them which I’m enjoying a lot lately are:

./bin/buildout install code-analysis

install allows you to override which parts are going to be installed and thus it allows, like in the example above, reduce the amount of packages to fetch and things to build. Which for something like jenkins.plone.org can make quite a lot of sense to use it more and more.

./bin/buildout annotate

annotate outputs the complete configuration that buildout will use as if it was everything in a single file. This is great for debugging “why my configuration is overriden or not being used at all” kind of errors.

1st WPOD recap

Last Friday the first WPOD happened around the globe.

Here is what was done on the Berlin gathering:

Hopefully other participants around the globe will share their achievements as well!

Are you already planning the next WPOD? Mark it on your calendar May 29th.

Happy hacking!

Testing pull requests and multi-repository changes

At Plone we use Continuous Integration (with Jenkins) to keep us aware of any change made on any of our +200 of packages that break the tests.

Thus making it feasible to spot where the problem was introduced, find the changes that were made and report back to the developer that made the changes to warn him/her about it.

A more elaborate step by step description is on our CI rules, be sure to read them!

At the same time though, we use GitHub pull requests to make code reviews easy and have a way to provide feedback and context to let everyone give their opinion/comment on changes that can be non-trivial.

Sadly, pull requests and multi-repository changes can not be tested directly with Jenkins, yes, there is a plugin for that, but our CI setup is a bit (note the emphasis) more complex than that…

Fortunately we came up with two solutions (it’s Plone at the end, we can not have only one solution :D)

Single pull requests

If you have a pull request on a core package that you want to test follow these steps:

  1. Get the pull request URL
  2. Go to http://jenkins.plone.org and login with your GitHub user
  3. Go to pull-request job: http://jenkins.plone.org/job/pull-request (you can see it always at the front page of jenkins.plone.org)
  4. Click on the Build with Parameters link on the left column
  5. Paste the pull request URL from #1 step
  6. Click on Build

Once it runs you will get an email with the result of the build. If everything is green you can add a comment on the pull request to let everyone know that tests pass.

Note: it’s your responsibility to run that job with your pull request and that changes made on other packages after tests started running can still make your pull request fail later on, so even if the pull-request job is green, be sure to keep an eye on the main jenkins jobs as soon as you merge your pull request.

Example: Remove Products.CMFDefault from Products.CMFPlone (by @tomgross)

Pull request: https://github.com/plone/Products.CMFPlone/pull/438

Jenkins job: http://jenkins.plone.org/job/pull-request/80

Multi-repository changes

When the changes, like massive renamings for example, are spread over more than one repository the approach taken before doesn’t work, as the pull-request Jenkins job is only able to change one single repository.

But we, the CI/testing team, have another ace on our sleeves: create a buildout configuration in the plips folder on buildout.coredev (branch 5.0) that lists all your repositories and which branch should be used, see some examples.

Once you have that cfg file, you can politely ask the CI team to create a Jenkins job for you. They will make a lot of clever things to make that work on jenkins (3 lines change plus following some instructions) and sooner or later a new Jenkins job will show up on the PLIPs tab on jenkins.plone.org.

Rinse and repeat!

Extra bonus and caveats

All Jenkins jobs, be it the pull-request, PLIPs and of course the core jobs, are configured to send an email to the one that triggered the job, so don’t worry about how long do they take to run, once they are finished you will get notified.

The caveat is that the above is only valid for changes targeting Plone 5. We didn’t put the extra effort to make it sure it also works for pull requests (either single or multi-repository) aimed at Plone 4.3. It’s quite trivial to add it for multi-repositories, a bit more work to make it run on single pull requests, still feasible to do if there’s enough people asking for it.

Hopefully the amount of pull requests for Plone 4.3 will decrease more and more as Plone 5 is getting closer and closer one pull request at a time :)

Now there’s no excuse on pushing changes to master without having tested them before on jenkins.plone.org!

Proposals on improvements and suggestions are always welcome on the issue tracker for jenkins.plone.org GitHub repository. Help on handling all those issues are, of course, also welcomed!

Happy testing!

WPOD

WPOD: World Plone Office Day

During this year’s PLOG I presented the simple idea behind WPOD:

  • every last Friday of the month
  • meet somewhere (remotely is fine as well, see below)
  • hack on Plone instead of your regular work
  • Rinse and repeat

That’s it, as simple and as easy as it can be.

Mark on your calendars every last Friday of the month, you have an appointment with the Plone community to bring Plone one step further ahead!

WPOD in Berlin

Preparations are being made for the first ever WPOD in Berlin that my company will gladly host. If you happen to be around Berlin, please contact me telling that you are coming!

The location is Hegelplatz 1, 10117 Berlin.

You are welcome during all day long. Plonistas are expected to come during the morning, enjoy some lunch together, and hack away until late afternoon.

WPOD around the world

If you happen to not be in Berlin, fear not, an irc channel will be available (#sprint on irc.freenode.net) so you can feel the same experience as in any other city hosting a WPOD.

Credit where credit’s due

That’s not an original idea of mine, nor is something that I thought of myself alone, Philip Bauer already tried to present the very same idea on last year’s Plone Conference in Bristol.

Later on, during the Bycicle sprint in Berlin, Stefania and I discussed about it and defined the format as it will start with.

Thanks to them for their bright minds and clever ideas!

Future

Within 10 days the first WPOD will happen, Plonistas will hack/plan/design away and Plone will get better and better.

I hope that other cities and individuals alike will start participating on WPOD, the more we are the bolder the change we will make.

There are some plans to put all the relevant information regarding WPOD on plone.org, either on the current website, or even better on the newer plone.org that is on the making (watch here for tickets ready to be fixed by any of you!).

Happy hacking!

Update: a meetup has been created, please RSVP there.

first GUADEC!

Following the meme…

Not for me nor for Sílvia of course, but for Ona!

She even attended some talks! :)

Ona on a talk

Sadly we only stayed for some days, more to catch with some of you[1] and the weather was quite hard for her.

All in all a short but intense GUADEC!

It was really nice to meet all of you again. Hopefully by next year Ona is going to speak some so she can send her first paper :D

[1] Now we know that Patri and Calvaris will take care of her for next GUADEC :)

GUADEC tip of the day

As everyone attending GUADEC is right now enjoying the party at Fléda, and Sílvia and me are missing it for a good reason :) let me give you a tip for the remaining days:

Are you feeling hungry and looking for a delicious place to eat some arabic/vegeterian food here at Brno? Look no longer Kupé is what you are looking for!

Not only the food is great but their stuff is really helpful, they speak perfect English (and the menu is also available in English) and the smoothies are awesome!

Oh and as a late notice:

I am attending GUADECI am attending GUADECI am attending GUADECTriple as for me, Sílvia and Ona :)