rgrunber

Tag: p2

Eclipse p2 Droplets in Rawhide

So p2 Droplets have been in Rawhide for a little while now, and since then we’ve converted most Eclipse plugins to build using the new format. With the exception of some cases that will be done manually, pretty much everything building with the XMvn macros (%mvn_build, %mvn_install) is guaranteed to be a p2 Droplet after a rebuild. We still support the old format (Dropins), and an installation on rawhide can detect both types, but the goal is to switch completely to Droplets.

If you’re unfamiliar with the term, you can think of p2 Droplets as a new way of packaging one or more Eclipse features. It contains the same jars and feature folders as before but with an additional file (fragment.info) containing some extra data.

The main reason for the switch was that Dropins (using the p2 Reconciler) has been deprecated for a while and has made it much more difficult to diagnose issues when it fails. Eclipse’s first launch actually caches a lot of data to make subsequent ones much faster, and the Reconciler can be a large chunk of that time. I have actually run into cases where installing just an extra package can take the first startup from around 10 seconds, to 3 minutes!

Testing was done with the following setup :

  • Fedora Rawhide Virtual Machine
  • ~700 OSGi bundles on the system
  • ~430 of these bundles to be tested through the Dropins and Droplets approaches
for i in {1..5}; do rm -rf $HOME/.eclipse/ ; (echo exit; echo y;) | /usr/bin/time -f "%E" eclipse -noexit -console ; done;

To test under Dropins, we simply took all the bundles packaged as Droplets and removed their ‘fragment.info’ file, along with the ‘p2.fragments’ line in ‘/etc/eclipse.ini’, just to be sure we completely disabled the new logic.

So what were the results ? Under Dropins, the average startup time was 17.63 seconds, and under Droplets, it was 9.44 seconds. The variance was very small, and nearly the same for both cases so there’s strong evidence that Droplets will be saving a lot of time on Eclipse’s first startup. Of course all subsequent launches (where $HOME/.eclipse is kept) would be much faster (~2s).

So one might ask what’s happened with all the work the Reconciler used to do.

  1. Fedora maintainers do a good job of making sure packages are using the latest version of a library even when upstream isn’t. As a result, things like the Reconciler aren’t as necessary in attempting to satisfy dependencies across different versions
  2. A lot of the logic to calculate dependencies and produce a self-sustaining package is done properly at build-time, and rightfully so because that’s where the provides/requires are generated.

We’ve been working on this change, along with the other aspects of simplifying Eclipse plugin packaging for a while now so it’s nice to see some easy wins immediately from adoption.

Advertisements

Changing the Behaviour of Eclipse’s Update Manager

If you’ve developed plugins for the Eclipse environment, you’re moderately aware that Eclipse’s update manager can behave in strange ways from a user perspective. Things have gotten better with the p2 Remediation Support in Kepler (4.3.0) but what about dependency resolution done by Maven plugins, like Tycho, at build-time ? You get to specify a list of repositories, their content is aggregated, and if your request is satisfiable, it will be satisfied. Of course there’s some criteria p2 will attempt to optimize. For example, preferring highest version with fewest dependencies (minimize transitive closure) from a set of identically named units.

However with the increasing usage of things like software collections, I’m starting to care much more about where and how dependencies are being resolved during a build.

To understand how to even go about changing p2’s behaviour, we first need to know what is actually happening when you request to have some unit installed.  There’s a few different things happening :

  1. Installable Units are collected from the user input (units to install, and repositories provided)
  2. Installable Units are made part of a Profile Change Request
  3. Planner takes the Profile Change Request and delegates solution finding to the Projector
  4. Projector transforms request into a boolean satisfiability problem and delegates to SAT4J library
  5. If solution is found, Projector passes result back to Planner, which creates a plan (set of operations on the profile) to achieve the new state
  6. The plan is then executed on the profile by the Engine

For more information on some of the finer details of (4) I would recommend reading http://www.cril.univ-artois.fr/spip/publications/lash2010.pdf .

As a demonstration we can define an OSGi bundle org.foo.root versioned 1.0.0 which has a Require-Bundle on org.foo.bar (unversioned). We then provide org.foo.bar at various versions (eg. 1.0.0, 5.0.0, 10.0.0, 20.0.0, 50.0.0, 99.0.0). Performing the installation of org.foo.root 10 times yields the following results :

$ for i in {1..10}; do eclipse -nosplash -application org.eclipse.equinox.p2.director -repository fedora:/tmp -installIU org.foo.root -destination $(pwd)/install_$i ; done;
Installing org.foo.root 1.0.0.
Operation completed in 742 ms.
...
...
$ (for f in `find ./install_*/plugins/ -type f`; do basename $f ; done;) | sort | uniq -c
     10 org.foo.bar_99.0.0.jar
     10 org.foo.root_1.0.0.jar

So in every case that we attempted to install org.foo.root, p2 decided to satisfy the version-less requirement org.foo.bar using the latest version available (99.0.0).

Note that the p2 Director will always use the latest version of a root, so to truly test the Projector and SAT4J it is necessary to define some root and see how its dependencies are satisfied, as we did above. Also, if that “fedora:” repository scheme seemed a little strange to you, it’s because we ship a special plugin in Fedora that allows treating filesystem locations as p2 repositories , without needing all that metadata/artifact repository data on disk.

To make p2 prefer certain installable units over others, we need to hook into the component that sets the constraints for the SAT4J Solver. The Projector seems like the place for this and a quick look reveals createOptimizationFunction is probably a good starting point. We’ll want to define our own optimization (instead of OptimizationFunction) function.

To favour certain locations over others, we could look at sets of units that have the same ID, and are not installed, or roots, and assign them weights based on their location. Since SAT4J will attempt to satisfy the contstraints while minimizing the objective function, we can assume that lower weighted units will be preferred over higher weighted ones. In fact, installed/root units have a weight of 1. All we need to do is subclass OptimizationFunction, override createOptimizationFunction to call its parent, and then just modify the weight for the units we care about as per the repository precedence before returning the final list of weighted units.

Once we’re done adding the necessary code, and rebuilding our modifications, we’re ready to test things out.

We now change the structure of our repository as follows :

  • /tmp/repo_priority/low_priority contains org.foo.bar versions 5.0.0, 20.0.0 and 50.0.0
  • /tmp/repo_priority/high_priority contains org.foo.bar version 10.0.0.
  • /tmp/repo_priority/ contains org.foo.bar versions 1.0.0 and 99.0.0
$ export JAVACONFDIRS='/tmp/repo_priority:/tmp/repo_priority/high_priority:/tmp/repo_priority/low_priority'
$ for i in {1..10}; do eclipse -nosplash -application org.eclipse.equinox.p2.director -repository fedora:/tmp -installIU org.foo.root -destination $(pwd)/install_$i -vmargs -Dfedora.p2.scl.order=/tmp/repo_priority/high_priority,/tmp/repo_priority,/tmp/repo_priority/low_priority ; done;
Installing org.foo.root 1.0.0.
Operation completed in 550 ms.
...
...
$ (for f in `find ./install_*/plugins/ -type f`; do basename $f ; done;) | sort | uniq -c
     10 org.foo.bar_10.0.0.jar
     10 org.foo.root_1.0.0.jar

So clearly we’ve made p2 favour units within /tmp/repo_priority/high_priority over others. This was a very basic example, and I didn’t really define what should happen when multiple units are in the same “preferred” repository but hopefully even a modification as basic as this shows the kind of things possible.

Extending the p2 Repository Format in Fedora

We ship a lot of OSGi bundles in Fedora.

$ repoquery --repofrompath=foo,http://kojipkgs.fedoraproject.org/repos/f21-build/latest/x86_64/ --repoid=foo -q --whatprovides "osgi(*)" | wc -l
693

We already allow packagers to have requirements based on the OSGi metadata to some extent. We’re supporting Require-Bundle in specfiles through use of the “osgi(…)” provide but even with this, building Eclipse plugins has always seemed like a giant hack, and even more so when compared to the latest Java Maven Packaging Examples. Building with Tycho requires that your OSGi dependencies be in a p2 repository (update site). We obviously can’t just point to the eclipse.org update sites for our Fedora builds since all of our dependencies must be from packages within the Fedora infrastructure that were built from sources. Initially the simplest way to achieve this was to look on the system for all OSGi bundles, publish them to a p2 repository, and then feed this directly into Tycho’s build target platform.

This approach certainly works (and that’s what the copy-platform-all script did) but it still requires performing the same steps every time one needs to feed OSGi bundles to Tycho or some other application that uses p2 repositories. The end-goal is to have tighter integration with tools like XMvn, and to make packaging an easier task. To do this we really need to have our system locations be recognized as p2 repositories so that we could also take advantage of p2 APIs.

When I got back from EclipseCon NA 2014, I started playing around with this idea of having filesystem directories being recognized as a p2 repository.

Imagine :

$ eclipse -application org.eclipse.equinox.p2.director -repository fedora:/usr/share/java/ -installIU org.swtchart

with the result being that ‘org.swtchart’ from somewhere in ‘/usr/share/java’ gets installed into the eclipse instance. This was easy to implement and fairly simple to integrate with Tycho so that we only inject a few system locations and get proper dependency resolution.

I’ve named this fedoraproject-p2 mainly because Fedora’s requirements are the driving force but I could see this being useful in many other cases.

This has been used for some time now on rawhide, and with the addition of things like the xmvn-p2-installer-plugin, it won’t be long until Eclipse plugin packaging will be drastically simpler.