Tag: Fedora

Improving Eclipse Platform Stability On Rawhide

The Eclipse platform on Fedora Rawhide can be pretty unstable at times. Every update to one of its dependencies requires a rebuild. As a result, it has been on our TODO list for a while to work out some way of making Eclipse more resilient to these kind of dependency updates (at least in cases where a rebuild shouldn’t be required). Looking upstream, there are quite a few bugs relating to this topic (410710, 410785, 408138) .

For simple rebuilds where project metadata likely hasn’t changed there’s fix for symbolic links in place.

Another common breakage happens when dependencies that are listed as plugins within a feature get updated.

Unlike regular Eclipse plugins that might contribute certain capabilities, a feature represents a set of plugins. One can think of them as RPM meta-packages and some features are used especially by end-users to install a larger set of plugins (eg. org.eclipse.cdt.feature.group is “C/C++ Development Tools”). A feature lists the set of plugins it provides with the <plugin> tag and may also specify dependencies with the <requires> tag. The difference is that the <plugin> tag locks onto the exact version discovered at build time, and may only resolve against that exact version. The <requires> tag on the other hand allows you some flexibility in terms of dependencies with ranges and some compatibility levels.

Sometimes one might see features listing dependencies as content. Does the JDT provide, or own org.junit and org.hamcrest ? Do we really mean to say that changing the versions of those plugins implies a completely different feature ? Clearly this isn’t the case, and it would make much more sense to use <requires> but the former practice seems quite common.

I don’t think this is done out of lack of understanding, but because projects also want to include their dependencies into the repositories they deploy. The <plugin> definition accomplishes this. Using <requires> one would also need to list the plugins to be included in the repository definition (site.xml/category.xml) and some platform projects have to jump through some additional hoops to change this file so it’s not a huge surprise that many go for the simpler option. Sadly, this causes some problems for us in Fedora land.

Luckily there’s now fix for platform features that should help reduce the number of rebuilds.

Changing the Behaviour of Eclipse’s Update Manager

If you’ve developed plugins for the Eclipse environment, you’re moderately aware that Eclipse’s update manager can behave in strange ways from a user perspective. Things have gotten better with the p2 Remediation Support in Kepler (4.3.0) but what about dependency resolution done by Maven plugins, like Tycho, at build-time ? You get to specify a list of repositories, their content is aggregated, and if your request is satisfiable, it will be satisfied. Of course there’s some criteria p2 will attempt to optimize. For example, preferring highest version with fewest dependencies (minimize transitive closure) from a set of identically named units.

However with the increasing usage of things like software collections, I’m starting to care much more about where and how dependencies are being resolved during a build.

To understand how to even go about changing p2’s behaviour, we first need to know what is actually happening when you request to have some unit installed.  There’s a few different things happening :

  1. Installable Units are collected from the user input (units to install, and repositories provided)
  2. Installable Units are made part of a Profile Change Request
  3. Planner takes the Profile Change Request and delegates solution finding to the Projector
  4. Projector transforms request into a boolean satisfiability problem and delegates to SAT4J library
  5. If solution is found, Projector passes result back to Planner, which creates a plan (set of operations on the profile) to achieve the new state
  6. The plan is then executed on the profile by the Engine

For more information on some of the finer details of (4) I would recommend reading http://www.cril.univ-artois.fr/spip/publications/lash2010.pdf .

As a demonstration we can define an OSGi bundle org.foo.root versioned 1.0.0 which has a Require-Bundle on org.foo.bar (unversioned). We then provide org.foo.bar at various versions (eg. 1.0.0, 5.0.0, 10.0.0, 20.0.0, 50.0.0, 99.0.0). Performing the installation of org.foo.root 10 times yields the following results :

$ for i in {1..10}; do eclipse -nosplash -application org.eclipse.equinox.p2.director -repository fedora:/tmp -installIU org.foo.root -destination $(pwd)/install_$i ; done;
Installing org.foo.root 1.0.0.
Operation completed in 742 ms.
$ (for f in `find ./install_*/plugins/ -type f`; do basename $f ; done;) | sort | uniq -c
     10 org.foo.bar_99.0.0.jar
     10 org.foo.root_1.0.0.jar

So in every case that we attempted to install org.foo.root, p2 decided to satisfy the version-less requirement org.foo.bar using the latest version available (99.0.0).

Note that the p2 Director will always use the latest version of a root, so to truly test the Projector and SAT4J it is necessary to define some root and see how its dependencies are satisfied, as we did above. Also, if that “fedora:” repository scheme seemed a little strange to you, it’s because we ship a special plugin in Fedora that allows treating filesystem locations as p2 repositories , without needing all that metadata/artifact repository data on disk.

To make p2 prefer certain installable units over others, we need to hook into the component that sets the constraints for the SAT4J Solver. The Projector seems like the place for this and a quick look reveals createOptimizationFunction is probably a good starting point. We’ll want to define our own optimization (instead of OptimizationFunction) function.

To favour certain locations over others, we could look at sets of units that have the same ID, and are not installed, or roots, and assign them weights based on their location. Since SAT4J will attempt to satisfy the contstraints while minimizing the objective function, we can assume that lower weighted units will be preferred over higher weighted ones. In fact, installed/root units have a weight of 1. All we need to do is subclass OptimizationFunction, override createOptimizationFunction to call its parent, and then just modify the weight for the units we care about as per the repository precedence before returning the final list of weighted units.

Once we’re done adding the necessary code, and rebuilding our modifications, we’re ready to test things out.

We now change the structure of our repository as follows :

  • /tmp/repo_priority/low_priority contains org.foo.bar versions 5.0.0, 20.0.0 and 50.0.0
  • /tmp/repo_priority/high_priority contains org.foo.bar version 10.0.0.
  • /tmp/repo_priority/ contains org.foo.bar versions 1.0.0 and 99.0.0
$ export JAVACONFDIRS='/tmp/repo_priority:/tmp/repo_priority/high_priority:/tmp/repo_priority/low_priority'
$ for i in {1..10}; do eclipse -nosplash -application org.eclipse.equinox.p2.director -repository fedora:/tmp -installIU org.foo.root -destination $(pwd)/install_$i -vmargs -Dfedora.p2.scl.order=/tmp/repo_priority/high_priority,/tmp/repo_priority,/tmp/repo_priority/low_priority ; done;
Installing org.foo.root 1.0.0.
Operation completed in 550 ms.
$ (for f in `find ./install_*/plugins/ -type f`; do basename $f ; done;) | sort | uniq -c
     10 org.foo.bar_10.0.0.jar
     10 org.foo.root_1.0.0.jar

So clearly we’ve made p2 favour units within /tmp/repo_priority/high_priority over others. This was a very basic example, and I didn’t really define what should happen when multiple units are in the same “preferred” repository but hopefully even a modification as basic as this shows the kind of things possible.

tmpfs all of the things, with anything-sync-daemon

A little while ago, I made a post on profile-sync-daemon. It’s a useful tool for reducing disk I/O with respect to a web browser profile. With that said, there’s other development tools I use as that could benefit from moving the I/O into memory. (eg. default location of local maven repository, $HOME/.m2)

There’s some pretty good documentation and upstream sources , but I don’t think this is packaged in Fedora, and haven’t found an RPM of it anywhere. Luckily it’s actually not that different from profile-sync-daemon, so I was able to use the profile-sync-daemon specfile with some very minor modifications to build it.

To set which directories should be handled by anything-sync-daemon, just modify the /etc/asd.conf file and set the WHATTOSYNC list accordingly . You can parse your config file as well much like profile-sync-daemon to ensure the configuration is correct.

From there, it’s just a matter of starting the service :

$ systemctl start asd.service

In the profile-sync-daemon post I mentioned that it might be nice to put the $HOME/.eclipse under a tmpfs, but this location has fairly predictable file access, and content is rarely written out (mainly during non-RPM/external plugin updates). As a result, this isn’t a great candidate for use with anything-sync-daemon.

One issue that came up was {profile,anything}-sync-daemon’s rsync call attempts to preserve SELinux file contexts (-X) during transfer to/from the tmpfs. SELinux policy will not permit the relabeling but luckily there’s a  nice explanation of this along with some workarounds. I went with changing /usr/bin/rsync from rsync_exec_t to bin_t and the issue seems to have gone away.