rgrunber

Changing the Behaviour of Eclipse’s Update Manager

If you’ve developed plugins for the Eclipse environment, you’re moderately aware that Eclipse’s update manager can behave in strange ways from a user perspective. Things have gotten better with the p2 Remediation Support in Kepler (4.3.0) but what about dependency resolution done by Maven plugins, like Tycho, at build-time ? You get to specify a list of repositories, their content is aggregated, and if your request is satisfiable, it will be satisfied. Of course there’s some criteria p2 will attempt to optimize. For example, preferring highest version with fewest dependencies (minimize transitive closure) from a set of identically named units.

However with the increasing usage of things like software collections, I’m starting to care much more about where and how dependencies are being resolved during a build.

To understand how to even go about changing p2’s behaviour, we first need to know what is actually happening when you request to have some unit installed.  There’s a few different things happening :

  1. Installable Units are collected from the user input (units to install, and repositories provided)
  2. Installable Units are made part of a Profile Change Request
  3. Planner takes the Profile Change Request and delegates solution finding to the Projector
  4. Projector transforms request into a boolean satisfiability problem and delegates to SAT4J library
  5. If solution is found, Projector passes result back to Planner, which creates a plan (set of operations on the profile) to achieve the new state
  6. The plan is then executed on the profile by the Engine

For more information on some of the finer details of (4) I would recommend reading http://www.cril.univ-artois.fr/spip/publications/lash2010.pdf .

As a demonstration we can define an OSGi bundle org.foo.root versioned 1.0.0 which has a Require-Bundle on org.foo.bar (unversioned). We then provide org.foo.bar at various versions (eg. 1.0.0, 5.0.0, 10.0.0, 20.0.0, 50.0.0, 99.0.0). Performing the installation of org.foo.root 10 times yields the following results :

$ for i in {1..10}; do eclipse -nosplash -application org.eclipse.equinox.p2.director -repository fedora:/tmp -installIU org.foo.root -destination $(pwd)/install_$i ; done;
Installing org.foo.root 1.0.0.
Operation completed in 742 ms.
...
...
$ (for f in `find ./install_*/plugins/ -type f`; do basename $f ; done;) | sort | uniq -c
     10 org.foo.bar_99.0.0.jar
     10 org.foo.root_1.0.0.jar

So in every case that we attempted to install org.foo.root, p2 decided to satisfy the version-less requirement org.foo.bar using the latest version available (99.0.0).

Note that the p2 Director will always use the latest version of a root, so to truly test the Projector and SAT4J it is necessary to define some root and see how its dependencies are satisfied, as we did above. Also, if that “fedora:” repository scheme seemed a little strange to you, it’s because we ship a special plugin in Fedora that allows treating filesystem locations as p2 repositories , without needing all that metadata/artifact repository data on disk.

To make p2 prefer certain installable units over others, we need to hook into the component that sets the constraints for the SAT4J Solver. The Projector seems like the place for this and a quick look reveals createOptimizationFunction is probably a good starting point. We’ll want to define our own optimization (instead of StandardOptimizationFunction) function.

To favour certain locations over others, we could look at sets of units that have the same ID, and are not installed, or roots, and assign them weights based on their location. Since SAT4J will attempt to satisfy the contstraints while minimizing the objective function, we can assume that lower weighted units will be preferred over higher weighted ones. In fact, installed/root units have a weight of 1. All we need to do is subclass OptimizationFunction, override createOptimizationFunction to call its parent, and then just modify the weight for the units we care about as per the repository precedence before returning the final list of weighted units.

Once we’re done adding the necessary code, and rebuilding our modifications, we’re ready to test things out.

We now change the structure of our repository as follows :

  • /tmp/repo_priority/low_priority contains org.foo.bar versions 5.0.0, 20.0.0 and 50.0.0
  • /tmp/repo_priority/high_priority contains org.foo.bar version 10.0.0.
  • /tmp/repo_priority/ contains org.foo.bar versions 1.0.0 and 99.0.0
$ export JAVACONFDIRS='/tmp/repo_priority:/tmp/repo_priority/high_priority:/tmp/repo_priority/low_priority'
$ for i in {1..10}; do eclipse -nosplash -application org.eclipse.equinox.p2.director -repository fedora:/tmp -installIU org.foo.root -destination $(pwd)/install_$i -vmargs -Dfedora.p2.scl.order=/tmp/repo_priority/high_priority,/tmp/repo_priority,/tmp/repo_priority/low_priority ; done;
Installing org.foo.root 1.0.0.
Operation completed in 550 ms.
...
...
$ (for f in `find ./install_*/plugins/ -type f`; do basename $f ; done;) | sort | uniq -c
     10 org.foo.bar_10.0.0.jar
     10 org.foo.root_1.0.0.jar

So clearly we’ve made p2 favour units within /tmp/repo_priority/high_priority over others. This was a very basic example, and I didn’t really define what should happen when multiple units are in the same “preferred” repository but hopefully even a modification as basic as this shows the kind of things possible.

tmpfs all of the things, with anything-sync-daemon

A little while ago, I made a post on profile-sync-daemon. It’s a useful tool for reducing disk I/O with respect to a web browser profile. With that said, there’s other development tools I use as that could benefit from moving the I/O into memory. (eg. default location of local maven repository, $HOME/.m2)

There’s some pretty good documentation and upstream sources , but I don’t think this is packaged in Fedora, and haven’t found an RPM of it anywhere. Luckily it’s actually not that different from profile-sync-daemon, so I was able to use the profile-sync-daemon specfile with some very minor modifications to build it.

To set which directories should be handled by anything-sync-daemon, just modify the /etc/asd.conf file and set the WHATTOSYNC list accordingly . You can parse your config file as well much like profile-sync-daemon to ensure the configuration is correct.

From there, it’s just a matter of starting the service :

$ systemctl start asd.service

In the profile-sync-daemon post I mentioned that it might be nice to put the $HOME/.eclipse under a tmpfs, but this location has fairly predictable file access, and content is rarely written out (mainly during non-RPM/external plugin updates). As a result, this isn’t a great candidate for use with anything-sync-daemon.

One issue that came up was {profile,anything}-sync-daemon’s rsync call attempts to preserve SELinux file contexts (-X) during transfer to/from the tmpfs. SELinux policy will not permit the relabeling but luckily there’s a  nice explanation of this along with some workarounds. I went with changing /usr/bin/rsync from rsync_exec_t to bin_t and the issue seems to have gone away.

Reducing disk I/O with profile-sync-daemon

A little while ago, I finally got a new laptop! After 8 years with a Dell Inspiron 6400 (that has served admirably), I decided on a Dell XPS 13 9333. Fedora 20 worked without any major tweaking. I basically just needed nomodeset on the livecd to get the display working properly, but after a full update, even that wasn’t necessary. Did I mention the touch screen works as well ?

With 8 GB of memory and a 256 GB SSD, it’s already a pretty fast machine, but I’m still trying to optimize performance, and improve battery life, making compromises where necessary. Aside from tuning some kernel parameters, and filesystem options, I’ve come across profile-sync-daemon . It’s a nice tool specifically designed to reduce the disk I/O on a web browser’s profile by ensuring all changes are mainly done in memory (tmpfs) and written back to disk at a later time. It’s also quite easy to set up.

$ yum -y install profile-sync-daemon

Next, simply edit ‘/etc/psd.conf’ and set USERS to contain the list of users the daemon should act upon, along with the list of BROWSERS to manage.

You can even parse the configuration to confirm you’ve set it correctly :

$ profile-sync-daemon parse
Profile-sync-daemon v5.45.1 on Fedora release 20 (Heisenbug).

 Systemd service is currently inactive.
 Systemd resync service is currently inactive.

Psd will manage the following per /etc/psd.conf settings:

 browser/psname:  firefox/firefox
 owner/group id:  user/1000
 sync target:     /home/user/.mozilla/firefox/w74lprxm.default-1408456117113
 tmpfs dir:       /tmp/user-firefox-w74lprxm.default-1408456117113
 profile size:    22M

Now just make sure you’ve temporarily closed any browsers that will be affected by the daemon in order to call :

$ systemctl start psd.service
$ systemctl enable psd.service

and you’re good to go.

As it turns out profile-sync-daemon mainly exists to handle some special cases that are specific to browser profiles. However there’s also anything-sync-daemon that may be used for .. everything else. I have Eclipse open quite often so having things like the $HOME/.eclipse folder or even certain workspace metadata locations (workspace/.metadata/.mylyn) moved into tmpfs might prove useful.

Follow

Get every new post delivered to your Inbox.