Ken's Digital Den

Category: foss

Java 7 (OpenJDK/IcedTea) for the BUG 2.0

NOTE: this post was originally at http://community.buglabs.net/kgilmer/posts/251-Java-7-OpenJDK-IcedTea-for-the-BUG-2-, however the webserver seems only intermittently available so I’m moving the post here.

For anyone interested in Java 7, I have built OpenJDK 7 (via IcedTea) for BUG 2.0 (Angstrom, armv7, Zero VM).  The binaries are available here.  The binaries are not provided as Angstrom opkgs, so the installation process is manual. This is because it was built natively on a BUG, rather than using the OpenEmbedded cross compiling build system.  You’ll also need to install libcups2 via opkg if you want to replace the Java 6 JRE to run BUG applications.  Here is my installation steps, assuming the j2re-image tarball is decompressed in /home/root:


$ mkdir -p /usr/lib/jvm/java-7-openjdk/jre
$ mv ~/j2re-image/* /usr/lib/jvm/java-7-openjdk/jre
$ rm /usr/bin/java
$ ln -s /usr/lib/jvm/java-7-openjdk/jre/bin/java /usr/bin/java
$ opkg install libcups2

Now if you restart Felix (via Knapsack if you are running a 3.0 release), you’ll notice that the JVM running is java.runtime.version = 1.7.0_147-icedtea-b147 via http://bug/support.html.  This is provided on an experimental basis and I would not be surprised if problems pop up!

An interesting benefit that IcedTea 7 provides (other than ARM compatibility via Zero) is alternative VM implementations CACAO and JamVM.  The binary build I have provided unfortunately does not include these but I’m going to try and build them.  Hopefully I can get Zero, Shark, CACAO, and JamVM all happily living inside the BUG!  :)

Special thanks to xranby@irc.oftc.net#openjdk for his invaluable assistance!

Fresh from the oven: Knapsack for Apache Felix

Both for my various weekend projects and on the BUG, I’ve been unhappy with the bundled launcher that comes with Felix.  Equinox is no better.  Typically people use the Eclipse tooling to create and configure framework instances, but when setting up for production or working with embedded targets this is often not possible. So I set out to make something that would allow me to easily get a framework up and running from the terminal.  Luckily, Felix has excellent embedding support and documentation so it was pretty easy to get started.

What is it?  Essentially an alternative launcher for the Felix OSGi framework (3.2.2) that uses the OS’s native shell for framework management, and creates all of the configuration files automatically, letting the user just download, run, and start adding their application bundles, with interactive feedback.

I’ve much more detail on the design details and usage on the project page and in the readme.

However, here is the summary of how to go from Zero to OSGi in 30 seconds*:

$ mkdir foo && cd foo
$ wget https://leafcutter.ci.cloudbees.com/job/knapsack/lastSuccessfulBuild/artifact/knapsack.jar
$ java -jar knapsack.jar &

* Depending, of course, on your copy-paste skillz.

Release 0.1 for Sprinkles

After several months of actively developing with Sprinkles I think it’s time to make a tag and call it a version.  I’ve still got far to go in terms of documentation but with Cloudbees free Jenkins instance I have a build running with JUnit and Javadocs available.  I’ve also created a recipe for OpenEmbedded so embedded devices can also taste the functional fruits in Java.  Oh, and Sprinkles exports its package just like a good OSGi bundle should!

Sprinkles: The functional fruits must be tasted!

Some problems are a natural fit for functional style programming.  By this I mean applying functions, often recursively, to lists of data.  Some languages support this out of the box, in a compact elegant fashion, such as Python.  Frustrated with the OOP approach to a problem involving files, I began to write some static helper methods to try to restructure my program in functional way.  The result, after two weeks of thoroughly enjoyable exploration of recursion and self-reference, is Sprinkles.  Sprinkles consists of a “function” interface and a set of static methods that allow the functions, such as map() and fold(), to be applied to elements and collections.  The base function interface is simply:


public interface Function {
    public Object apply(Object element);
}

I know I know, how could anything useful come from such a simple interface…?  Calling the Fn.map(Function, Object) static method will cause every element contained in Object (assuming what’s passed implements Iterable) to be passed to Function, with the result being put in a new Collection as the return value.  When this pattern is applied multiple times, I find that certain problems that would require a lot of code in a traditional OOP approach are quite clean and concise.

To get a sense of the value of Sprinkles, let’s look at some examples.  In these examples I use annonymous inner classes for brevity.  But in real code, I typically define my functions explicitly, as I’ve found I want to reuse them.

Let’s say I want to “walk” the filesystem for all the files from a given directory:

Collection results =
    Fn.map(ReturnFilesFunction.GET_FILES_FN, new File("/tmp"));

See how easy that was? results now contains every file that is a child of /tmp. Ok, I admit this is cheating, but bear with me. I wrote some static functions for common activites like this.
Well, now let’s say I want to figure how the total size of all those files.  Well, the results of my first function is a list of all files.  What if apply another function that returns the sizes of those files:

Collection results = Fn.map(
    new Function(){
          public Object apply(Object element) {
                  return ((File) element).length();
          }
    }, Fn.map(ReturnFilesFunction.GET_FILES_FN, new File("/tmp")));

Now, results will contain a list of the sizes of all the files.  Thankfully, Java autoboxes the primitive types for me since I’m using Java 6.

To get my final result, I simply need to “collapse” the list of sizes into a sum using the fold() function:

</span>

<span style="font-family: Consolas, Monaco, 'Courier New', Courier, monospace; font-size: 12px; line-height: 18px; white-space: pre;">Object result = Fn.fold(new FoldFunction() {</span>
<pre>    public Object apply(Object element, Object result) {
        Long ival = (Long) element;
        Long rval = (Long) result;
        if (rval == null)
               return ival;
        return ival + rval;
    }
    }, Fn.map(new Function() {
            public Object apply(Object element) {
                    return ((File) element).length();
            }
    }, Fn.map(ReturnFilesFunction.GET_FILES_FN, new File("/tmp"))));
System.out.println("Total size: " + result);

You may be thinking, well that’s really hard to read, and I admit it’s true.  There are a few things to make it better.  First, as I mentioned, do not use the anonymous inner classes:

Object result = Fn.fold(
    new AddLongElements(), Fn.map(
        new FileToSize(), Fn.map(
                ReturnFilesFunction.GET_FILES_FN, new File("/tmp"))));

The other trick is using the version of Map() that takes in a Collection<Function>. This let’s the function chain be composed in a linear, non-nested way:

Collection<Fn.Function> functions = new ArrayList<Fn.Function>();
functions.add(ReturnFilesFunction.GET_FILES_FN);
functions.add(new FileToSize());
Object result = Fn.fold(new AddLongElements(),
              Fn.map(functions, new File("/tmp")));

Albiet, in our file example this approach doesn’t help much, since we are only applying a few functions.  But as the chain grows, this code style quickly ends up being easier to read.

One of the interesting twists of Sprinkles is that a Function can return a Collection (list), and that list is then acted upon as such.  When a Function does not want to return a value, it simply returns null and it will not be appended to the list.  However, where ever Collections are used in the API, empty Collections are passed/returned instead of null, making the code much cleaner, since there is no need for null checks everywhere.  And finally, anything can be passed into map().  If what’s passed in is iterable, then the function is applied as a list, otherwise the function is applied as if it’s a list of single element.  This has great impact when composing sets of recursive functions that can expand and contract based on the results of a function.

Drawbacks?  Well, in general this approach works well for certain kinds of problems.  Also, the Object-in-Object-out approach essentially boils down to dynamic typing, since only at runtime is it know what types will be passed and returned.  However since types can be constrained from the entry point of a function, in practice I don’t have much problem with this.  Function reuse is somewhat limited in that a function has to be explicitly written to handle all static types that may be passed to it.  And finally, there are no generics in the API.  I simply don’t understand generics as implemented in the Collections framework well enough yet.

There are more techniques to cover later, such as using find() and depth-first function traversal.  With Sprinkles, I can finally taste the forbidden fruits of functional programming in Java.  It’s amazing to find how easy functions can be composed and reused!

Thoughts on OSGiFS

I’ve spent some time working with the prototype implementation for a filesystem-based management interface to OSGi.  Some points that have become apparent:

  • It should probably be called something like “filesystem admin” or “native admin” to be more inline with the OSGi naming style.
  • I’m unable to find a way around the ‘one thread per file’ problem.  I was hoping there was some magical NIO thing, but if there is I can’t find it.  Due to this, I’m considering either using FUSE (which would limit it to Linux only) or consolidating what files are available in the interface to as few as possible.  The former has the possibility of a much richer user experience, but requires JNI integration with FUSE.  I have had little luck with with existing libraries for this on Linux.  The latter is less work but could end up making the whole thing more of a pain than it’s worth.
  • The logic of having directories for run levels, and symlinks in them that point to an install directory is a mixed bag.  It’s nice to have one directory where all the bundles go, and the runlevel convention is pretty simple, but having the management agent (osgifs) delete the symlinks when a bundle fails to start is not a good approach.  In general, osgifs should never delete files, it just causes user confusion.  So, there needs to be another way (via the filesystem metaphor) of defining runlevels for bundles, providing a clear view of state and handling cases of run level changes and bundle start failure.
  • The felix auto install feature is quite handy.  It’s nice to be able to just scp the jar to a remote, machine, run it, and have an OSGi framework ready to go.
  • The shell integration becomes quite natural.  And when combined with screen, tee, etc. it is a very nice way of running and managing remote OSGi framework instances.  Also, running a framework in the background of a running shell great for debugging, in that any log or stdout output from the framework is printed in the shell.  This gives you immediate feedback if something happens.
  • I should look into integrating with Karaf, but it seems like the first order of business would be dealing with the Maven dependencies, which is something I’ve never quite had the time to learn.

So, in summary I’d say the prototype is a “success” to the degree that I’m going to continue to use it, but it needs a lot of work before I’d consider it relevant and valuable in a larger scope.

Some notes on XMPP, pubsub, Smackx, and ejabberd.

  • Smack and smackx employ a custom classloading scheme to load extensions at runtime.  This mechanism looks for a configuration file in a specific place in a jar.  If you are running from source (in an IDE) or do not have the file (from a custom build), the classloading scheme will fail.  Rather than an error like “Can’t load extensions”, you’ll simply find intermittent ClassCastExceptions being thrown.  Most if not all of the test suites will fail as well.  Running from a binary jar or from a build utilizing the included build scripts is recommended.
  • For PubSub to work with ejabberd, when creating the PubSubManager, you must specify the toAddress in the PubSubManager() constructor.  If you do not do this subscriptions will fail.  The toAddress that worked for me was “pubsub.” + connection.getHost().
  • Psi is a great XMPP chat client for Linux that lets you see the raw XML.  Smack also has a debug mode triggered by a system property that will create a Swing XMPP message log view.
  • Do not expect to get blown away by the detail and verbosity of error messages from ejabberd.

A shell-less OSGi shell

Today from my tweetstream I read from @njbartlett about a new OSGi enterprise initiative.  My interest heightened when I saw a line in the press release about a bash-like shell called Posh.  Very clever name!  I took a peek at the Paremus documentation on the shell.  Very full featured and comes the closest yet I think to a real shell environment for OSGi.  This seems like a good time to talk about my experiements with a shell-less OSGi shell.

Early on at Bug when I started to get serious about OSGi, I found that using the shells included with Equinox and Concierge helped a great deal in understanding runtime dynamics and structure.  The shell is invaluable in how it can shed light into how the system is running, how services and packages are associated, and of course being able to change the state and debug dependency issues.  I ended up writing my own shell just so I could Have It My Way, which was also a good learning exercise.  But, there are problems in general with OSGi shell systems no matter good they get.  They are isolated from the operating system; walled gardens.  Additionally while they often support a subset of real shell commands and constructs, they are cut off from a real shell session, making utility limited.  Conversely, the more full featured a shell environment is, the more frustrating it is that it’s not the native shell. And of course each shell environment has it’s own commands, behaviors, and quirks.  So, on BUG I wanted to get rid of the OSGi shell, and trade it for something more natural for a Linux environment.  I like the “everything is a file” philosophy and have seen it put to great and simple use in Linux with sysfs.  I needed to come up with a system that does the three things that OSGi shells do:

  • Inspect the state of a running OSGi system.
  • Install/start/stop/remove bundles from running system.
  • Make system changes (runlevel, shutdown, etc.)

Using the Linux init system as found on my Ubuntu Linux dev machine as inspiration, I created a mock-up filesystem layout like this:

/installed
/runlevel-1 (...n)
/status
/control

The idea being that “/installed” would be like /etc/init.d, holding all the programs or OSGi bundles that would be installed in this particular OSGi instance.  And then, similiarly to /etc/rc3.d, /runlevel-x would have symlinks to bundles living in /installed that should be set to run when the framework runlevel is set appropriately.  To stop a bundle I simply remove the symlink from the runlevel directory. Also, unlike the rc init system, the filesystem would be periodically scanned for changes, so if a new bundle goes into installed, the running OSGi instance is just updated.

Next, with inspiration from the Linux sysfs filesystem, using out-only pipes, I created files (such as /status/services) that would return OSGi state information to the caller.  So from my (real) shell I could say “cat /status/bundle/23/headers” and it would return me the MANIFEST.MF contents, essentially, of bundle ID 23.  This worked pretty well and I was able to quickly make use of this, grepping for services, packages, installing things in specific places.  All of this being done from a native shell environment, using my native tools that I’m familiar with and having access to things like scp, cron jobs, etc..

For a control mechanism, I created some write-only  pipes that would let me make state changes to the running system.  For example to “bounce” a bundle: “echo 1 > /control/bundle/12/bounce” would restart bundle 23.  Shutting the framework down is simply: “echo 1> /control/shutdown”.

Lastly taking cues from bnd, I made the bundle have multiple entry points, creating a simple OSGi installer for when the bundle is called from a Java context.  This let me quickly setup a new framework layout quickly.  For now it only works for Felix.

So, with this implemented I can now remove my OSGi shell and treat it just like another subsystem.  The prototype implementation has some problems.  It lacks an extension mechanism to allow other bundles to contribute files and behavior.  The pipe mechanism requires a thread per-file.  And there are times when the symlink behavior is awkward, like when a bundle fails to start (current policy: delete the symlink).   And of course if you’re using Windows your probably out of luck.

The prototype implementation is named “osgifs” and source is available at svn://bugcamp.net/bug/trunk/com.buglabs.osgi.filesystem and the binary is at http://bugcommunity.com/downloads/files/osgifs.jar .  To get started simply copy the jar to an empty directory, run “java -jar osgifs.jar”.  This will download felix and setup the initial directories.  Run the start script and from that point osgifs will represent the runtime system somewhat like a filesystem.

What’s next?  For efficiency it may be better to rely on INotify instead of filesystem polling, and eventually switch to FUSE and make the entire thing virtual.  Other ideas include a nested-framework mechanism and transparently OSGi-ifing dumb jars.  At this point though I’m mainly curious if anyone has written something like this before, and if there is any other feedback?

Follow

Get every new post delivered to your Inbox.