Involvement Engineering

Yesterday I stumbled upon KDE Plasma, which is an attempt to bring some innovation into KDE's next major release.

So they discovered that desktops, except for minor incremental fixes, have hardly improved over several decades. This is a good start.

Sadly, it mostly already stops here. It seems they made the discovery, but have no idea what to do about it. Reading their "Vision" document, it is much like reading Microsoft announcing a new Windows version: Everything will be so much better, and more buzzword-compliant, and new possibilites, and stuff... But going down to the details, you realize that hardly anything has really changed.

OK, that is a bit unfair. There is a bit more in Plasma beyond the realization that change is necessary. There is for example some rough notion (in the form of extenders), that the desktop geometry needs to become more dynamic -- though they constrain it to a specific domin, instead of making it a more fundamental concept. There is also a rough notion of alternative sessions for different use situations -- though they create a specific solution for desktop layout, instead of considering a more fundamental concept. Yes, there are some things going in the right directin, even if in extremly small steps.

And there is this applet concept. It isn't really new; similar concepts existed in specific environments for a long time. It isn't even terribly new as applied to the desktop. Beginnings existed in the form of dockapps and similar for quite long; and Apple, though not very long, has something -- I believe -- quite similar to what they are planning in Plasma.

Nevertheless, this made me consider the topic a bit closer. So, what are the distinctive concepts that make up these applets?

Basically, we have comparatively small, independantly developed hacks, adding functionality to the application/system, without touching the core. There are numerous examples of such an approach resulting in a thriving community of extension developers: TeX/LaTeX, Emacs, Perl, Firefox and Greasemonkey come to mind.

The most fundamental property is probably the fact that those hacks -- whether called extensions, widgets, applets, whatever -- do not touch the application/system core, but are developed using some extension language/API. There is a number of implications from this: People can start writing improvements without knowing the language of the core (usually more low-level than the extension language), or the core code -- just build on some simple APIs. People can do their improvements independantly, without discussing with the core developers, according to an own release schedule etc. They can distribute them independantly. Problems are less critical, because they do not touch the core, and because they affect only those people who explicitely chose to use the extension. For the same reason obscure features can be implemented without bloating the core. And so on. While some of the advantages may be actually organisational or even psychological rather than technical, all of them contribute to lowering entry barriers for people to contribute improvements, for getting actively involved.

The good news is: The Hurd lends itself to such a concept. With a hurdish application infrastructure constructed of small components building upon each other, the whole system actually could be considered the applet idea taken to extreme. (Which is one possible explanation why I like it...) All services being exported via filesystem interfaces, we also always have very accessible extension APIs to easily plug applets into. And with the concept being consequently implemented throughout the system, it's much more powerful than any solution limited to the topmost layer like in Plasma or any other specific approaches possible on traditional systems.

Now the only thing we really need to be careful about, is to ensure Hurd components are really easy to create, distribute, and install...

Persistance vs. Insistance

Session management is a thread of thought I only recently took up in my considerations. Nevertheless, it touches on many other aspects -- it looks like this has been the missing link in various contexts. But more on this later.

Session management basically means: I don't want to manually restore my work environment on each bootup; I want it to come up just as I left it -- usually at least.

There is little doubt working session management is something users really really want to have. I say working, because retrofitted concepts like X/KDE/GNOME/whatever session management (which is the most explicit most users know), tend to be incompletely implemented and thus useless.

In a UNIX environment, most of the system doesn't know session management. To change the startup behaviour of something, you usually need to explicitely change settings, either by editing config files; or in GUI applications, usually there are a number of startup-related settings in the options dialog.

Nevertheless, besides the not-working X session management, you'll find many places where session management aspects have sneaked in, if you look closer. A typical example would be the soundcard mixer init scripts, which save the current mixer settings on shutdown and restore them on next startup. Some applications, like Opera or Vim, have options to explicitely and/or automatically save/restore their sessions. The shell (and some other command line based applications) save the command line history. In the Hurd, we have passive translators, as kind of session management at the filesystem tree level.

And then there is screen, which implements kind of a poor man's universal session management: When you detach a screen session from the terminal, the software just runs on in the background -- instead of doing real shutdown/startup, you just detach and reattach your session. (Screen sessions are usually used on servers, i.e. systems that run permanently with rare interruptions.) Those who know this feature usually love it -- by this little "trick" it implements something resembling session management that really and always works, because the applications needn't be aware of it at all...

Another, less obscure example of transparent and thus (more or less) reliably working and widely established session managment-like behaviour, is suspend to disk. Just puts the whole memory image to disk, and restores the exact same situation on resume. (The exception are most hardware drivers, which obviously need to be aware of the suspend. This was actually the main starting point for my considerations -- the suspend infrastructure of the hardware driver framework could and should be extended to the application level...)

A very similar approach -- which could be considered an extension of suspend, although it has a completely different origin -- is persistency like in EROS. The difference is that here the image is saved not only on suspend, but periodically (every 5 minutes), so the system will come up in the last state even on a power outage or so.

One side effect of system-wide persistence (again, with the exception of system core and hardware drivers), is that you need no sophisticated system boot and program startup mechanisms -- you just add objects to the system once, and they live on forever. (Unless you decide eternity is a bit too long, and get rid of them earlier...) Which is the very reason why EROS has it: Making the whole system persistent seemed easier than creating a method for secure explicit storage/retrival of capabilities. But they claim it's desirable for usability purposes also...

However, there are downsides to this. Completely transparent session management works fairly well in case of specific things like screen, because of the scope of a screen session usually being quite limited; and suspend, because of it typically being used only for fairly short breaks in work that means to be continued exactly where you left off. But a completely and always persistent system, creates various problems.

For one, to update software, you basically need to create a new object in place of the old one. If you want to preserve state information in the process, you need to implement this explicitely: Either run the new version in parallel and pass the state from the old one to the new (session handoff), or dump the old state to a third party temporarily and read it when starting the new version (session saving). In both cases, you are effectively implementing explicit session management -- all of the nice transparency is gone.

This actually doesn't only happen on updates: If you want to move some state information, e.g. when replacing hardware or working on a different machine or whatever, you need the very same protocols -- only that the actual state transfer gets more complicated.

Another, related problem is flexibility in general. Transparent persistance is a sledge hammer approach: You always get the whole thing. But in many situations, you do not actually want to, or can't, restore the system in the exact same state. Maybe your system configuration changed meanwhile, so some stuff won't work or doesn't make sense anymore. Maybe you screwed up and explicitely want to easily get rid of parts of the old state. Maybe you want an easy way to active only parts of your session at times. Maybe you want to carry along part of your environment to different machines. And so on. You might even have your system on a USB stick and need it to adapt to a different machine each time you boot it!

So what we really want is a flexible system of subsessions, that can be restored or not upon demand and/or resource availability. One session for the core system, various sessions for individual hardware components, a number of sessions for background services, some sessions for higher-level system components traditionally handled by runlevels (networking, windowing environment), and lots of sessions for individual parts of your application envirionment -- a music player session, a news reading session, a communication session, various sessions for projects you are working on at times, etc. You get a whole tree of subsessions and sub-subsessions building upon each other. (Those subsessions are also related to various other mechanisms like resource management; security; and forming an application infrastructure from generic components, as touched in my post on a hurdish X implementation -- that's the "missing link" aspect mentioned at the start. I'll handle those in other posts.)

So between disfunctional retrofitted session management, and sledge hammer total persistence, we really want a session management approach that is not quite transparent, but fully integrated and consistently implemented throughout the system.

Implications from that requirements, as well as ideas how it could be implemented, I'll leave out for now, as this is already getting quite lengthy...

Welcome to HELL

An idea I'm contemplating for a while now: Hurd Emulation Layer for Linux.

The idea is to create an emulation environment, which would allow running hurdish applications on top of a familar GNU/Linux system. Now this is a bit unsubstational of course, not having explained so far what I mean by hurdish applications, except for the mention how I imagine a hurdish X implementation.

Well, doubtless this will be motif in many coming entries. So just for starters: The fundamental concept is to replace bloated monolithic all-in-one applications by many small translators providing specific services, and some mechanisms to combine them into systems providing the desired overall functionality. Ideas how this would work in particular, as well as why I believe that on the Hurd -- unlike on traditional UNIX systems -- this actually *can* work, I will also cover in other posts.

Well, having said it is not possible on traditional UNIX systems *natively*, I do think it should be possible to create a limited, somewhat Hurd-like environment on top of such systems -- far from perfect, but just enough to run hurdish applications with no or only little changes. Some mechanism to emulate translators, something to map Hurd free form RPCs on UNIX IPC mechanisms (terribly inefficient but probably possible), something to emulate some of the Hurd extensions of the traditional UNIX interfaces. Of course it would break as soon as programs try to dig too low, using specific functionality of Hurd core components etc. But for most higher-level applications, it can probably work.

As for actual implementation, I only have a number of rough ideas. We might be able to emulate translator functionality by playing with FUSE. To run hurdish programs, we could link them to a special modified variant of the Hurd libc, emulating the functionality on top of Linux instead of contacting Hurd servers; or we could employ some proper sandboxing, intercepting system calls etc. For some commonly used Hurd services, we might even try to run hacked Hurd servers, or replacements providing the functionality in the foreign environment. Non-hurdish processes could be provided with some hooks via LD_PRELOAD to better cope with the emulated Hurd features when interfacing to hurdish processes.

One fundamental question is whether to wrap every single hurdish process in its own HELL, mapping the Hurd functionality to UNIX mechanisms such that the individual processes can be combined in a Hurdish manner; or to provide one single big HELL environment (kind of a userspace Hurd implementation), interfacing to the non-hurdish world only at the outer borders. The first one may be more desirable, as it integrates better into the system. However, it probably means more overhead; could cause some troubles due to non-perfect mapping of Hurd features; might be harder to use; and most likely is harder to implement.

Of course, there are also strategical considerations involved -- do we really want to have HELL? This is a similar uncertain question to whether porting free software to proprietary systems like Windows or Mac OS should be encouraged. (Though not politcal in the HELL case.) On one hand, it encourages authors to write hurdish applications, as with HELL the audience isn't limited to actual Hurd users; people who for various reasons can't or don't want to do the switch, can still use these programs. (I'm considering making netrik a native Hurd application. However, I don't want to exclude GNU/Linux users alltogether. That's how I originally came up with HELL.)

Also, discovering some of the advantages of the Hurd approach while using HELL, people might be encouraged to switch to the Real Thing (TM), where these advantages come into their own, the hurdish concepts being used consequently throughout the whole system, not only in a limited environment.

OTOH, people being able to use hurdish applications and reap some of the advantages on a traditional system, might actually get lower inclination to switch; they might get stuck with a suboptimal system forever.

I tend to believe HELL would act more as an enabler for Hurd than a stopper -- YMMV.