Theory of Filesystem Relativity

It has been pointed out that the Hurd chroot implementation has serious problems in connection with passive translators, resulting in unexpected behaviour and gaping security holes: When someone sets a passive translator (e.g. a firmlink) within a chroot, and then accesses the translated node from within the chroot, the translator will run *outside* the chroot, but will be accessible from inside it -- meaning you can easily escape the chroot. (Using something like settrans -p tunnel /hurd/firmlink / )

This is a serious flaw in how passive translators work; and it has been used to demonstrate that supposedly passive translators are broken by design, and should be replaced by transparent system-wide persistence. However, I doubt such a radical conclusion is really appropriate.

For one, the original Hurd designers said at the setout that chroot will only be supported for compatibility if it can be done without too much hassle. One could very well claim that a secure and perfectly consistent chroot implementation was never intended. Yet, I do think that chroot can be fixed without totally overthrowing the passive translator concept.

But before diving into this, I'd like to mention that IMHO the current chroot implementation, using a system call handled in the actual filesystem servers, is wrong. It seems much more hurdish, more flexible, and more robust, to use a filesystem proxy as / for the chrooted process. One advantage is that the same mechanism can be used not only for chroot -- which uses a proxy that simply translates all paths so that they point to a subtree instead of the global / -- but also with other kinds of proxies to achieve different semantics: For example using a proxy that mirrors the global /, and only replaces a few specific locations. (/servers/* is a likely candidate, which allows replacing default system servers.)

Now back to passive translators. I can think of quite a lot of possible approaches:

• Simply don't allow setting passive translators inside a chroot at all. After all, chroot is only for UNIX compatibility, and translators are not a UNIX concept...
• Allow setting passive translators, but only temporarily, not storing them in the underlying filesystem. When accessing the translated node, the translator is started by the chroot. Allowing passive translators but not really storing them is a bit unelegant, of course...
• Store the passive translator, but also store the chroot information; and only start the translator if the node is accessed from within the same chroot.
• Store the passive translator and the chroot, and whenever the node is accessed, run the translator in a matching chroot. This might be the most elegant solution. Only problem I see is that the translator is run in an identical, but not the *same* context. For chroot this shouldn't be a problem I believe; but some other kinds of chroot-like subenvironments might break: If you have some kind of subenvironment, where some things are local to the specific instance, running the translator in a different instance might not do the trick. But as I said, for a normal chroot it should be fine.
• Last but not least, we could simply allow setting passive translators from within a chroot normally like it happens now, but when a translated node is accessed, the translator started would run in the context of the process accessing it -- which is different for a chrooted process than for a normal one. (For consistency, any active translators running outside a chroot would have to be ignored inside it...)

One could claim that the last variant is actually the only sane one: It's a bit confusing that the translators will refer to something else within the chroot than outside it -- but in most situations that probably is actually the most useful behaviour. Also, it's how symlinks behave in a chroot.

Of course, in some cases you actually want the other behaviour. There is really no solution that always does the desired thing. A similar problem arises with translators or symlinks on NFS: Should they be resolved on the client or the server side? Sometimes it's desirable for links or translators always to refer to the same physical location on the server, no matter whether the FS is accessed through NFS or directly; while sometimes it's more desirable to always refer to the same logical location, so you always get an appropriate local resoure on the machine where the program runs...

Another situation, which also isn't specific to chroots or even to translators (I experienced the problem with symlinks on Linux), is handling of mount points: Should translators and symlinks refer to the root of the actual file system (partition), rather than the VFS root, so they always point to the same physical location no matter where the FS is mounted?...

It seems that in most other cases involving symlinks or translators, the rule is to have them referring to the same *logical* location in changing contexts; so, while I'm not sure whether it's the more useful behaviour, it at least would be most consistent to go with the last suggestion, i.e. make passive translators within a chroot always run in the context of the chroot.

External Insistence

In an earlier post, I explained my concerns with transparent system-wide persistence. One of the problems I pointed out, is that in such a system, you have to manually serialize all important state on upgrades and in some other situations anyways, relativating the value of transparent persistence.

Marcus Brinkmann now showed me a nice text explaining the upgrade problem in great detail. It's a good read, at least the first half.

BTW, meanwhile I refined some of the ideas I tried to explain in the original discussion, and I might post an upgrade at some point; but all in all, my concerns with transparent system-wide persistence haven't changed.

MehtaHurd

Recently, I saw a presentation on DeepaMehta. Sounds quite promising: Completely new approach instead of the traditional (broken) desktop metaphor; adapted to the way we think... Of course, I was mighty sceptical. It turned out more interesting than I expected.

Putting it bluntly, DeepaMehta is kind of an extensible filesystem browser with an advanced bookmarking/navigation system.

Of course, this statement is a misrepresentation in several regards. Most notably, DeepaMehta's objects aren't really meant as simple bookmarks. They can store considerable amounts of metadata, as well as actual data, and only optionally can reference traditional files or other external data, but normally they exist on their own -- in fact, the idea is that ideally (almost?) all data is stored in DeepaMehta itself. Also, DeepaMehta doesn't care at all about the traditional filesystem structure... Well, let's get more specific.

DeepaMehta is based on three core observations regarding existing desktop systems. One is that overlapping windows are an abomination that needs to be avoided. This is a very important issue of course. DeepaMehta always displays only one navigation window, and one window presenting the content of the currently active object -- just like Norton Commander in quick view mode... (Somewhat resembling Oberon, which also has a content pane plus another pane for other stuff -- though the division is different there.)

Limiting to always just one content window is a bit extreme IMHO: Seeing several things at the same time can be useful sometimes, given a sane (tiled and dynamic) window manager. Nevertheless, it's a Good Thing (TM) that DeepaMehta addresses the problem of overlapping windows.

The second observation is that traditional applications make no sense. Instead, we need a generic shell application, with plugins to be able to perform specific actions on various object types -- again a very important issue. To some extent, we see this happening with things like Nautilus already; but DeepaMehta is much more consequential: All navigation actions are performed in the navigation window, using standard mechanisms supported by plugins for creating specific object associations; and all object viewing/editing actions are performed in the content window, again supported by plugins for specific object types.

The third observation is that traditional navigation mechanisms are quite unsuitable to the way we work; for many tasks, a new approach can be vastly more efficient and convenient. This is the strong side of DeepaMehta, and the really innovative part: It uses a navigation system based on mind maps. Basically, you have all your objects, as well as various types of bilateral associations between them.

As there are far too many objects and associotions to make this manageable directly, you never look at all of them at once, but instead use various partial views (maps). A new view can be created by starting from some object, uncovering the associations (and associated objects) you are presently interested in, and going on from there, until you see all you want in this view. Once you have a view with all the objects interesting for your present occupation, you can easily swith which of these objects is visible in the content window at any given time, by focusing it in the map.

Constructing a map can also happen automatically, e.g. when browsing a web site in the content window: Each time you click a link, the linked site and the link association is added to the map. Othewise, creating a map by hand can quite tedious. (Especially as DeepaMehta, while introducing totally different high-level concepts, is very old-fashioned at the level of actual UI elements -- all the map management for example is done through nested context menues, requirering lots of searching and endless clicking...)

For some tasks, this seems way too static: To efficiently browse a filesystem (or generally to quickly move through any larger object structure), some more automated approach seems necessary. Not sure how to do that; but I think it should be possible to come up with something nicely extending the concepts in this direction.

There are some more fundamental problems with the existing DeepaMehta implementation, however. (Aside from the fact that it is written in Java...)

For one, it is centered around the idea that the future is in the net, with all intelligence in servers, and workstation machines being only dumb clients -- a pretty absurd notion. In spite of all the babbling of dotcom freaks, it's pretty obvious in any realistic view that the local system should always be the primary focus of a desktop framework. Networking is a mostly orthogonal issue, which can be nicely integrated using other mechanisms, for example a transparently networked filesystem like Plan9. Designing a desktop system as client-server from the ground up, only makes it more complicated and less flexible, creating much more problems than it solves.

The second major problem with DeepaMehta is that it creates a world of its own, with little relation to existing environments. This obviously makes transition pretty hard. Moreover, as it's impossible (or at least very unrealistic) to run a whole operating system, including administration etc., with DeepaMehta alone, there will always be the DeepaMehta layer on top/beside the rest of the system; and like with any sub-plattform, this creates very serious integration problems.

For these reasons, I don't consider DeepaMehta as is very useful in a broader view. Yet, the navigation approach seems immensly valuable. So, how could it be integrated at a generic level in more traditional systems? I have a number of ideas for various methods how to implement the DeepaMehta navigation concept through extensions to the normal filesystem. (The Hurd architecture makes such extensions easy...)

The basic idea is, instead of having a special object database like DeepaMehta, to use traditional filesystem entities (files and/or directories) as the primary objects. We already have links in the filesystem, so one might imagine something based on that for the object associations.

In a very crude approach, the DeepaMehta semantics could be expressed by some defined structure in a traditional filesystem. Every object would need to have its own directory, containing the main file as well as the object type designator, additional attributes, and object associations. An association could be represented by a link referencing the associated object, plus some file or a second link denoting the object type. (The object type itself can be described by a special object...) Maps could be either stored in additional subdirs inside the object directories, with pointers (links) to the associations visible in the particular map; or in completely distinct directories, linking all the objects and associations visible in the map.

This approach should work fine with programs that are aware of the special semantics. However, it creates lots of other problems otherwise. If you access the objects with normal programms, you always have to specify the main file inside the object dir. If you copy or delete objects, you have to do it recursively on the object dir. Generally, it's very complicated and not really intuitive. Also, it requires a special structure, but this structure can easily get destroyed when accessing stuff with traditional filesystem tools. As traditional links are unidirectional, it would either be very inefficient to figure out the links in the reverse direction, or it would require redundant links in both directions, which need to be kept consistent.

An alternative is to extend the standard filesystem mechanisms. This would allow to attach all necessary information to the files themself, instead of putting them as additional items in a special directory. Various modifications to the linking mechanism are necessary for that: Normally, links point from a directory entry to a file or another directory. For DeepaMehta-like relationships, we'd need links that directly connect one file to another. Also, they would need to be bidirectional, and have a type attribute. Maps could be represented as special filesystem entities, which can be browsed like directories, but with a map-like instead of the traditional hierarchical structure; presenting an alternative view of the main directory tree.

This way, navigating through maps should work in a pretty nice manner even with traditional applications; and when accessing objects with normal programs, they behave just like ordinary files. On the other hand, there is no way to access the object relationships and other additional attributes, or to modify maps, with tools not aware of the new features. Also, this approach is very invasive in general, making quite fundamental changes to various aspects of the filesystem functionality.

I think a more hurdish approach is to introduce a Mehta-translator: If you want some file to appear as a DeepaMehta-like object, just set the translator on it. Without the translator, it appears like a normal file; but with the translator set, it is presented as a pseudo-directory, with all the additional information. This way, the main file can be easily accessed and managed with normal applications (without the translator active), but the additional properties can also be represented in such a way that they can be accessed with traditional tools. Unlike with the directory representation on a traditional filesystem described above, the translator can enforce consistency when properties are manipulated through the pseudo-directory. It can also introduce other special semantics for the pseudo-filesystem entities, making it more intuitive.

Maps can be implemented with another translator, which creates a pseudo-filesystem representing the map as a special directory structure. (Much like in the variant with special FS features, but through a simple translator instead of modifications in the normal FS.)

Some light and some shadow

Today I stumbled upon a very interesting article on propsed design concepts for KOffice, which makes Martin Pfeiffer the winner of the KOffice design competition. I haven't looked at the other contributions; but taken by itself, it looks like the award is well deserved.

The proposal contains lots of innovative, mostly good ideas -- as I've already hinted in some of my other postings, a feature IMHO sorely lacking from most UI concepts.

It implements, at least partially, many many principles I'm badly missing from existing GUIs. (Though by far not all, of course...) It also proposes some very interesting totally new ideas I haven't even thought about so far. It suggests solutions for some problems I'm seeing in existing systems. It makes me think more consciously about some things I vaguely considered before. All in all, it starts a lot of valuable thoughts. And, of course, it also has a couple of ideas that I do not agree with at all. (Well, what you'd expect, perfection? ;-) )

I won't dwell on all the specific aspects now. (I might pick up some in later posts.) It's way too much interesting stuff. Instead, I'll pick a single, very very fundamental issue -- one which this proposal sadly gets (almost) all wrong: Integration.

While it's obviously right that better integration between the various office applications is desirable, there is a fundamental misconception how to achieve this. The idea of having individual "viewers" for the various document types, and something that integrates them, is even basically right -- in spirit it goes in the right direction, towards what I call a hurdish application infrastructure. (Though not very far.)

There is also the very important realization that a shared canvas implementation should be used for displaying the contents, making the different document type "viewers" basically only transform the documents. But having the canvas in a shared library is a poor choice -- though typical for traditional monolithic UNIX systems, which do not offer an integrated object access approach like Hurd's translators (or Plan9's FS servers), to make implementing common functionality as server processes easy.

Where the proposal is really fatally wrong, is what that "something" integrating the components should be: It wants a single specific main application, gluing the whole into a sealed system, creating a desktop for office work inside the main desktop -- instead of fixing that one to provide the necessary integration. Reenforcing the dominance of closed monster applications, buying inner integration at the cost of making any outside interaction awkward, thus creating ever growing subworlds, ever more alien among each other. Ever more duplicated functionality, as it becomes so painful to use anything not part of the closed subworld. Creating another Emacs.

Barring fun

Regarding the last post: Thinking about it a bit more, my statement that this is unrelated to the issues I'm usually talking about here, is not quite true.

On the contrary: "-Ofun" is a nice way of putting it; but a large amount of the stuff dicussed in the essay (maybe even all of it), really boils down to tearing down entry barriers -- here, in a social context. Just shows how crucial this principle is in general :-)

The importance of having fun

This is somewhat unrelated to the technical issues I'm usually discussing here. But I'm agreeing so fully, and the issue seems so crucially important to me, that I want to point to it anyways.

Eric S. Raymond pointed out many organisational issues for successfull free software projects in his famous Cathedral and Bazaar paper. Most of the things he mentions there are right. But all of them are actually only a function of the fundamental idea the -Ofun, pointed out in this article.

And I know no other project so badly following it, so exactly doing the opposite of -Ofun, like the Hurd :-( It's really that, and nothing else, what makes Hurd's progress so slow. Can it be fixed? I hope so...

The other side

After recently commenting on KDE, I've now also stumbled upon GNOME's project Topaz, which -- together with some other pages it links -- describes future ideas for GNOME.

The Focus is a bit different than KDE's. (Fundamental UI changes, including "extensions", are mentioned at the setout, but there are very little actual ideas. Most stuff revolves around under-the-hood technical improvements.) However, with regard to new ideas, it's just as disappointing.

There are a couple of quite fundametal changes suggested -- mostly not terribly new, but still. There doesn't seem to be any special emphasize on any particular one of these. I'm just picking the one most relevant in my opinion: The VFS.

There is a page on the new GNOME VFS, which in turn links to the Desktop VFS from freedesktop.org. They are both describing mostly the same ideas, and I'm not sure about the exact relation between those projects, so I'll just treat them as one.

So they have -- correctly -- discovered, that POSIX file handling is unsuitable for most of today's applications. Full agreement here: We made the same discovery when designing various Hurd translators. open(), read()/write(), close() were sufficient in times when most Unix tools worked as filters, sequentially reading input files, sequentially writing to output files. Most of today's applications require different semantics.

One very important operation, as the GNOME folks have accurately observed, is atomically reading or writing a whole file. They want it because most of today's applications read the whole file when "loading" a document, and store the whole file when "saving" -- which is wrong IMHO, but that's a different story :-)

Nonetheless, in the Hurd it is probably even more fundamental: When a translator is exporting data through file nodes, it is extremely common for clients to read the node contents into a string, or store a string in the node. A simple operation doing this in one call, would be awfully useful. Not only because both client and server need considerably less handling for that, but also because knowing that the client just wants to write the whole file, is very important information for the server.

Generally, we need much more semantics in file operations than POSIX offers. Think of inserting data in the middle of a file. With POSIX, the only way to do that, is to overwrite the entire rest of the file. This is not only complicated and terribly inefficient: In case the underlying file is served by some more interesting translator, it can actually pose serious functional problems, if parts of the file are overwritten for no real purpose.

For operations that don't fit any of the generic semantics (write entire file, insert data, ...), we probably need to introduce transactions, to allow manually grouping primitives into semantic units. (This is probably what OGI's comment to an older post was referring to -- which would mean that at last I've understood the idea behind this comment :-) ) For many translators, it's crucial to know whether an operation is completed and data should be processed, written to the store/network, whatever; or whether following calls will alter it further. With POSIX only, some translators can only be usefully implemented by employing quite sophicticated caching and heuristics, if at all.

So, the GNOMEs are right about the necessity of new filesystem semantics -- though I don't know if they'll get all the issues mentioned above right. Sadly, that's where sanity stops in their proposal(s). A special filesystem API for desktop use only? That's absurd. How did they get that silly notion, that the file access requirements of "desktop" applications are fundamentally different from command line tools or daemons, so much as to warrant a special API for desktop use alone?

Oh well, I guess that's the general problem of GNOME (and KDE): Considering the underlying system(s) as given, they tend to pile layers on layers of workarounds, instead of much more simply and usefully fix it right at the system core level. (Reminds me of MS Windows, which started as a desktop environment, and ended up being an OS... In just a few years more, we will probably hear people say: "GNU/Linux? Isn't that obsolete? I'm using GNOME!") This just shows that we really need a GNU kernel, so the developers of a GNU desktop environment won't need to sink tremendous amounts of time into working around limitations of systems they have to run on in lack of a native one, where they could get all the functionality they need... But well, that's a different rant.

So, now that we agree ;-) an extra API for better FS access is silly, what is the alternative? That's obvious: Just like we already have Hurd extensions to POSIX interfaces in many other places, we should try to extend the standard POSIX file operations with the stuff we need, without forsaking compatibility. I'm pretty confident we can do this. (I've already discussed some aspects in conjuction with device drivers.)

Clients not aware of the new semantics, can continue using the old ones. Those that want to use the new features, will check with the server whether it implements them, and fallback to the traditional stuff otherwise. Most of this can probably be handled transparently in libc (client side) and/or the FS server helper libs: If a particular server doesn't know about the atomic file read/write operations for example, it will just get a series of standard POSIX requests doing the job instead. No need to force a switch to an incompatible new API.

Involvement Engineering

Yesterday I stumbled upon KDE Plasma, which is an attempt to bring some innovation into KDE's next major release.

So they discovered that desktops, except for minor incremental fixes, have hardly improved over several decades. This is a good start.

Sadly, it mostly already stops here. It seems they made the discovery, but have no idea what to do about it. Reading their "Vision" document, it is much like reading Microsoft announcing a new Windows version: Everything will be so much better, and more buzzword-compliant, and new possibilites, and stuff... But going down to the details, you realize that hardly anything has really changed.

OK, that is a bit unfair. There is a bit more in Plasma beyond the realization that change is necessary. There is for example some rough notion (in the form of extenders), that the desktop geometry needs to become more dynamic -- though they constrain it to a specific domin, instead of making it a more fundamental concept. There is also a rough notion of alternative sessions for different use situations -- though they create a specific solution for desktop layout, instead of considering a more fundamental concept. Yes, there are some things going in the right directin, even if in extremly small steps.

And there is this applet concept. It isn't really new; similar concepts existed in specific environments for a long time. It isn't even terribly new as applied to the desktop. Beginnings existed in the form of dockapps and similar for quite long; and Apple, though not very long, has something -- I believe -- quite similar to what they are planning in Plasma.

Nevertheless, this made me consider the topic a bit closer. So, what are the distinctive concepts that make up these applets?

Basically, we have comparatively small, independantly developed hacks, adding functionality to the application/system, without touching the core. There are numerous examples of such an approach resulting in a thriving community of extension developers: TeX/LaTeX, Emacs, Perl, Firefox and Greasemonkey come to mind.

The most fundamental property is probably the fact that those hacks -- whether called extensions, widgets, applets, whatever -- do not touch the application/system core, but are developed using some extension language/API. There is a number of implications from this: People can start writing improvements without knowing the language of the core (usually more low-level than the extension language), or the core code -- just build on some simple APIs. People can do their improvements independantly, without discussing with the core developers, according to an own release schedule etc. They can distribute them independantly. Problems are less critical, because they do not touch the core, and because they affect only those people who explicitely chose to use the extension. For the same reason obscure features can be implemented without bloating the core. And so on. While some of the advantages may be actually organisational or even psychological rather than technical, all of them contribute to lowering entry barriers for people to contribute improvements, for getting actively involved.

The good news is: The Hurd lends itself to such a concept. With a hurdish application infrastructure constructed of small components building upon each other, the whole system actually could be considered the applet idea taken to extreme. (Which is one possible explanation why I like it...) All services being exported via filesystem interfaces, we also always have very accessible extension APIs to easily plug applets into. And with the concept being consequently implemented throughout the system, it's much more powerful than any solution limited to the topmost layer like in Plasma or any other specific approaches possible on traditional systems.

Now the only thing we really need to be careful about, is to ensure Hurd components are really easy to create, distribute, and install...

Persistance vs. Insistance

Session management is a thread of thought I only recently took up in my considerations. Nevertheless, it touches on many other aspects -- it looks like this has been the missing link in various contexts. But more on this later.

Session management basically means: I don't want to manually restore my work environment on each bootup; I want it to come up just as I left it -- usually at least.

There is little doubt working session management is something users really really want to have. I say working, because retrofitted concepts like X/KDE/GNOME/whatever session management (which is the most explicit most users know), tend to be incompletely implemented and thus useless.

In a UNIX environment, most of the system doesn't know session management. To change the startup behaviour of something, you usually need to explicitely change settings, either by editing config files; or in GUI applications, usually there are a number of startup-related settings in the options dialog.

Nevertheless, besides the not-working X session management, you'll find many places where session management aspects have sneaked in, if you look closer. A typical example would be the soundcard mixer init scripts, which save the current mixer settings on shutdown and restore them on next startup. Some applications, like Opera or Vim, have options to explicitely and/or automatically save/restore their sessions. The shell (and some other command line based applications) save the command line history. In the Hurd, we have passive translators, as kind of session management at the filesystem tree level.

And then there is screen, which implements kind of a poor man's universal session management: When you detach a screen session from the terminal, the software just runs on in the background -- instead of doing real shutdown/startup, you just detach and reattach your session. (Screen sessions are usually used on servers, i.e. systems that run permanently with rare interruptions.) Those who know this feature usually love it -- by this little "trick" it implements something resembling session management that really and always works, because the applications needn't be aware of it at all...

Another, less obscure example of transparent and thus (more or less) reliably working and widely established session managment-like behaviour, is suspend to disk. Just puts the whole memory image to disk, and restores the exact same situation on resume. (The exception are most hardware drivers, which obviously need to be aware of the suspend. This was actually the main starting point for my considerations -- the suspend infrastructure of the hardware driver framework could and should be extended to the application level...)

A very similar approach -- which could be considered an extension of suspend, although it has a completely different origin -- is persistency like in EROS. The difference is that here the image is saved not only on suspend, but periodically (every 5 minutes), so the system will come up in the last state even on a power outage or so.

One side effect of system-wide persistence (again, with the exception of system core and hardware drivers), is that you need no sophisticated system boot and program startup mechanisms -- you just add objects to the system once, and they live on forever. (Unless you decide eternity is a bit too long, and get rid of them earlier...) Which is the very reason why EROS has it: Making the whole system persistent seemed easier than creating a method for secure explicit storage/retrival of capabilities. But they claim it's desirable for usability purposes also...

However, there are downsides to this. Completely transparent session management works fairly well in case of specific things like screen, because of the scope of a screen session usually being quite limited; and suspend, because of it typically being used only for fairly short breaks in work that means to be continued exactly where you left off. But a completely and always persistent system, creates various problems.

For one, to update software, you basically need to create a new object in place of the old one. If you want to preserve state information in the process, you need to implement this explicitely: Either run the new version in parallel and pass the state from the old one to the new (session handoff), or dump the old state to a third party temporarily and read it when starting the new version (session saving). In both cases, you are effectively implementing explicit session management -- all of the nice transparency is gone.

This actually doesn't only happen on updates: If you want to move some state information, e.g. when replacing hardware or working on a different machine or whatever, you need the very same protocols -- only that the actual state transfer gets more complicated.

Another, related problem is flexibility in general. Transparent persistance is a sledge hammer approach: You always get the whole thing. But in many situations, you do not actually want to, or can't, restore the system in the exact same state. Maybe your system configuration changed meanwhile, so some stuff won't work or doesn't make sense anymore. Maybe you screwed up and explicitely want to easily get rid of parts of the old state. Maybe you want an easy way to active only parts of your session at times. Maybe you want to carry along part of your environment to different machines. And so on. You might even have your system on a USB stick and need it to adapt to a different machine each time you boot it!

So what we really want is a flexible system of subsessions, that can be restored or not upon demand and/or resource availability. One session for the core system, various sessions for individual hardware components, a number of sessions for background services, some sessions for higher-level system components traditionally handled by runlevels (networking, windowing environment), and lots of sessions for individual parts of your application envirionment -- a music player session, a news reading session, a communication session, various sessions for projects you are working on at times, etc. You get a whole tree of subsessions and sub-subsessions building upon each other. (Those subsessions are also related to various other mechanisms like resource management; security; and forming an application infrastructure from generic components, as touched in my post on a hurdish X implementation -- that's the "missing link" aspect mentioned at the start. I'll handle those in other posts.)

So between disfunctional retrofitted session management, and sledge hammer total persistence, we really want a session management approach that is not quite transparent, but fully integrated and consistently implemented throughout the system.

Implications from that requirements, as well as ideas how it could be implemented, I'll leave out for now, as this is already getting quite lengthy...

Welcome to HELL

An idea I'm contemplating for a while now: Hurd Emulation Layer for Linux.

The idea is to create an emulation environment, which would allow running hurdish applications on top of a familar GNU/Linux system. Now this is a bit unsubstational of course, not having explained so far what I mean by hurdish applications, except for the mention how I imagine a hurdish X implementation.

Well, doubtless this will be motif in many coming entries. So just for starters: The fundamental concept is to replace bloated monolithic all-in-one applications by many small translators providing specific services, and some mechanisms to combine them into systems providing the desired overall functionality. Ideas how this would work in particular, as well as why I believe that on the Hurd -- unlike on traditional UNIX systems -- this actually *can* work, I will also cover in other posts.

Well, having said it is not possible on traditional UNIX systems *natively*, I do think it should be possible to create a limited, somewhat Hurd-like environment on top of such systems -- far from perfect, but just enough to run hurdish applications with no or only little changes. Some mechanism to emulate translators, something to map Hurd free form RPCs on UNIX IPC mechanisms (terribly inefficient but probably possible), something to emulate some of the Hurd extensions of the traditional UNIX interfaces. Of course it would break as soon as programs try to dig too low, using specific functionality of Hurd core components etc. But for most higher-level applications, it can probably work.

As for actual implementation, I only have a number of rough ideas. We might be able to emulate translator functionality by playing with FUSE. To run hurdish programs, we could link them to a special modified variant of the Hurd libc, emulating the functionality on top of Linux instead of contacting Hurd servers; or we could employ some proper sandboxing, intercepting system calls etc. For some commonly used Hurd services, we might even try to run hacked Hurd servers, or replacements providing the functionality in the foreign environment. Non-hurdish processes could be provided with some hooks via LD_PRELOAD to better cope with the emulated Hurd features when interfacing to hurdish processes.

One fundamental question is whether to wrap every single hurdish process in its own HELL, mapping the Hurd functionality to UNIX mechanisms such that the individual processes can be combined in a Hurdish manner; or to provide one single big HELL environment (kind of a userspace Hurd implementation), interfacing to the non-hurdish world only at the outer borders. The first one may be more desirable, as it integrates better into the system. However, it probably means more overhead; could cause some troubles due to non-perfect mapping of Hurd features; might be harder to use; and most likely is harder to implement.

Of course, there are also strategical considerations involved -- do we really want to have HELL? This is a similar uncertain question to whether porting free software to proprietary systems like Windows or Mac OS should be encouraged. (Though not politcal in the HELL case.) On one hand, it encourages authors to write hurdish applications, as with HELL the audience isn't limited to actual Hurd users; people who for various reasons can't or don't want to do the switch, can still use these programs. (I'm considering making netrik a native Hurd application. However, I don't want to exclude GNU/Linux users alltogether. That's how I originally came up with HELL.)

Also, discovering some of the advantages of the Hurd approach while using HELL, people might be encouraged to switch to the Real Thing (TM), where these advantages come into their own, the hurdish concepts being used consequently throughout the whole system, not only in a limited environment.

OTOH, people being able to use hurdish applications and reap some of the advantages on a traditional system, might actually get lower inclination to switch; they might get stuck with a suboptimal system forever.

I tend to believe HELL would act more as an enabler for Hurd than a stopper -- YMMV.

Return of the Bad Guys: A tale about a little interface

Once upon a time, The Fathers were dissatisfied with the then current situation of graphics support on Linux. At those ancient times, there were basically two options: svgalib or X. It was an or, not an and, because these two didn't mix terribly well.

So The Fathers set out to fix this situation, and created a sophisticated scheme for a General Graphics Interface. It was designed to encompass the existing options in a nice overall framework, allowing the mixed use of both svagalib and X. It should allow running svgalib applications in X, and X on top of svgalib; even nesting them at will. In short, it should allow seamlessly running any graphical application in any environment, all reliably and securely in parallel.

Fulfilling those goals basically required two major components. The environment independance would be implemented by a library abstracting the different backend targets in a common interface, plus a set of associated frontends to allow using all the existing applications on top of it. (svgalib wrapper for running svgalib applications, and XGGI for running X.) Of course, applications could also use the native GGI interface directly, making use of the backend abstraction it provides.

libggi only as an abstraction for X and svgalib targets however couldn't fulfill all of the desired goals: Having no provisions for secure sharing in svgalib, and sharing only between its clients but not with the outside world in X, reliably using an X server and some svgalib programs on the console in parallel is not possible. Doing this required a more sophisticated, native target implemented mostly in libggi, but requiring some kernel support for secure sharing. That support would be accessed by a Kernel Graphics Interface -- the second major component -- which would implement the actual hardware access and sharing in a kernel driver, just like other drivers do. It would cover just the stuff really necessary for secure sharing (mode setting, framebuffer setup, acceleration pipe access), but not any higher-level logic that can do without kernel support. (It would also require some rework of the console system, to work with KGI.)

So what happend when The Fathers introduced GGI/KGI? Well, libggi found it's niche as a nice multi-target graphics library when used directly throuh the GGI API; it is still active and becoming more and more powerful to this day. The little sister KGI however had less luck: It was immediately faced with strong backlash, from several sides. The X window folks considered X to be The One And Only (TM) graphics interface, that should be used by everything. The kernel guys opposed the idea of integrating graphics drivers into the kernel, maybe because "putting graphics in the kernel" is considered a windows thing or something, and people suggesting it were considered the Bad Guys; maybe because people didn't realize the fundamental difference between low level graphics drivers -- which like other drivers belong into the kernel -- and higher level graphics handling that obviously does not.

Maybe the time wasn't right for an idea that seemed so radical back then, when graphics support was still considered something very special that is best handled by an external entity; when people still believed X could be integrated better into the system over time, removing the entry barriers and making other approaches unnecessary.

While The Fathers struggled on afterwards, the steam was mostly out; KGI soon lost momentum.

Today, things are quite different. In the meantime, the simple fact that some platforms just do not have such a thing as a text mode, forced addition of framebuffer support in the kernel; but once there, it was warmly received -- it turned out many users *want* that kernel support, avoiding the considerable problems associated with the pure userspace implementation done by X. In fact, people even created a complete graphics system called DirectFB, which hacked accelerated graphics support on top of the kernel framebuffer interface. (But lacking kernel support for the acceleration features, inherited many of the problems of svgalib.)

Also, it turned out even X required some kernel support for efficient 3D acceleration, introducing the DRI/DRM interface.

In short, today kernel graphics is a widely accepted fact. Today, people no longer concentrate on preventing graphics support from entering the kernel, but on how to implement a *clean* interface; discussing a mode setting API and everything.

Maybe it's time for realizing that the little baby child KGI, while quite lifeless from the bad treatment it received, is still around; that it's not that ugly after all, but on the contrary offers quite exactly what people are looking for now: A clean, generic, well thought out interface for supporting mode setting, framebuffers, and acceleration feature access in the kernel; and that in fact it has done so all the time, though not recognized for that.

Maybe it's time to give this little child a hand, to let it grow, shape it a bit maybe -- so it can become really great.

The next step

Most people consider the Hurd only a project to replace monolithical kernels. IMHO, it is more.

On the irc.freenode.net##hurd channel, we just had another discussion on the X window system. (You can read it up in the channel log.) Which seems a good occasion to summarize some of my thougths here.

While there are a number of other things that are flawed about X (which I may touch in other posts sooner or later), there is one really fundemental problem: The X server is basically a gigantic monolithic beast, suffering from much the same problems as monolithic kernels. (Flexibility, extensibility, robustness, usability, security, etc.) And it needs to be fixed in much the same manner.

The nice thing is that the underlying Hurd concepts (RPC, translators, etc.) not only give the foundation for reimplementing the functionality of monolithic kernels with a multi-server system in userspace, but also for refactoring monolithic higher-level infrastructure components like X -- just like the Hurd is splitting monolithical kernels into a set of interacting servers handling individual parts of the functionality, a hurdish windowing system will split the functionality of X into individual servers. (And just like the Hurd uses libc to implement POSIX interfaces on top of the multi-server system, allowing for a smooth and flexible transition to more powerful concepts, we will need a replacement X library implementing X interfaces on top of the multi-server windowing system.)

Unite and Conquer

Returning to my POSIX level driver proposal: Many people expressed concerns that the standard filesystem semantics I want to use for (most) driver communication are too slow and not really appropriate.

The proposal explicitely mentions the possibility of using shortcuts wherever we experience serious performance problems with FS sementaics. Now recently I had some intitial discussion with Peter de Schrijver (p2-mate) at freenode.net#hug, on what specifically are the problems with the POSIX interfaces. It turns out that mostly the drivers have some generic, quite similar requirements. This means that instead of creating specific shortcut protocols only for some extremely demanding drivers, we should probably rather focus on a few generally useful extensions. I like this :-)

For one, drivers are often serving quite a large amount of very small requests. Pure POSIX semantics would introduce quite a considerable overhead here, because each single request needs to establish a session (open()/close()), unless it already has a permanent one; do addressing and other setup (seek()/ioctl()); and finally do the actual data transfer (read()/write()). In POSIX semantics, we need an extra RPC for each of those steps, plus some bookkeeping overhead. (If we want to avoid ioctl()s for the setup -- because they aren't very transparent, killing the major advantage of filesystem semantics -- there is even more overhead, as we need to introduce an additional file descriptor for setting request options.) A more appropriate protocol would wrap all of this in a single RPC.

Well, there is an important observation to make here: The optimization is useful not because we are dealing with drivers here, but because the drivers have a specific requirement (efficient handling of many small requests), which could also emerge in any other program -- I'm pretty sure many higher-level translators will profit from an optimization for that just as much. Which confirms my view that drivers aren't fundametally different from other programs, and we really want generic extensions to the POSIX/Hurd interfaces, rather than a special interface for drivers.

I guess we could implement this extension mostly transparently, with servers implementing it optionally as an optimization. Maybe even handle it in the FS server libraries, so translators not aware of the shortcut will just get the single RPC presented as a number of independant callbacks. Not sure about the exact implications, though.

Another property of drivers is that typically they are working with data that is structured in blocks, and doesn't really fit well with the POSIX assumption of all data being represented as sequential streams. Serializing the stuff (using XML or whatever) would by tremendously expensive, considering that drivers usually aren't processing the data at all, but only working with some status information, and passing the data on. What we really want is a memory container for the actual data, and a second independant memory container with per-block status information. (Including block boundaries in the case of variable-sized blocks.)

Again, we have a requirement that isn't really specific to drivers at all: Quite a lot of high-level programs are actually working with similar non-serial data, and would greatly profit from a generic extension allowing for passing several memory containers in a single read()/write() RPC.

Another possibility I'm considering, instead of adding a number of independant extensions handling different aspects that need optimization, could be just creating some generic method for merging several POSIX calls in a single RPC. This would be extremely flexible and powerful; however, I'm not sure it could be implemented without being too awkward. Also, the fact that it would need to handle the requests in a very generic fashion, might skyrocket complexity and nullify some of the performance gains we are striving for.

And Now For Something Completely Different

The last two days I spent a considerable amount of time thinking about acoustic processors. Not sure this really fits here, but maybe some consider it interesting nevertheless.

An acoustic processor is normally a box you integrate into your stereo to improve the sound. Specifically, by adjusting the audio signal in such a manner as to balance anomalies created by your hearing room and/or your equipment.

Now acoustic processors aren't terribly popular, so far. There are probably several reasons for that. For one, high end audio purists are generally sceptical about any modifications to the audio signal. Also, such a box isn't easy to integrate: For several reasons (price, calibration, distortion), such a processor can feasibly only work on digital signal. However, while CDs have been around for quite a while now, only recently most other audio sources are becoming digital. And even with digital sources like CD players, the signal is usually already converted in the source and passed on as analog. No home for the poor little acoustic processor.

Last but not least, the price tag: Such a box is all but simple, and consequently all but cheap. Those who need it most -- with cheap equipment and inappropriate hearing rooms -- can't afford it; those who could, have fairly little need.

Being one of those who'd need it but can't afford (my equipment isn't bad, but far from perfect in the low bass area; and my current hearing room is terrible), I've been playing for quite a while with the idea of going a route even I could afford: Use software to do the acousting processing offline. Grab all my CDs, torture them in the offline acousic processor, and burn the result again -- producing CDs that will sound terrible in any other setup, but should be perfect with my equipment and hearing room.

So, how does such an acoustic processor work? Well, the simplest variant is just an equalizer with a lot of bands (128 or so), and an auto-calibration system. (Measurement microphone in conjuction with a program to run a test and adjust the parameters.)

However, this simplistic variants do not work terribly well. The problem is that just adjusting absolute volumes doesn't help too much. Temporal effects play a big role: For one, if the speakers or the hearing room generate resonances, the effective volume may depend on the length of the sound. Even more importantly, psychoacoustic effects make sounds prolonged due tue resonances/reverberation seem relatively louder.

(Furthermore, it's desirable to correct phase discrepancies between channels and frequency bands, for improved positioning and naturalness; to correct dynamics for improved vitality and resolution; and so forth... But that's definitely beyond my amateur means.)

So what we want to do is adjust the volumes of individual frequency bands depending on the signal levels. When a sound sets in, the relevant frequency band's volume is adjusted by the stored volume factor for short sounds in this band; when it persists for a longer time, we successively move to the factors for longer sounds. Well, at least that's my idea on how it should work.

My major problem is my very limited knowledge of acoustics, psychoacoustics and digital signal processing. As a layman, I believe we first need a frequency analyzer, continuously tracking the signal level per frequency band in the input signal over time. This frequency analyzer needs to have similar properties to our hearing, I guess. Now using some function involving the different adjustment factors and the signal level history, we can determine the necessary current volume adjustment for each band. These levels are perpetually fed into an equalizer, processing the input audio signal.

Well, so much for the theory; now if someone could tell me exactly how to implement this...

Another complication is that having no measurement equipement, I'm trying to determine all the necessary volume adjustment factors by hand/ear, using various test sound. So far, my experiments were rather discouraging; but I still have hope... (For the first time in my life, I'm considering a wireless keyboard.)

And well, once remastering all my CDs in this manner, I'd like to use the occasion also to fix some evident recording errors... Most notably, undo this abominable moronic dynamic compression most CDs are fucked up with. How do we do that, again?...

Design by Bulldozer

In the previous post, I mentioned there are some very fundamental advantages in my POSIX level driver proposal, related to usability.

Now I want to pick up on one of those, which will be a recurring theme in this blog, being an extremely important issue: Accessibility.

I do not mean accessibility in the usual interface-related meaning of posing no barriers to people with disabilities. I mean accessiblity in the sense of not posing barriers to about everyone.

Most developers, and even many UI designers, seem completely unaware how extremely important accessibility is. Somehow they assume if something makes sense to them, it is good. The thought doesn't even cross their minds, that it might mean considerable work for others to learn the concept.

Look at which technologies are successful. The WWW is popular because it has low entry barriers. People are more likely to participate in wikis than contribute to static pages because they have a lower entry barrier. And so on.

Look at Firefox. Why is it so popular? Because it focuses on those features that are easily accessible. Popup blocking is a good feature, because it is obvious. So are tabs. Or the search bar.

Compare this to Opera. If you configure it to ask about setting cookies, for example. You get a dialog that presents you with more than half a dozen of options how to handle each cookie, some of them I don't even understand. (At least in the versions I tried.) And Firefox? It presents you with a very simple dialog, having only a few obvious options. Maybe its slightly less powerful; but still covers what you want in about 99% of all situations, and is at least three times simpler. Meaning about an order of magnitude more useful.

There are many other examples of features in Opera that are quite interesting, but so hard to use that they have no practical value. Features so complicated or obscure that hardly anybody will bother to learn them, are just useless. Sure, there are always a few nuts taking considerable pains learning even the most obscure feature of some program. However, if it takes more trouble to discover, learn, configure and get used to some feature than it saves in the long run, this is just an end to itself. You can boast how powerful your program is and/or how well you master it. But that's about all the value you will ever get out of it.

Accessibility is important not only for GUIs, but really at all levels of the system. Let's take one example from the driver proposal: Among many other possibilities, it allows control of who is allowed to run what drivers, simply by changing file permissions on the underlying device nodes. Now, of course, you could implement some kind of permission system in any other driver framework... But requiring some obscure special mechanism, with some kind of config files in the background or somehting, not only is it considerably less flexible, but actually much much harder to set up in the first place. Unix file permissions on the other hand are obvious and a well know concept to every Unix admin -- there is nothing you need to learn or remember; just by looking at the nodes you can guess what to do.

And it goes even further down. All of this is true for the system internals, programming interfaces, everything. Making functionality accessible, tearing down entry barriers, is among the most important design principles in about any kind of software developement. (I'll show how this applies in various contexts in other posts on more specific topics.)

You will be assimilated

I thought about a number of things today, but no time for writing it up, because I had to write a long answer mail regarding my POSIX level driver proposal for Hurd on L4.

The foundation of this proposal is the fact that on a multi-server microkernel system, we have quite a lot of freedom about how to implement hardware drivers -- my take on it being to treat them just like ordinary applications, with only a minimal set of special mechanisms for driver-specific stuff, on the premise that drivers actually aren't that much special in their nature, and shouldn't be in the implementation. This offers a large number or advantages over other approaches: To users, admins, and system distributors, and even application and driver developers. (Of course, the advantages mostly relate to usability :-) )

The most important of those advantages are of a very generic nature -- stuff that I will cover in other posts sooner or later. (Though maybe in different contexts.)

The linked document is a longish, very technical description of my proposal, explaining what it is about, trying to outline some of the advantages (though probably not very well), and describing my ideas on a possible implementation on Hurd/L4 in quite a lot of detail. Especially the last section requires some knowledge of the Hurd, L4, and the Hurd port to L4.

Now probably I will be dreaming about competing driver framework proposals for Hurd/L4. Not sure yet whether this will be pleasent dreams or nightmares.