[Simon F]

Just wondering if there are any *nix gurus here who could help with a tricky little problem (or perhaps point me at the right location).

I'd like to create (for want of a better term) a "synthesised" file, a bit like a named pipe/FIFO, except that when a program opens that file for reading, the system actually runs a program that generates the content 'on the fly'. (In a way it is also analogous to an executable script where the first line of the script tells how to interpret the remainder.)

An example of what I'd like to achieve would be, say, decompressing a gzipped file transparently so that the reading program doesn't even know that the real source is compressed. Simply opening the file would automatically launch the decompression tool. (Unfortunately, reading from stdin and using a pipe seems to be out of the question).

Has anyone seen this sort of thing before?

TIA
Simon

[Kalin]

FUSE might help.

[Simon F]

Thanks. I'll take a look

[SPM]

Originally Posted by Simon F

I'd like to create (for want of a better term) a "synthesised" file, a bit like a named pipe/FIFO, except that when a program opens that file for reading, the system actually runs a program that generates the content 'on the fly'.

Unix/Linux device drivers work in exactly this way - they appear as files on the system, and can be opened as character or block devices. What you may be looking for is how to write a Unix/Linux device driver.

As for reading compressed files transparently, lots of applications already do that. You may want to take a look at the source code of these applications.

[Simon F]

Originally Posted by SPM

Unix/Linux device drivers work in exactly this way - they appear as files on the system, and can be opened as character or block devices. What you may be looking for is how to write a Unix/Linux device driver.

As I am not the admin on these systems, it has to be something that is in user space.

As for reading compressed files transparently, lots of applications already do that. You may want to take a look at the source code of these applications.

Are you referring to, say, using zlib to transparently open either gzipped or standard files? I've used this in my own applications, but as I don't have either access to the source for the other tools or time to do the modifications, that option is out. <shrug>

[SPM]

Originally Posted by Simon F

As I am not the admin on these systems, it has to be something that is in user space.

Are you referring to, say, using zlib to transparently open either gzipped or standard files? I've used this in my own applications, but as I don't have either access to the source for the other tools or time to do the modifications, that option is out. <shrug>

Pretty well every Linux file manager will browse through compressed archives transparently and open files contained in them - for example on Fedora 9, nautilus (the native Gnome file manager), Thunar, and PC Man file managers, and even command line file managers like Midnight Commander (mc) allow this. These are open source user space programs, the source code is freely available, and it should be fairly straightforward to look at the source code for the libraries used and the techniques used to do the transparent access as an example and use it if your own code. mc for example is a fairly compact file browser and so there should be less code to look through. You should perhaps install Fedora Linux on a PC and check if the functionality is what you want. It may be just a question of adding a hook to an existing open source application to allow programmatical access.

I am not familiar with zlib, but it should surely be fairly straightforward to get a listing of the content of a compressed file and feed that back to your program for display to allow the user to select files of a subdirectory in the compressed file and so on. Since mc, Thunar etc. and other programs routinely do this, if you don't know how to do this, all you need to do is to look at the mc, Thunar code for an example of how to do it. Most GNU Linux utility programs and libraries have already been ported to Unix or vice versa or are trivial to port to Unix.

As for faking file systems from user space without root access, it is obviously a serious security violation if anyone can do this - it opens the door to Windows 98 style security (or lack of it) and rampant cracker/virus/trojan vulnerability, so Unix/Linux systems sensibly don't permit this. On a properly administered Unix system, users can normally only mount file systems which are declared as user mountable in the /etc/fstab file, and all device files are only accessible to root. FUSE allows user space file systems, but you still need to compile the options into the kernel, which again for obvious security reasons requires root access. FUSE used in a properly administered Unix/Linux system would normally be used in a way that would not compromise the system security - eg. the user space filesystem programs would run under a non-root psuedo-user who no ordinary user can log in as - a kind of a second level administrator if you will.

[Saem]

Hi Simon,

Just wondering if you found a solution to your problem?

[Simon F]

Originally Posted by Saem

Hi Simon,

Just wondering if you found a solution to your problem?

The sys admins just added N more terrabytes of storage!

[Saem]

LoL, I guess that works too. I suppose considering the price of hardware it was probably not that much more expensive, if at all, than your time on the problem.

Thinking about it, without admin access, I don't imagine you'd want that to be possible, as you could easily hijack things that way.