Keep your Filesystem tidy: a crazy Idea?

[taviso]

The other day I started thinking, my hard drive is a mess. Even a simple package like bash provides nearly 30 files and directories, man pages, info pages, documentation, etc.

Code: Select all

 $ wc -l /var/db/pkg/app-shells/bash-2.05b-r7/CONTENTS 
     28 /var/db/pkg/app-shells/bash-2.05b-r7/CONTENTS

Only 4 of the files it provides are actually binaries, the rest are what I'm going to call "static Files", that is, files that you would never normally edit and are only there for reference/documentation/etc. This includes man pages, info pages, skel rcfiles, etc.

Code: Select all

$ egrep 'man|info|doc' /var/db/pkg/app-shells/bash-2.05b-r7/CONTENTS  | wc -l
     20

So how about this, what if I were to embed all the static files into the ELF executables? let me explain, its relatively easy to create a new section in an ELF executable, it doesnt effect the program at runtime, as we can mark the section with the flags "noload, noalloc, readonly, etc", meaning that the data wont use any extra resources and wont take any longer to start.

I've made a working replacement for the portage `doman` and a utility to read embedded manpages that prototypes this idea, you can download them here:

http://dev.gentoo.org/~taviso/files/doman
http://dev.gentoo.org/~taviso/files/printman

Code: Select all

$ doman
* error: usage: doman <executable> <manpage> [<manpage> ...]
* error: not enough arguments
$ doman bash bash.1.gz bashbug.1.gz builtins.1.gz rbash.1.gz
$ printman     
* error: usage: ./printman <executable> [<index>]
* error: not enough arguments
$ printman bash
  # manual                         offset     size
-----------------------------------------------------
  1 bash.1                         0x97904  64807
  2 bashbug.1                      0xa762b    713
  3 builtins.1                     0xa78f4    442
  4 rbash.1                        0xa7aae    151
$ printman bash 1
BASH(1)                                                                BASH(1)
NAME
       bash - GNU Bourne-Again SHell
(...)

And there could be some wrapper for man so that users could still type `man bash`, and some ELF containers for non-ELF applications (perl scripts, etc).

metadata, maintainer information, etc could also be embedded in it, and with some friendly ncurses/X frontend to explore the embedded data I think users would be impressed.

Opinions?

[Ari Rahikkala]

Lessee if I'm the first to ask this question... would this effect the time needed to load those binaries?

Hey, it's the Gentoo community, performance is critical .

[taviso]

Lessee if I'm the first to ask this question... would this effect the time needed to load those binaries?

Hey, it's the Gentoo community, performance is critical .

hehe, nope it wont make a difference, you have to mark a section with the "alloc" and "load" flags for the loader to even notice them

[Azaghal]

I don't really think the current way is a mess... all binaries are with all binaries, all manpages with all manpages...

The windows way... now that's a messy filesystem!



Anyway, about your idea, it would make things far less transparent, if your man page is 'inside' the binary...

[idl]

Anyway, about your idea, it would make things far less transparent, if your man page is 'inside' the binary...

Not realy, he mentioned a wrapper for man to make it seem transparent.

[Azaghal]

Well it could be useful for metadata perhaps (icons), but certainly not for manual pages...

It only obscures things, it's like sweeping the dirt under the carpet.

If I have a binary I really don't want 10 man pages to be inside it. Let alone having to fiddle just to copy it onto a floppy or something...

This would be like an executable tar file, just more obscure.

[shash]

I mostly agree, Azaghal. It'd be great for things like default icons, bitmaps, and things, but not for man pages, and other larger things. I prefer having them outside - one reason being, man pages are useful for a whole lot of troubleshooting, and things, and if that were in the binary, and the binary is broken, the man page might not survive.

[JHuber]

this is what OS X does with its executables. its may look clean but to actually get to something to change it can be a pain in the neck.

[avenj]

I would think this would be handy in situations where you have a small disk. With a 4k blocksize, you're wasting a lot of space by keeping all of those files separate from the executable.

Very neat idea.

[dgt84]

This does seem like a very neat idea, though I don't know how useful it actually would be for most people. I [mostly] like the current directory layout of where all files are, and the fact that I can easily access each one, without having to decipher it out of an executable.... anyone remember how much of a pain it could be to get icons out of a .dll or .exe?
On the other hand, I also think the directory structure could use some cleaning up... Why are a lot of apps moving to /usr/local and such? It seems to get more split up than to be tightly integrated and standardized. But I guess that's just my 2 eurocents...

[idl]

This does seem like a very neat idea, though I don't know how useful it actually would be for most people. I [mostly] like the current directory layout of where all files are, and the fact that I can easily access each one, without having to decipher it out of an executable.... anyone remember how much of a pain it could be to get icons out of a .dll or .exe?

The tools would do this transparently, you'd never notice the difference when calling man.
Tools to extract man pages or other files would be very easy to write.

[dgt84]

This does seem like a very neat idea, though I don't know how useful it actually would be for most people. I [mostly] like the current directory layout of where all files are, and the fact that I can easily access each one, without having to decipher it out of an executable.... anyone remember how much of a pain it could be to get icons out of a .dll or .exe?

The tools would do this transparently, you'd never notice the difference when calling man.
Tools to extract man pages or other files would be very easy to write.

Well then, by all means, go for it! I guess there would be a use for most people for a cleaner file system structure, and it is rather cool how it is done on a mac. I didn't mean to sound too negative about it before!

[zhenlin]

this is what OS X does with its executables. its may look clean but to actually get to something to change it can be a pain in the neck.

False!!!

OS X executables are singe binary files with only code.

However, they come in a nice package, which is just like any other folder. Inside are the resources they need, images, localisation strings, default configurations, internal libraries etc.

What you're talking about comes much closer to Win32 executables, which seem to be a rip off of the Mac OS Classic executables.

I'm completely against this kind of bundling. I want to be able to view man pages using any vanilla text editor/viewer.

So - the Mac OS X system is the best compromise. A single folder containing all files needed for proper execution of the program.

[taviso]

I'm completely against this kind of bundling. I want to be able to view man pages using any vanilla text editor/viewer.

Well, they are not plain text files..they are currently gzipped troff text, there are at least two steps in getting them into plain text.

Code: Select all

$ gzip -dfc /usr/share/man/man1/true.1.gz | groff -S -s -p -t -e -Tascii -mandoc | less 

the reason why you can just run `less <manpage>` and get something that makes sense is that less uses its preprocessor (lesspipe.sh) to process the manpage before displaying it. In this system, i could also provide a rule in lesspipe.sh, and a vim plugin, etc, you would have to jump through no more hoops than you do now to get the data.

Would you have known how to get groff to format a manpage without looking it up? I know I wouldnt have.

It only obscures things, it's like sweeping the dirt under the carpet.

Thats true, but I quite like the idea of having everything belonging to an executable embedded within it.

this is what OS X does with its executables.

I didnt know that, interesting...there must be something in it from an end-user perspective if Apple are doing it ?

I would think this would be handy in situations where you have a small disk. With a 4k blocksize, you're wasting a lot of space by keeping all of those files separate from the executable.

Very true!

If I have a binary I really don't want 10 man pages to be inside it. Let alone having to fiddle just to copy it onto a floppy or something...

Sure, but you can easily remove ELF sections with `strip`, If you look at any binary on your system right now you will see there is a .comment section that just describes the compiler, if space is a premium you might want to save these few bytes...

Code: Select all

$ objdump -h true
(...)
 22 .comment      00000318  00000000  00000000  00002180  2**0
                  CONTENTS, READONLY

$ strip -R .comment true 
$ objdump -h true

All you would have to do is strip -R manpage, and they are gone.

[dgt84]

Wow, I never knew binaries contained so many parts...
/me starts to search on material to read up on this subject!
Also, exactly what will we be storing in the binaries, as the only things mentioned so far as I can tell are man pages and perhaps icons. Where do we draw the limit as to what is stored in the executable?

EDIT: also see info pages and skeleton rc files up in the original post!

[far]

I'm sorry, but I think this is a really stupid idea.
You want to put several files together, so you invent a new fileformat? That is why we have something called "directories".
What problem are you trying to solve? That storing small files is inefficient? Then make a better filesystem, like ReiserFS.
Is the filesystem messy, because installations spread files all over it? Then put each package in its own directory.

The beauty of of unix is that file formats as well as programs tend to be simple. They solve one problem an solve it well. Making Frankenstein file formats is better left to Microsoft.

If you really must bundle several files together into one file, use tar.

[taviso]

I'm sorry, but I think this is a really stupid idea.

maybe

You want to put several files together, so you invent a new fileformat? That is why we have something called "directories".

Not a new fileformat, the ELF standard has always allowed arbritrary data to be embedded into sections, its just not used very much.

What problem are you trying to solve?

• applications provide many files that cannot or should not be edited, but are still visible on the filesystem.
• spreading packages out is inefficient, using tail packing still takes a performance hit.
• i think it would be more user-friendly if, say /bin/bash, contains everything it needs to function , its online documentation, skel files, info pages, everything..maybe you disagree, but it seems that Apple, and according to an email I was just sent QNX, agree with me.
• Yes, I do think it is untidy..why am I seeing these files that I cannot run, cant read or do anything useful with them raw, and cant edit? (manpages, info pages, gzipped docs, etc).

The beauty of of unix is that file formats as well as programs tend to be simple. They solve one problem an solve it well. Making Frankenstein file formats is better left to Microsoft.

The ELF standard allows for embedding data, I'm not making up new file formats. The tools required to manipulate, examine, and everything else are provided in a base system (binutils) (okay, they are little technical, but frontends/scripts can be easily provided)

[zhenlin]

• i think it would be more user-friendly if, say /bin/bash, contains everything it needs to function , its online documentation, skel files, info pages, everything..maybe you disagree, but it seems that Apple, and according to an email I was just sent QNX, agree with me.

The people at the original Apple, maybe. The people at Apple now are mostly people from NeXT. The NeXT solution is most compatible with Unix philosophy.

If Apple packaged bash...

Code: Select all

/Applications/bash/
    Contents/
       i686/
           Linux-GNU/
               rbash
               bash
               bashbug
       Documentation/
           Man/
               man1/
                   rbash.1
                   bash.1
                   bashbug.1
                   builtins.1
           Info/
               bashref.info
               bash.info
           README.txt
           NEWS.txt
           AUTHORS.txt
           CHANGES.txt
           COMPAT.txt
           COPYING.txt
           Y2K.txt
           FAQ.txt
           INTRO.txt

And I would prefer that layout to

Code: Select all

/bin/
    bash

Because embedding everything into the executable is a silly thing.

[far]

Not a new fileformat, the ELF standard has always allowed arbritrary data to be embedded into sections, its just not used very much.

Actually this even more evil than inventing a new format. It is changing the conventions for a format that has had well-established conventions for many years. Since far from everyone would use this, there would be much confusion.

[*] applications provide many files that cannot or should not be edited, but are still visible on the filesystem.

So you want to hide stuff from the user? Even Apple doesn't do that anymore. I think most people who are attracted by Linux are the kind of people who like to find out how things work.

Why do you hate the filesystem? The filesystem is your friend. It is there to do the dirty work. It is not there to look good. Do you remove your car engine because it is ugly and dirty? Me, I just keep the hood closed.

[*] i think it would be more user-friendly if, say /bin/bash, contains everything it needs to function , its online documentation, skel files, info pages, everything..maybe you disagree

I don't disagree, but if you are going to do something, do it right. Put all files in one directory. If that is not good enough put them in a tarchive, or better yet, use ReiserFS-style metadata.
Don't create a custom solution for a problem that has a general solution.

[taviso]

Actually this even more evil than inventing a new format. It is changing the conventions for a format that has had well-established conventions for many years. Since far from everyone would use this, there would be much confusion.

I dont understand. Im not changing anything, the standards clearly describe how to embed data into ELF executables, the tools to do so are distributed with every ELF system.

The binaries that have data embedded into them are 100% compliant, and can be extracted, manipulated, removed, embedded all on the command line with standard tools.

The files are still totally portable, you could copy them to any other ELF system that supports Linux and the target architecture, and they would still be 100% valid and compliant executables. Even the embedded data could still be manipulated/extracted/etc.

You could do the same thing to any ELF executables, from NetBSD, Linux, SCO, SysV, etc.

Nothing will break, and the original executable that the data was embedded in can be recreated.

Please explain how I am being evil!

Do you remove your car engine because it is ugly and dirty? Me, I just keep the hood closed.

Isnt that what I'm doing? hiding the files that I cant use but still need to be there?

[taviso]

Code: Select all

/bin/
    bash

Because embedding everything into the executable is a silly thing.

Okay, I accept your not convinced

But think about this, those files distributed with bash are totally useless without the man and info applications, to use man and info, you do not need to know where the manpages are, it just fetches them for you..correct?

for example, I dont have to do this:

Code: Select all

$ man /usr/share/man/man1/manfile1.gz

I can just do this:

Code: Select all

$ man manfile

Imagine you are not aware how the man system works, does it really matter where or how man gets this information from?

You cannot read the manpage without using man (unless you want to gunzip/groff it, in my system you could dd it out and do the same), so does it matter how man gets it?

also, isnt having 1 file more robust and more intuitive than having 30 to keep track of?

Anyway, thanks for your thoughts

[telex4]

Actually this even more evil than inventing a new format. It is changing the conventions for a format that has had well-established conventions for many years. Since far from everyone would use this, there would be much confusion.

I dont understand. Im not changing anything, the standards clearly describe how to embed data into ELF executables, the tools to do so are distributed with every ELF system.

...

Please explain how I am being evil!

As somebody who spends a lot of time helping and teaching newbies, and who writes documentation for newbies, I can tell you one problem you'll create, and that is that your system will make it more difficult to help people, and it will confuse people who are told to look in, for example, "/usr/share/man" and find it is empty because they are all embedded in some binary.

Now that isn't evil, it's just something that needs to be considered.

[far]

I understand that this is not a format change per se. You are trying to change the way people treat an established format. Inventing a new format would be less confusing. If I have a random executable, how do I know, without having the specialized tools, if it contains a man page or not?

But think about this, those files distributed with bash are totally useless without the man and info applications,

Useless to you maybe. I may want to read the source without going through layers of obfuscation. And you don't need man to read a manpage. The roff format is plain text. A developer should not be so arrogant as to tell the user what is useful and what is not.

to use man and info, you do not need to know where the manpages are, it just fetches them for you..correct?

Yes, so why do you need to change it? If you don't want to see the horrible horrible files, don't look in /usr/share/man/.

also, isnt having 1 file more robust and more intuitive than having 30 to keep track of?

So put them in a directory and don't delete them. How hard can it be? Also, if I think the documentation is a waste of space, I can delete it.

[kesuari]

The thing I hate most about Linux is the way files are scattered around, all the executables are in /.../bin, the man pages are in /usr/share/man, the docs are in /usr/share/doc, etc. etc. etc. All very ugly. I don't care what everyone else here says, I'll be using it on my computer :)

[Genone]

I think it might be nice as an option, but not as the new default way. So, if you want this new style you can enable it in make.conf (so portage embeds the files) and install a new tool to use the files, but unless you do so it will stay as it is now.