| View previous topic :: View next topic |
| Author |
Message |
Freakazoid
Tux's lil' helper
Joined: 13 Apr 2004
Posts: 95
|
 Posted: Fri Dec 23, 2011 9:40 pm Post subject: Changing the on disk sorting of files? |
 |
|
first: *not* looking for info on how to display a sorted list with LS but how a directory is presented to a requesting program. Namely, a badly written requesting program.
The program: Minecraft server, more specifically, ModLoaderMP, it doesn't do it's own internal sorting and i'm having problems getting <b> to load after <a> and everything i've tried to re-order the list fails miserably.
Tried moving the files around and back. Tried deleting the files and making a new copy. Tried one or two other things and it's always in the same order.
So until the author of ModloaderMP fixes things i'm stuck trying to fix it another way.
Filesystem == ext3 if that matters. |
|
| Back to top |
|
 |
psdasilva
Apprentice
Joined: 03 Sep 2005
Posts: 211
|
| Posted: Sat Dec 24, 2011 4:57 am Post subject: |
|
|
That depends on the filesystem you use. I am using reiserfs and it always presents the filenames sorted.
If your program(s) are reading filenames thru stdin you can use sort and pipe its output.
HTH |
|
| Back to top |
|
|
Freakazoid
Tux's lil' helper
Joined: 13 Apr 2004
Posts: 95
|
| Posted: Sat Dec 24, 2011 3:59 pm Post subject: |
|
|
| psdasilva wrote: | That depends on the filesystem you use. I am using reiserfs and it always presents the filenames sorted.
If your program(s) are reading filenames thru stdin you can use sort and pipe its output.
HTH |
note the phrase "Filesystem == ext3"......
anyway, it's got it's own method of reading the files in, hence me asking how to re-order them on disk. |
|
| Back to top |
|
|
Ant P.
Veteran
Joined: 18 Apr 2009
Posts: 1295
Location: UK
|
| Posted: Sat Dec 24, 2011 6:37 pm Post subject: |
|
|
Try this.
_________________
sig temporarily out of order |
|
| Back to top |
|
|
Freakazoid
Tux's lil' helper
Joined: 13 Apr 2004
Posts: 95
|
| Posted: Sat Dec 24, 2011 7:06 pm Post subject: |
|
|
Nope, hence me asking for how to sort it on disk. That's for bukkit... bukkit's sorta broken for forge mods on the whole. Bukkit also takes a *long* time to port for new versions of the server, hence me just running vanilla. |
|
| Back to top |
|
|
eccerr0r
Advocate
Joined: 01 Jul 2004
Posts: 2354
Location: USA
|
| Posted: Tue Dec 27, 2011 9:09 pm Post subject: |
|
|
The actual orderring of files on disk is dependent on the filesystem, and it tends to be first added first out, and if files are removed the entries are reused. In advanced systems like reiserfs, the ordering is somewhat more strange as it tries to optimize access (tree form instead of array).
I'm sorry you'll really have to wait for application fix. As a workaround, technically it should be possible to use ext3fs and insert files into a FRESH directory one by one in NAME order (use a shell that has * expansion which sorts by name (bash is fine), don't just copy -r a directory name).
Such as
| Code: | mkdir newdir
cd olddir
cp * ../newdir |
This will NOT work on reiserfs. I'm not 100% sure if this will guarantee results on ext3fs. This will also NOT copy recursively. Any subdirectories will have to be done manually.
Try "ls -f" to see what the disk order is.
One filesystem this should work fine on is vfat/msdos.
Simply doing
| Code: | | cp -r olddir newdir |
will NOT work. This will read disk order and the new directory will be a collapsed but same order directory.
You'll have to do this each time you add a new file as new files will tend to be tacked on. Also watch out for EDITING files, most editors create temporary files which will once again muck with directory orderring.
_________________
Core2Quad 9550S/4GB/4x500G RAID5/RadeonHD 5770
What the heck am I advocating? |
|
| Back to top |
|
|
Freakazoid
Tux's lil' helper
Joined: 13 Apr 2004
Posts: 95
|
| Posted: Tue Dec 27, 2011 10:59 pm Post subject: |
|
|
| eccerr0r wrote: | The actual orderring of files on disk is dependent on the filesystem, and it tends to be first added first out, and if files are removed the entries are reused. In advanced systems like reiserfs, the ordering is somewhat more strange as it tries to optimize access (tree form instead of array).
I'm sorry you'll really have to wait for application fix. As a workaround, technically it should be possible to use ext3fs and insert files into a FRESH directory one by one in NAME order (use a shell that has * expansion which sorts by name (bash is fine), don't just copy -r a directory name).
Such as
| Code: | mkdir newdir
cd olddir
cp * ../newdir |
This will NOT work on reiserfs. I'm not 100% sure if this will guarantee results on ext3fs. This will also NOT copy recursively. Any subdirectories will have to be done manually.
Try "ls -f" to see what the disk order is.
One filesystem this should work fine on is vfat/msdos.
Simply doing
| Code: | | cp -r olddir newdir |
will NOT work. This will read disk order and the new directory will be a collapsed but same order directory.
You'll have to do this each time you add a new file as new files will tend to be tacked on. Also watch out for EDITING files, most editors create temporary files which will once again muck with directory orderring. |
Bother.. I did try the stuff files in the one at a time method but was still greeted by the same lack of ordering.
Almost wants me to look at FUSE to have a meta-filesystem mounted in the right place that sorts things. Not sure if i'm up to the task of writing such though.
Thanks for the insight though.... looks like i'll have to continue pestering mr. not my problem author of ModloaderMP |
|
| Back to top |
|
|
palmer
Guru
Joined: 17 Nov 2004
Posts: 318
Location: Cambridge, MA
|
| Posted: Wed Dec 28, 2011 12:36 am Post subject: |
|
|
EDIT: Looks like the code tags munged my code, they're online
liboverride.c: http://paste.pocoo.org/show/526660/
run.sh: http://paste.pocoo.org/show/526663/
dirtest.c: http://paste.pocoo.org/show/526664/
Back to the original message:
I think the best way to go about this is to override whatever library functions are being used to read directory entries with some functions of your own that reads the directory entries in sorted order. Assuming the code uses the normal opendir()/readdir()/closedir() stuff, the following should work:
| Code: | $ cat liboverride.c
#include <sys/types.h>
#include <dirent.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>
#include <syscall.h>
#define BUF_SIZE 128
struct info
{
int dirent_cur;
int dirent_count;
struct dirent *dirents;
};
struct linux_dirent {
long d_ino;
off_t d_off;
unsigned short d_reclen;
char d_name[];
};
static int dirent_compar(const void *a_uncast, const void *b_uncast)
{
struct dirent *a;
struct dirent *b;
a = (struct dirent *)a_uncast;
b = (struct dirent *)b_uncast;
return strcmp(a->d_name, b->d_name);
}
DIR *opendir(const char *name)
{
struct info *info;
int fd, nread;
char buf[BUF_SIZE];
struct linux_dirent *d;
int bpos;
char d_type;
info = malloc(sizeof(*info));
if (info == NULL)
return NULL;
fd = open(name, 0);
info->dirent_count = 0;
for ( ; ; )
{
nread = syscall(SYS_getdents, fd, buf, BUF_SIZE);
if (nread == -1)
handle_error("getdents");
if (nread == 0)
break;
for (bpos = 0; bpos < nread;) {
d = (struct linux_dirent *) (buf + bpos);
bpos += d->d_reclen;
info->dirent_count++;
}
}
close(fd);
info->dirents = malloc(sizeof(*info->dirents) * info->dirent_count);
fd = open(name, 0);
info->dirent_cur = 0;
for ( ; ; )
{
nread = syscall(SYS_getdents, fd, buf, BUF_SIZE);
if (nread == -1)
handle_error("getdents");
if (nread == 0)
break;
for (bpos = 0; bpos < nread;) {
d = (struct linux_dirent *) (buf + bpos);
info->dirents[info->dirent_cur].d_ino = d->d_ino;
info->dirents[info->dirent_cur].d_off = d->d_off;
info->dirents[info->dirent_cur].d_reclen = d->d_reclen;
info->dirents[info->dirent_cur].d_type = 0;
strncpy(info->dirents[info->dirent_cur].d_name,
d->d_name, 256);
bpos += d->d_reclen;
info->dirent_cur++;
}
}
close(fd);
info->dirent_cur = 0;
qsort(info->dirents, info->dirent_count, sizeof(*info->dirents),
&dirent_compar);
return (DIR *)info;
}
struct dirent *readdir(DIR *dir)
{
struct info *info;
info = (struct info *)dir;
if (info->dirent_cur >= info->dirent_count)
return NULL;
return info->dirents + (info->dirent_cur++);
}
int closedir(DIR *dir)
{
struct info *info;
info = (struct info *)dir;
free(info->dirents);
free(info);
return 0;
} |
It works with my little test program:
| Code: | $ cat dirtest.c
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <dirent.h>
int main(int argc, char **argv)
{
DIR *dip;
struct dirent *dit;
dip = opendir(".");
while ((dit = readdir(dip)) != NULL)
printf("dirname: '%s'\n", dit->d_name);
closedir(dip);
return 0;
} |
You can test it with the following. You'll notice that the first run of the program prints in arbitrary order, but the second prints in sorted order
| Code: | #!/bin/bash
gcc dirtest.c -o dirtest
./dirtest
gcc -shared -fPIC -Wl,-soname,liboverride.so -o liboverride.so liboverride.c
export LD_PRELOAD=liboverride.so
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
./dirtest |
|
|
| Back to top |
|
|
eccerr0r
Advocate
Joined: 01 Jul 2004
Posts: 2354
Location: USA
|
| Posted: Wed Dec 28, 2011 10:33 pm Post subject: |
|
|
Nice, thank you for writing this...
Was hoping to not have to go to such extreme to fix a software bug, I guess a lot of application software bugs can be fixed in OS or hardware... I just hope that, as mentioned, the application uses the usual functions to access directory contents. I suspect these are "low" enough that they should work for most apps... |
|
| Back to top |
|
|
palmer
Guru
Joined: 17 Nov 2004
Posts: 318
Location: Cambridge, MA
|
| Posted: Thu Dec 29, 2011 12:03 am Post subject: |
|
|
| eccerr0r wrote: | Nice, thank you for writing this...
Was hoping to not have to go to such extreme to fix a software bug, I guess a lot of application software bugs can be fixed in OS or hardware... I just hope that, as mentioned, the application uses the usual functions to access directory contents. I suspect these are "low" enough that they should work for most apps... |
opendir()/readdir()/closedir() are what I use for listing directories, and as far as I'm concerned they're pretty standard, but there are always many ways to do things (as you can see, I've implemented those library functions using system calls, so it's not inconceivable that other code does as well).
dev-util/ltrace claims it can trace library calls, and it seems to work reasonably in my little test code. That would probably help more than strace for figuring out what the code uses to get directory contents. |
|
| Back to top |
|
|
Freakazoid
Tux's lil' helper
Joined: 13 Apr 2004
Posts: 95
|
| Posted: Sat Dec 31, 2011 8:10 pm Post subject: |
|
|
| eccerr0r wrote: | Nice, thank you for writing this...
Was hoping to not have to go to such extreme to fix a software bug, I guess a lot of application software bugs can be fixed in OS or hardware... I just hope that, as mentioned, the application uses the usual functions to access directory contents. I suspect these are "low" enough that they should work for most apps... |
Not sure if you were referring to ModLoaderMP or not but he *finally* relented to fixing it on his end. Not sure if the page on MC forums has been updated or not yet. |
|
| Back to top |
|
|
Ant P.
Veteran
Joined: 18 Apr 2009
Posts: 1295
Location: UK
|
| Posted: Sun Jan 01, 2012 12:51 am Post subject: |
|
|
If KDE can fix their software to behave on a POSIX filesystem, so can he!
_________________
sig temporarily out of order |
|
| Back to top |
|
|
eccerr0r
Advocate
Joined: 01 Jul 2004
Posts: 2354
Location: USA
|
| Posted: Sun Jan 01, 2012 4:13 pm Post subject: |
|
|
At least it's faster listing the contents of a directory unsorted if you don't need the sort...
Suppose that's why this was moved into userspace to begin with.
_________________
Core2Quad 9550S/4GB/4x500G RAID5/RadeonHD 5770
What the heck am I advocating? |
|
| Back to top |
|
|
|
Other Things Gentoo
