What limits cp, rsync argument list sizes?

[jkcunningham]

I was trying to use rsync to update a network drive from a local hard disk on a directory that has about 6500 files in it. It gives me an error message: "Argument list too long". Trying to copy the files with 'cp * <remote drive> does the same thing. If I shorten the list by doing it in pieces it works. So, it appears the error message is correct.

I have no control over the number of files in this case. Is there a way to increase the buffer length used by rsync and cp?

Regards,

-Jeff

[meowsqueak]

The kernel defines this:

http://www.ussg.iu.edu/hypermail/linux/ ... /0200.html

In include/linux/binfmts.h we can see

/*
* MAX_ARG_PAGES defines the number of pages allocated for arguments
* and envelope for the new program. 32 should suffice, this gives
* a maximum env+arg of 128kB w/4KB pages!
*/
#define MAX_ARG_PAGES 32

Not sure if it's safe to change. You can probably do something equivalent by rearranging your command and/or using find -exec in some way.

[jkcunningham]

Yeah, I sort of figured that. There are lots of ways to get around it. But it was a drag having it happen in front of some Windows weenies who, of course, pointed out that they could select that many files in Explorer and move them around without running into any limits. (I retorted that then they would spend the next two hours watching the little piece of paper drift across the frozen progress box - but it wasn't adequate).

Thanks.

Jeff

[meowsqueak]

You could always go back a directory and cp -R /path/to/dir/ instead. Then you don't have to specify all the files. However it will be recursive.

Or how about something like this (from inside the dir you're copying from):

Code: Select all

$ find . -name \* -exec cp -v \{\} /path/to/dest \;

That's quite an impressive way to copy files around

[jkcunningham]

You're right. Either method would work - and the recursion isn't a problem. But what I really want to do is update, not copy all the files. Only a subset generally need to be copied. The original comand I used which failed was:

> rsync -tuv path/to/directory/* path/to/network/directory

When I tried this, there were really only about a hundred files that needed updating. To do any of the techniques you mentioned and I've thought of would move all the files - which is a horrible waste of bandwidth and time.

-Jeff