Compressing many files with Tar

Original content: http://rizal.orami.com/2016/12/compressing-file-with-tar/

Ever tried to compress many (hundreds of thousand) files with tar? With this command:

$ tar -cf file_names.tar *.jpg

I bet you will found error message like this:

$ -bash: /bin/tar: Argument list too long

All shell have / has a limit for the command line length. UNIX / Linux / BSD system has a limit on how many bytes can be used for the command line argument and environment variables. The question is how do i find out current command line length limit? the answer is:

$ getconf ARG_MAX

or

$ echo $(( $(getconf ARG_MAX) - $(env | wc -c) ))

or in more detail and if your system have xargs command:

$ xargs --show-limits
Your environment variables take up 2658 bytes
POSIX upper limit on argument length (this system): 2092446
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 2089788
Size of command buffer we are actually using: 131072

Usually linux starting kernel 2.6.23, the ARG_MAX value is about ¹⁄₄ of stack size(ulimit -s), and linux kernel pre 2.6.23 the ARG_MAX value is about PAGE_SIZE * MAX_ARG_PAGES (4096 * 32) minus 4, which means, any number of file lists that exceed this number will break the bash command.

So, what is the answer? Quite simple:

• use find or xargs command
• use shell for / while loop
• use file manifest(need to define before)

In this case i will show you how to compress hundreds of thousand file using tar with file-manifest, tar command provide us with ‘-T’ option for this!

Steps:

• create the file-manifest find . -name '*.jpg' -print >/tmp/file.manifest
• add the file manifest content to tar tar -cvzf textfiles.tar.gz --files-from /tmp/file.manifest or tar -cv -T /tmp/file.manifest -f tarball.tar
• remove the jpg files find . -name '*.jpg' | xargs rm -v
• or tar -cvzf textfiles.tar.gz --files-from /tmp/file.manifest --remove-files

the.end.