Dealing with Unix arguments

No, I’m not referring to your coworkers fighting over which version of Linux is the best or even what command is most helpful on the command line. Instead, I’m referring to the arguments that you use on the command line or within scripts to get Unix to do specific work for you.

The things that you want to do with arguments when working with scripts include:

• verifying that arguments were provided
• checking that the provided arguments are valid
• assigning arguments to variables when it makes them easier to use or makes the script more readable

We also need to watch out for those occasions when you might run into the “argument list too long” error.

Fortunately, the basic things that you do with arguments in scripts are easy. $1 represents the first argument, $2 the second, and so forth. But surround one or more strings with quotation marks like “hello, world” and those two strings become one argument.

$0 represents the name of the script itself. Since a script can easily be renamed, it’s smarter within a script to use $0 rather than the intended script name in error messages like “USAGE: $0 username”.

If you want to see how many arguments have been passed to a script, you can use $* or $@. These two options equate to everything you type after the name of the script and the blank that separates it from the rest of the command line. These expressions make it easy to loop through the arguments when doing that is useful.

for arg in $@
do
    echo $arg
done

If you want to focus on just the last argument, you can use the expression ${@: -1} or ${!#}. Either of these refers to just the last string on the line or the last quoted string.

The expression $# can be very useful when you need to ensure that the person running a script has provided the right number of arguments. If they don’t, you might want to display an error message or remind the user what syntax is expected.

if [$# != 3]; then
    echo “USAGE: $0 city state zip”
    exit 1
fi

Here’s a simple script that runs through all of the options for displaying arguments:

#!/bin/bash

echo first  $1
echo second $2
echo third  $3
echo last   ${@: -1}
echo last   ${!#}
echo number $#
echo all    $*
echo all    $@

echo argument list
for arg in $@
do
    echo $arg
done

Is there a limit to how many arguments you can provide to a bash script? Maybe. But, if there is, it’s beyond anything I would need to do. I’ve personally verified that running a script with 43,531 arguments doesn’t seem to phase bash – even when I then echo all of them in a single command.

But there are limits that apply to the argument list for many Unix commands. The expr command, for example. The script shown below is only going to loop so many times before the expr command can no longer accommodate the text that doubles in length every time through the loop.

#!/bin/bash

arg=$@
loop=0

while true
do
    ((loop++))
    arg="$arg$arg"
    echo $arg
    echo $loop
    expr length "$arg"
    if [ $? != 0 ]; then
        exit 1
    fi
done

Can you have too many arguments? Too long an argument string? Yes, you can run into the error shown below if you try to supply too many arguments to some Unix commands.

./expandArgs: line 12: /usr/bin/expr: Argument list too long

Even Unix places limits on the amount of information it’s prepared to handle. Fortunately, these limits are generally outside the range of what you’re likely to do. But when you need to work with tens of thousands of files, you just might find yourself staring at just that error.

The script below will generate the problem fairly quickly by doubling the length of $arg each time it passes through the loop.

#!/bin/bash

if [ $# == 0 ]; then
    arg="12345"
else
     arg=$1
fi

loop=0

while true
do
    ((loop++))
    arg="$arg$arg"
    echo $arg
    echo $loop
    expr length "$arg"
    if [ $? != 0 ]; then
        exit 1
    fi
done

Running this script as shown below, the expr command is running out of steam before sometime after it hits 121,086 characters and before it hits 242,172.

$ ./expandString Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont Virginia Washington West Virginia Wisconsin Wyoming
1
944
2
1890
3
3782
4
7566
5
15134
6
30270
7
60542
8
121086
9
./expandString: line 12: /usr/bin/expr: Argument list too long

Where does it finally fail? As the error shown above indicates, we fail on the expr command. While we would have been ok with echoing the very long line of text, determining its length crossed a system limit.

You can run into problems with excessively long argument lists with other Unix commands as well – if you try to remove, move or copy too many files with one command, if you try to create a tar file using thousands of individual file names.

To get a feel for exactly what this feels like, I created a file with 354,960 unique words. I then ran a command to create files by each file name.

$ for word in `cat words`; do    touch $word; done

What commands lead to the “Argument list too long” problem? Not ls or find. Both of these commands worked just fine.

$ ls | more
1080
10th
1st
2
2nd
…

$ find . -type f -print | more
./tiptopsome
./karyon
./minivet
…

The cat, more, rm, mv, and tar commands, on the other hand, all ran into problems and made no changes in the directory.

$ cat *
-bash: /bin/cat: Argument list too long
$ more *
-bash: /bin/more: Argument list too long
$ rm *
-bash: /bin/rm: Argument list too long
$ mv * ../dump
-bash: /bin/mv: Argument list too long
$ tar cf /tmp/files.tar *
-bash: /bin/tar: Argument list too long

The way around most of these problems is to replace the commands as shown with versions that run through a loop.

$ for file in `ls`
> do
>     cat $file
> done

The tar command can be used on a list of files that we want included in the archive, so we could do something like this to overcome the argument list problem.

$ find . -type f -print > /tmp/files
$ tar czf /tmp/files.tar.gz --files-from /tmp/files
$ ls -l /tmp/files.tar.gz
-rw-rw-r-- 1 ec2-user ec2-user 6727448 Sep  6 19:09 /tmp/files.tar.gz

There’s almost always some way to work around problems with arguments in Unix — except maybe those that deal with which Linus distribution is the best.