How to Create One or Many Files in Linux

Updated

Linux command line in a terminal window

Do you need an ultra fast way to create a large number of files in Linux? Perhaps you are load testing an application and you need 10,000 or even 1,000,000 files in a matter of seconds. Two commands do it — build one master file, then split it:

dd if=/dev/zero of=masterfile bs=100000 count=1   # 100,000 bytes in one write
split -b 10 -a 10 masterfile                      # -> 10,000 files of 10 bytes each

Sizing the master file

Multiply the number of files by the size of each one, in bytes. That product is the size of the master file: 10,000 × 10 is 100,000, not 1,000,000. A 1,000,000-byte master file gives 100,000 files, not 10,000. Counted:

$ split -b 10 -a 10 masterfile          # masterfile is 1,000,000 bytes
$ find . -type f -name 'x*' | wc -l
  100000

So pick your product deliberately: 10,000 files × 10 bytes needs bs=100000 count=1. For a million 10-byte files it is 10,000,000 bytes, and you should expect the split itself to take a while — creating files is the slow part, not writing the data.

Don’t write it one byte at a time

Writing the master file with dd if=/dev/zero of=masterfile bs=1 count=1000000 asks for a million separate one-byte writes. bs is the block size and count is how many blocks, so the same file can be written in a single block. Both commands on the same machine:

$ dd if=/dev/zero of=masterfile bs=1 count=1000000
1000000+0 records in
1000000+0 records out
1000000 bytes transferred in 1.481854 secs (674830 bytes/sec)

$ dd if=/dev/zero of=master2 bs=1000000 count=1
1+0 records in
1+0 records out
1000000 bytes transferred in 0.000217 secs (4608294931 bytes/sec)

$ cmp masterfile master2 && echo "byte-identical"
byte-identical

Identical output, and the block version finished in a rounding error. Those numbers are from macOS with BSD dd; the absolute figures on your machine will differ and GNU dd is not the same binary, but the reason is arithmetic rather than platform — one syscall instead of a million. Running bash-create-files.sh gives you your own timings for this dd comparison and the touch comparison further down, along with both file counts, inside a temporary directory that it removes on exit. If you want random contents instead of zeroes, use /dev/urandom as the input.

What -a controls

-b 10 is the size of each output file in bytes. -a 10 is the length of the generated suffix, so with the default x prefix you get 11-character names:

$ ls | head -1
xaaaaaabkkp

The suffix has to be long enough to number all the pieces or split runs out of names. Ten alphabetic characters is far more than 100,000 files needs. GNU split also offers -d for numeric suffixes and --additional-suffix, neither of which exists in the BSD version.

5 ways to create a single file

touch newfile.txt                      # empty, or update the timestamp if it exists
echo "This is some text" > newfile.txt # create with one line
printf "This is some text" > newfile.txt   # same, without the trailing newline
cat > newfile.txt                      # type content, then Ctrl-D to finish
nano newfile.txt                       # or vi, emacs

cat > newfile.txt with no redirect reads your terminal until you press Ctrl-D. That is not a hang. printf differs from echo in that it adds no newline unless you write \n, and it does not vary between shells the way echo -e does.

The trap: > truncates, touch does not

This is the one that costs people real data. touch on an existing file only updates its timestamps. > empties it before you have typed anything else:

$ printf 'important data\n' > f.txt; touch f.txt; cat f.txt
important data

$ printf 'important data\n' > g.txt; : > g.txt; wc -c < g.txt
0

To make the shell refuse rather than clobber, set noclobber:

$ bash -c 'set -o noclobber; echo overwrite > h.txt'
bash: line 1: h.txt: cannot overwrite existing file

$ echo "forced" >| h.txt          # >| overrides it deliberately

That also gives you a cheap exclusive create: under noclobber, : > lockfile succeeds only if the file did not already exist, and the check-and-create happens in one open() with O_EXCL rather than as a racy test-then-write. For scratch files, mktemp is better still — it picks an unused name and creates it with restrictive permissions (600 for a file, 700 for mktemp -d) instead of whatever your umask allows:

tmp="$(mktemp)"; trap 'rm -f "$tmp"' EXIT

Parent directories

Neither touch nor > will create the directories leading to a file:

$ touch a/b/c.txt
touch: a/b/c.txt: No such file or directory

mkdir -p a/b && touch a/b/c.txt is the portable fix. On GNU systems install -D does both at once — the coreutils help text for -D reads "create all leading components of DEST except the last, or all components of --target-directory, then copy SOURCE to DEST", and the coreutils manual entry for install documents the same behaviour. Be careful reaching for it in a script that has to run anywhere: on macOS and the BSDs, -D is an entirely different option that takes an argument, documented in the BSD install(1) man page as "Specify the DESTDIR (top of the file hierarchy) that the items are installed in to". Same letter, unrelated meaning, and it will not create parent directories for you.

Creating many named files

If you want predictable names rather than split's generated ones, let brace expansion build the whole argument list so touch runs once instead of 10,000 times:

$ time touch file{1..10000}      # one process
0.434779s

$ time for i in $(seq 1 10000); do touch file$i; done   # 10,000 processes
14.591944s

Same 10,000 files either way. Past a few hundred thousand arguments you will hit the system's argument-length limit (getconf ARG_MAX), and at that point xargs, which batches for you, is the tool.

Photo of author
Sudhir P. founded HeatWare.com in 1999 and has built and operated it full-stack ever since; it is now used by more than 88,000 people. He writes here about the PostgreSQL, MySQL, Linux and DevOps work that keeps it running. Articles are rewritten only after the commands in them have actually been run, and the verification scripts are published alongside them so anyone can check the claims. Reach him at blog@heatware.net.