The Trouble with $(wildcard)

$(wildcard) explained can be used anywhere in a Makefile or rule to get a list of files matching one or more glob style patterns.

For example, $(wildcard *.foo) returns a list of files ending in .foo. Recall that a list is a string where list elements are separated by spaces, so $(wildcard *.foo) might return a.foo b.foo c.foo. (If a filename itself contains a space then the returned list may appear incorrect because there's no way to spot the different between the list separator (a space) and then space in a filename.)

$(wildcard) can be called with a list of patterns, so $(wildcard *.foo *.bar) returns all the files ending in .foo or .bar. The $(wildcard) operator supports the following 'globbing' operators: * (match 0 or more characters), ? (match 1 character), and [...] (matches characters [123] or a range of characters [a-z]).

Another useful feature of $(wildcard) is that if the filename passed to it does not contain a pattern then that file is simply checked for existence. If the file exists then its name is returned, otherwise $(wildcard) returns an empty string. Thus $(wildcard) can be combined with $(if) to create an if-exists function:

if-exists = $(if ($wildcard $1),$2,$3)

Why This Happens

The unexpected, and apparently inconsistent, results above are because GNU Make contains its own cache of directory entries. $(wildcard) reads from that cache (and not directly from disk like ls) to get its results. When that cache is filled is vital to understanding the results the $(wildcard) will return.

GNU Make only fills the cache when it has too (for example, when it needs to read the directory entries to satisfy a $(wildcard) or other globbing request). If you know that GNU Make only fills the cache when needed then it's possible to explain the results seen above.

In the first example, GNU Make fills the cache for the current working directory when it starts. Hence the file b.foo doesn't appear in the output of $(wildcard) because it wasn't present when the cache was filled. In the second example, GNU Make hasn't filled the cache with entries from subdir until they were needed. The entries were first needed for the $(wildcard) which is performed after subdir/b.foo is created and hence subdir/b.foo does appear in the $(wildcard) output. In the final example, the $(warning) happens at the start of the Makefile which filled the cache (because it did a $(wildcard)) and hence subdir/b.foo was missing from the output of $(wildcard) for the duration of that make.

Predicting when the cache will be filled is very hard. $(wildcard) will fill the cache, but so will use of a globbing operator like * in the target or prerequisite list of a rule. Consider this example Makefile that builds two files (subdir/b.foo and subdir/c.foo) and does a couple of $(wildcard)'s:

DIRECTORY=subdir/

.PHONY: all
all: $(DIRECTORY)b.foo
@echo $(WILDCARD_LIST)
@echo $(LS_LIST)

$(DIRECTORY)b.foo: $(DIRECTORY)c.foo
@touch $@
@echo $(WILDCARD_LIST)
@echo $(LS_LIST)

$(DIRECTORY)c.foo:
@touch $@

The output may surprise you:

wildcard returned 'subdir/a.foo subdir/c.foo'
ls returned 'subdir/a.foo subdir/c.foo'
wildcard returned 'subdir/a.foo subdir/c.foo'
ls returned 'subdir/a.foo subdir/b.foo subdir/c.foo'

Even though the first $(wildcard) is being done in the rule that makes subdir/b.foo and after the touch that created subdir/b.foo there's no mention of subdir/b.foo in the output. Nor is there mention of subdir/b.foo in the output of the ls.

That's because the complete block of commands is expanded into its final form before any of the lines in the rule are run. So the $(wildcard) and $(shell ls) are done before the touch has run.

The output of $(wildcard) is even more unpredictable if the make is run in parallel with the -j switch. In that case the exact order in which the rules are run is not predictable and hence the output of $(wildcard) can be even less predictable.

Recommendations

Don't use $(wildcard) in a rule; only use $(wildcard) in the Makefile at parsing time (i.e. before any rules start running). If you restrict use of $(wildcard) to parsing time you can be assured of consistent results: $(wildcard) will show the state of the filesystem before make was run.