Makefile Optimization: $(shell) and := go Together

$(shell) explained
$(shell) is GNU Make's equivalent of the back tick (`) operator in the shell. $(shell) executes a command, flattens the result (i.e. turns all line endings into spaces) and returns the resulting string.

For example, if you want to get the output of the date command into a variable called NOW you'd write

NOW = $(shell date)

Or if you want to count the number of files in the current directory and get that number into FILE_COUNT do

FILE_COUNT = $(shell ls | wc -l )

Note that since $(shell) flattens output you can get the names of all the files in the current directory into a macro with:

FILES = $(shell ls)

The newline between files is replaced with a single space making FILES a space-separated list of filenames.

You'll commonly see in Makefiles an execution of the pwd command to get the current working directory into a variable (in this case CWD):

CWD = $(shell pwd)

It's this command we'll take a look at later on when considering how to optimize an example Makefile that wastes time getting the working directory over and over again.

The difference between = and := — 99% of the time you'll see macro definitions in Makefiles that use the = form:

FOO = foo
 BAR = bar
 FOOBAR = $(FOO) $(BAR)
 
 all: $(FOOBAR)
 $(FOOBAR):
 	@echo $@ $(FOOBAR)
 FOO = fooey
 BAR = barney

In the example above macros FOO, BAR and FOOBAR are “recursively expanded” macros. That means that when we ask for the value of a macro any macros that it references are expanded at that point. For example, if we ask for the value of $(FOOBAR) it will get the value of $(FOO) and $(BAR) put them together with the space in between and return foo bar. Expansion through as many levels of macros as necessary is done only when the variable is used.

In the Makefile above this has the interesting side effect that FOOBAR
appears to have two different values. Running it prints out:

foo fooey barney
 bar fooey barney

The value of FOOBAR is used to define the list of prerequisites to the all rule and is expanded as foo bar, the same thing happens for the next rule which defines rules for foo and bar.

But when the rules are run, the value of FOOBAR as used in the echo comes out as fooey barney. (You can verify that the value of FOOBAR was foo bar when the rules were defined by looking at the value of $@---the target being built---when the rules are run).

There are two cases to remember:

1. When a rule is being defined in a Makefile, macros will evaluate to their value at that point in the Makefile.
2. Macros used in rule bodies (i.e. in the commands) have their final value: whatever value the macro had at the end of the Makefile.

If we change the definition of FOOBAR to use a := instead of = running the Makefile gives a very different result.

foo foo bar
 bar foo bar 

Now FOOBAR has the same value everywhere. That because := forces the right-hand side of the definition to be expanded at that moment. Rather than storing $(FOO) $(BAR) as the definition of FOOBAR, GNU Make stores the expansion of $(FOO) $(BAR) which at that point is foo bar. The fact that FOO and BAR are redefined later in the Makefile is irrelevant, FOOBAR has already been expanded and set to a fixed string. GNU Make refers to variables defined in this way as “simply expanded.”

Once a macro has become simply expanded it remains that way unless it is redefined using the = operator. This means that if you append to a simply expanded macro the text being appended is expanded before the append.

For example,

FOO=foo
 BAR=bar
 BAZ=baz
 FOOBAR := $(FOO) $(BAR)
 FOOBAR += $(BAZ)
 BAZ=bazzy

results in FOOBAR being foo bar baz. If = had been used instead of := then when $(BAZ) was appended it would not have been expanded and the resulting FOOBAR would have been foo baz bazzy.
Take a look at this example Makefile.

CWD = $(shell pwd)
 SRC_DIR=$(CWD)/src/
 OBJ_DIR=$(CWD)/obj/
 OBJS = $(OBJ_DIR)foo.o $(OBJ_DIR)bar.o $(OBJ_DIR)baz.o
 $(OBJ_DIR)%.o: $(SRC_DIR)%.c ; @echo Make $@ from $<
 all: $(OBJS)
 	@echo $? $(OBJS)
 

It gets the current working directory into CWD, defines a source and object directory as sub-directories of the CWD, defines a set of objects (foo.o, bar.o and baz.o) to be built in the OBJ_DIR, sets up a pattern rule showing how to build a .o from a .c and finally states that by default the Makefile should build all the objects and print out a list of those that were out of date ($? is the list of prerequisites of a rule that were out of date) and a full list of objects.

You might be surprised to learn that this Makefile ends up making eight shell invocations just to get the CWD value. Imagine how many times GNU Make would make costly calls to the shell in a real Makefile with hundreds or thousands of objects!

There are so many calls to $(shell) because the Makefile uses recursively expanded macros: i.e. macros whose value is determined when the macro is used and not at definition time. OBJS references OBJ_DIR three times which references CWD each time; every time OBJS is referenced there are three calls to $(shell pwd). Any other reference to SRC_DIR or OBJ_DIR (e.g. the pattern rule definition) results in another $(shell pwd).

But there's a quick fix for this. Just change the definition of CWD to simply expanded by inserting a : to turn = into :=. Since the working directory doesn't change during the Make we can safely get it once:

CWD := $(shell pwd)

Now there's a single call out to the shell to get the working directory. In a real Makefile this could be a large time saver.

Since it can be hard to follow through a Makefile to see everywhere a macro is used there's a simple trick that will cause Make to print out the exact line at which a macro is expanded. If we insert $(warning Call to shell) in the definition of CWD so that its definition is

CWD = $(warning Call to shell)$(shell pwd)

then we get the following output when we run Make:

Makefile:8: Call to shell
 Makefile:8: Call to shell
 Makefile:10: Call to shell
 Makefile:10: Call to shell
 Makefile:10: Call to shell
 Make /somedir/obj/foo.o from /somedir/src/foo.c
 Make /somedir/obj/bar.o from /somedir/src/bar.c
 Make /somedir/obj/baz.o from /somedir/src/baz.c
 Makefile:11: Call to shell
 Makefile:11: Call to shell
 Makefile:11: Call to shell
 /somedir/obj/foo.o /somedir/obj/bar.o /somedir/obj/baz.o /somedir/obj/foo.o
 /somedir/obj/bar.o /somedir/obj/baz.o
 

The $(warning) doesn't change the value of CWD, but it does output a message to STDERR. From the output you can see that eight calls to the shell and which lines in the Makefile caused them.

If CWD is defined using := the $(warning) trick verifies that CWD is expanded only once:

Makefile:1: Call to shell
 Make /somedir/obj/foo.o from /somedir/src/foo.c
 Make /somedir/obj/bar.o from /somedir/src/bar.c
 Make /somedir/obj/baz.o from /somedir/src/baz.c
 /somedir/obj/foo.o /somedir/obj/bar.o /somedir/obj/baz.o /somedir/obj/foo.o
 /somedir/obj/bar.o /somedir/obj/baz.o
 

Quick Recipe
A quick way to find out if you are using the expensive combination of = and $(shell) in your Makefile is to run the command:

grep -n \$\(shell Makefile | grep -v :=

It'll print out the line number and details of every line in Makefile that contains a $(shell) and doesn't contain a :=. Take a look at what it finds to see if your Makefile can be optimized.