In this article, Mr. Make outlines a pattern for non-recursive Make systems that supports the familiar make-anywhere style common to recursive Make systems. Typing make in a directory will build everything in that directory and below.
In this article I outline a pattern for non-recursive Make systems that supports the familiar make-anywhere style common to recursive Make systems. Typing 'make' in a directory will build everything in that directory and below.
A simple recursive Make
Imagine a simple project with the following sub-directories:
/src/
/src/library/
/src/executable/
/src/ is the top level directory and is where you'd type 'make' to get a full build. Inside /src/ there's the library/ directory which build a library called lib.a from source files lib1.c and lib2.c:
/src/library/lib1.c
/src/library/lib2.c
The /src/executable/ directory builds an executable file called exec from two source files (foo.c and bar.c) and the library lib.a:
/src/executable/foo.c
/src/executable/bar.c
The classic recursive Make solution to this means putting a Makefile in each subdirectory that contains the rules to build the objects and a top-level Makefile that recurses into each sub-directory. Here's the contents of /src/Makefile
SUBDIRS = library executable
.PHONY: all
all:
for dir in $(SUBDIRS); do \
$(MAKE) -C $$dir; \
done
It enters each directory in turn and runs Make to build first the library and then the executable. The dependency between the executable and the library (i.e. the fact that the library needs to be built before the exectuable) is implicit in the order in which the directories are specified in SUBDIRS.
An improvement on that is to unwind the loop inside the rule for all, create separate rules for each sub-directory and then explicitely specify the dependency between executable and library.
Here's how to do that using phony targets for each directory:
SUBDIRS = library executable
.PHONY: $(SUBDIRS)
$(SUBDIRS):
$(MAKE) -C $@
.PHONY: all
all: $(SUBDIRS)
executable: library
That's much clearer, but it's still recursive with separate Make invocations for each sub-directory.
A flexible non-recursive Make system
When moving to non-recursive Make the ideal top-level Makefile would look like this:
SUBDIRS = library executable
include $(addsuffix /Makefile,$(SUBDIRS))
That simply says to include the Makefile in each sub-directory. The trick is to make that work! Before showing you how I make that actually work here are the skeletons of the contents of the Makefiles in the library and executable sub-directories.
# /src/library/Makefile
include root.mak
include top.mak
SRCS := lib1.c lib2.c
BINARY := lib
BINARY_EXT := $(_LIBEXT)
include bottom.mak
and
# /src/executable/Makefile
include root.mak
include top.mak
SRCS := foo.c foo.c
BINARY := exec
BINARY_EXT := $(_EXEEXT)
include bottom.mak
Each of those Makefiles specifies the source files to be built (in the SRCS macro), the name of the final linked binary (in the BINARY macro) and the type of the binary (using the BINARY_EXT macro which is set from special macros _LIBEXT and _EXEEXT).
To actually make this work they both include common Makefiles root.mak, top.mak and bottom.mak which are located in the /src/ directory.
Since the .mak included Makefiles are not in the sub-directories GNU Make needs to go
looking for them. The simplest way to do that is to use the [t]-I[/tt] command-line option to GNU Make that adds a directory to the include search path. To find the .mak files in the /src you can do:
make -I /src
It's unfortunate to ask a user to add anything to the Make command-line so I use a simple method of automatically walking up the source tree to find the .mak files. Here's the actual Makefile for /src/library:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
include $(_ROOT)/top.mak
SRCS := lib1.c lib2.c
BINARY := lib
BINARY_EXT := $(_LIBEXT)
include $(_ROOT)/bottom.mak
The _find function walks up a directory tree starting from the directory in $1 looking for the file named $2. The actual find is achieved by calling the _walk function which walks up the tree finding every instead of the file $2 in each of the successively shorter paths from $1.
The block of code at the start of the Makefile finds the location of root.mak (which is in the same directory as top.mak and bottom.mak (i.e. /src/) and saves that directory in _ROOT.
Subsequently, the Makefile can use $(_ROOT)/ to include the root.mak, top.mak and bottom.mak Makefiles without any need to type anything other than make.
Here's the contents of the first included Makefile (root.mak):
_push = $(eval _save$1 := $(MAKEFILE_LIST))
_pop = $(eval MAKEFILE_LIST := $(_save$1))
_INCLUDE = $(call _push,$1)$(eval include $(_ROOT)/$1/Makefile)$(call _pop,$1)
DEPENDS_ON = $(call _INCLUDE,$1)
DEPENDS_ON_NO_BUILD = $(eval _NO_RULES := T)$(call _INCLUDE,$1)$(eval _NO_RULES :=)
For the moment we'll ignore its contents and return to what these functions are used for when we look at dependencies between modules. The real work begins with top.mak:
_OUTTOP ?= /tmp/out
.PHONY: all
all:
_MAKEFILES := $(filter %/Makefile,$(MAKEFILE_LIST))
_INCLUDED_FROM
:= $(patsubst $(_ROOT)/%,%,$(if $(_MAKEFILES),$(patsubst
%/Makefile,%,$(word $(words $(_MAKEFILES)),$(_MAKEFILES)))))
ifeq ($(_INCLUDED_FROM),)
_MODULE := $(patsubst $(_ROOT)/%,%,$(CURDIR))
else
_MODULE := $(_INCLUDED_FROM)
endif
_MODULE_PATH := $(_ROOT)/$(_MODULE)
_MODULE_NAME := $(subst /,_,$(_MODULE))
$(_MODULE_NAME)_OUTPUT := $(_OUTTOP)/$(_MODULE)
_OBJEXT := .o
_LIBEXT := .a
_EXEEXT :=
The _OUTTOP macro defines the top-level directory into which all binary output (object files etc.) will be placed. It's defined with a ?= so that it can be overriden on the command-line, and I've given it a default vaule of /tmp/out/.
Next top.mak sets up the default target for Make as the classic all. Here is has no dependencies, but we'll add them later on for each module that's going to be built.
After that a number of macros end up setting the _MODULE_PATH to the full path to the module directory being built. For example, when building the library module _MODULE_PATH would be /src/library.
Setting this macro is complex because determining the module directory
has to be independent of the directory from which GNU Make was executed
(so that the library can be built from the top-level, for a 'make all',
or from the individual library directory, for an individual developer
build, or even can be included as a dependency on a different module).
The _MODULE_NAME is simple the path relative to the root of the tree with / replace by _. For the simple example the two modules have _MODULE_NAME's library and exectuable. But if library had a sub-directory containing a module called sublibrary then its _MODULE_NAME would be library_sublibrary.
The _MODULE_NAME is also used to create the special macro $(_MODULE_NAME)_OUTPUT which has a computed name based on _MODULE_NAME. So, for the library module the macro library_OUTPUT is created with the full path of the directory into which library's object files should be written. The output path is based on _OUTTOP and the relative path to the module being built. In that way the /tmp/out/ tree mirrors the source tree.
Finally, some standard definitions of extensions used on file names are set up.
Here I've shown definitions for Linux systems, but these can easily be changed for systems such as Windows that don't use .o for an object file or .a for a library.
bottom.mak uses these macros to set up the rules that will actually build the module:
$(_MODULE_NAME)_OBJS := $(addsuffix $(_OBJEXT),$(addprefix $($(_MODULE_NAME)_OUTPUT)/,$(basename $(SRCS)))) $(DEPS)
$(_MODULE_NAME)_BINARY := $($(_MODULE_NAME)_OUTPUT)/$(BINARY)$(BINARY_EXT)
ifneq ($(_NO_RULES),T)
ifneq ($($(_MODULE_NAME)_DEFINED),T)
all: $($(_MODULE_NAME)_BINARY)
.PHONY: $(_MODULE_NAME)
$(_MODULE_NAME): $($(_MODULE_NAME)_BINARY)
_IGNORE := $(shell mkdir -p $($(_MODULE_NAME)_OUTPUT))
_CLEAN := clean-$(_MODULE_NAME)
.PHONY: clean $(_CLEAN)
clean: $(_CLEAN)
$(_CLEAN):
rm -rf $($(patsubst clean-%,%,$@)_OUTPUT)
$($(_MODULE_NAME)_OUTPUT)/%.o: $(_MODULE_PATH)/%.c
@$(COMPILE.c) -o '$@' '$<'
$($(_MODULE_NAME)_OUTPUT)/$(BINARY).a: $($(_MODULE_NAME)_OBJS)
@$(AR) r '$@' $^
@ranlib '$@'
$($(_MODULE_NAME)_OUTPUT)/$(BINARY)$(_EXEEXT): $($(_MODULE_NAME)_OBJS)
@$(LINK.cpp) $^ -o'$@'
$(_MODULE_NAME)_DEFINED := T
endif
endif
The first thing bottom.mak does is set up two macros with computed names: $(_MODULE_NAME)_OBJS (which is the list of object files in the module computed from the SRCS macro by transforming the extension) and $(_MODULE_NAME)_BINARY (which is the name of the binary file created by the module; this would typically be the library or executable being built).
The $(_MODULE_NAME)_OBJS also includes any object files that are needed by the module but not built by it by including the DEPS macro. We'll see later how this is used to define a dependency between the library and executable in the example.
Next, if rules for this module have not previously been set up (controlled by the $(_MODULE_NAME)_DEFINED macro) and have not been explicitely disabled by the _NO_RULES macro the actual rules to build the module are defined.
In this example I show rules for Linux, this is where you'd change this example for another operating system.
First, all has the current binary (from $(_MODULE_NAME)_BINARY)
added as a prerequisite so that the module gets built when a full build
is done. Then there's a rule that associates the module name with
the module binrary so that it's possible to type something like make library at the top level of the build to build just the library.
Then there's a general clean rule and a module specific clean (for the library module there's a rule to just clean its objects called clean-library). Clean is implemented as a simple rm -rf since all the output is organized in a specific directory of _OUTTOP.
After that a $(shell) is used to setup up the directory into which the module's output will
go. Finally, specific rules associated the object files in this module's output directory and source files in this module's source directory.
So, with all that infrastructure in place we can finally come to the Makefile in the executable directory:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
$(call DEPENDS_ON,library)
include $(_ROOT)/top.mak
SRCS := foo.c bar.c
BINARY := exec
BINARY_EXT := $(_EXEEXT)
DEPS := $(library_BINARY)
include $(_ROOT)/bottom.mak
It looks very like the Makefile for the library, but there are a couple of differences. Since the executable needs the library the DEPS line specifies that the executable depends on the binary file created by the library. Since each module has unique macros for objects and
binaries, it's easy to define that dependency just by referring to $(library_BINARY) which will expand to the full path to the library file created by the library module.
To actually ensure that $(library_BINARY) is defined its necessary to include the Makefile from the library directory. The root.mak file provides two functions that make this trivial: DEPENDS_ON and DEPENDS_ON_NO_BUILD.
DEPENDS_ON_NO_BUILD
just sets up the macros for the specified module so that they can be used in the Makefile. If that function were used in the executable Makefile then the library (lib.a) would have to exist for the executable to build successfully. On the other hand, of DEPENDS_ON is used this ensures that the library will get built if necessary.
DEPENDS_ON_NO_BUILD
provides functionality similar to a classic recursive build which doesn't know how to build that library, but depends on it anyway. DEPENDS_ON is more flexible since without recursion you can specify a relationship and make sure that code is built.
OK, so what can we do with this?
This system provides a great deal of flexibility. Here are just a few examples that illustrate that the non-recursive Make system is just as flexible as a recursive one (and more so!):
Building everything from the top-level is a simple Make (in all these examples I do a make -n so that the commands are clearly shown):
$ cd /src
$ make -n
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
As is cleaning everything:
$ cd /src
$ make -n clean
rm -rf /tmp/out/library
rm -rf /tmp/out/executable
From the top-level directory its possible to ask for any individual module to be built or cleaned:
$ cd /src
$ make -n clean-library
rm -rf /tmp/out/library
$ make -n library
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
And if we ask that the executable module be built the library gets built at the same time because of the dependency:
$ cd /src
$ make -n executable
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
OK, so much for the top-level; if we pop down into the library module we can build or clean it just as easily:
$ cd /src/library
$ make -n clean
rm -rf /tmp/out/library
$ make -n
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
and, of course, doing this in the executable directory will build the library as well:
$ cd /src/executable
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
What sub-modules?
Suppose, that the source tree was actually:
/src/
/src/library/
/src/library/sublibrary
/src/executable/
where there's an additional sublibrary underneath the library which builds slib.a from slib1.c and slib2.c using the following Makefile:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
include $(_ROOT)/top.mak
SRCS := slib1.c slib2.c
BINARY := slib
BINARY_EXT := $(_LIBEXT)
include $(_ROOT)/bottom.mak
To define it as a dependency of library means simply adding a DEPENDS_ON call to the Makefile in the library directory:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
$(call DEPENDS_ON,library/sublibrary)
include $(_ROOT)/top.mak
SRCS := lib1.c lib2.c
BINARY := lib
BINARY_EXT := $(_LIBEXT)
include $(_ROOT)/bottom.mak
In this example I didn't add a DEPS line so the library doesn't depend on sublibrary at the object level we're simply declaring sublibrary as a sub-module of library that needs to be built if library is.
Going back and repeating the examples above we see that the sublibrary has been successfully included in the library build (and automatically in the executable build).
Here's the full build from the top, followed by a clean:
$ cd /src
$ make -n
cc -c -o '/tmp/out/library/sublibrary/slib1.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib1.c'
cc -c -o '/tmp/out/library/sublibrary/slib2.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib2.c'
ar r '/tmp/out/library/sublibrary/slib.a' /tmp/out/library/sublibrary/slib1.o /tmp/out/library/sublibrary/slib2.o
ranlib '/tmp/out/library/sublibrary/slib.a'
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
$ make -n clean
rm -rf /tmp/out/library/sublibrary
rm -rf /tmp/out/library
rm -rf /tmp/out/executable
Asking for the sublibrary to be built:
$ cd /src
$ make -n clean-library_sublibrary
rm -rf /tmp/out/library/sublibrary
$ make -n library_sublibrary
cc -c -o '/tmp/out/library/sublibrary/slib1.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib1.c'
cc -c -o '/tmp/out/library/sublibrary/slib2.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib2.c'
ar r '/tmp/out/library/sublibrary/slib.a' /tmp/out/library/sublibrary/slib1.o /tmp/out/library/sublibrary/slib2.o
ranlib '/tmp/out/library/sublibrary/slib.a'
And if we ask that the executable module be built the library gets built at the same time because of the dependency (and also the sublibrary):
$ cd /src/executable
$ make -n executable
cc -c -o '/tmp/out/library/sublibrary/slib1.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib1.c'
cc -c -o '/tmp/out/library/sublibrary/slib2.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib2.c'
ar r '/tmp/out/library/sublibrary/slib.a' /tmp/out/library/sublibrary/slib1.o /tmp/out/library/sublibrary/slib2.o
ranlib '/tmp/out/library/sublibrary/slib.a'
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
Conclusion
Although nothing like as simple as recursive Make, this non-recursive system is
very flexible (allowing dependencies between individual binary files across modules; that's not possible with recursive Make) without losing the 'go to any directory and type Make' that engineers are familiar with.
Let me know if you use it, or improve on it.
