In this article I present a simple technique that causes GNU Make to create a XML file containing a "bill of materials" or BoM. The BoM contains the names of all the files built by the Makefile and is nested to show the prerequisites of target.
A difficult question to answer with standard GNU Make output is: "What got built and why?"
In this article I present a simple technique that causes GNU Make to create a XML file containing a "bill of materials" or BoM. The BoM contains the names of all the files built by the Makefile and is nested to show the prerequisites of target.
An Example
Let's start with an example Makefile and its related BoM and work backwards to how the BoM XML file was generated. Take for example, the following simple Makefile:
all: foo bar
@echo Making $@
foo: baz
@echo Making $@
bar:
@echo Making $@
baz:
@echo Making $@
It makes all from foo and bar. In turn foo is made from baz. Running this in GNU Make gives the following output:
Making baz
Making foo
Making bar
Making all
From the output it's impossible to tell the actual tree-ordering of the build, or which files depend on which. In this case, the Makefile is very small and it's relatively easy to trace by hand; in any real Makefile hand tracing is almost impossible.
Later I'll show you the code that can translate the above Makefile into an XML document like this:
<rule target="all">
<prereq>
<rule target="foo">
<prereq>
<rule target="baz" />
</prereq>
</rule>
<rule target="bar" />
</prereq>
</rule>
In that document each rule run by the Makefile has a <rule> tag added with a target attribute giving the name of the target that the rule built.
If the rule had any prerequisites then within the <rule>/</rule> pair there's a list of prerequisite rules enclosed in <prereq>/</prereq>.
In the example above you can see the structure of the Makefile reflected in the nesting of the tags. Loading that XML document into an XML editor (or simply into a web browser like Firefox) allows you to expand and contract the tree at will to explore the structure of the Makefile.
How it works
To create the output above the example is Makefile is modified to include a special 'bom' Makefile using the standard include bom method. With that included its possible to generate the XML output by running GNU Make with a command line such as make bom-all.
The bom-all instructs GNU Make to build the build of materials starting with the all target. It's just as if you'd typed make all but now an XML document will be created.
By default the XML document has the same name as the Makefile with .xml appended. If the example Makefile above was in example.mk then the XML document created will be called example.mk.xml.
Here's the contents of the bom Makefile to include (I've added line numbers to make it easy to refer to parts of this Makefile below):
1 PARENT_MAKEFILE := $(word $(words $(MAKEFILE_LIST)),x $(MAKEFILE_LIST))
2 bom-file := $(PARENT_MAKEFILE).xml
3
4 bom-old-shell := $(SHELL)
5 SHELL = $(bom-run)$(bom-old-shell)
6
7 bom-%: %
8 @$(shell rm -f $(bom-file))$(call bom-dump,$*)
9
10 bom-write = $(shell echo '$1' >> $(bom-file))
11
bom-dump = $(if $(bom-prereq-$1),$(call bom-write,<rule
target="$1">)$(call bom-write,<prereq>)$(foreach
p,$(bom-prereq-$1),$(call bom-dump,$p))$(call
bom-write,</prereq>)$(call bom-write,</rule>),$(call
bom-write,<rule target="$1" />))
12
13 bom-run = $(if $@,$(eval bom-prereq-$@ := $^))
The first two lines figure out the correct name for the XML file. This is done by extracting the name of the Makefile that included bom into PARENT_MAKEFILE, appending .xml and storing the resulting name in bom-file.
Lines 3 and 4 use a trick I've talked about a number of times: my SHELL hack. Briefly, GNU Make will expand the value of $(SHELL) for every rule that's run in the Makefile. And at the time that $(SHELL) is expanded the per-rule automatic variables (such as $@) have already been set. Thus by modifying SHELL it's possible to perform some task for every rule in the Makefile as it runs.
Line 3 stores the original value of SHELL in bom-old-shell using an immediate assignment (:=) and then redefines SHELL to be the expansion of $(bom-run) (which does all the work for this article) and the original shell. Since $(bom-run) actually expands to an empty string the affect is that bom-run is expanded for each rule in the Makefile, but the actual shell used is unaffected.
bom-run itself is defined on line 13. All it does is use $(eval) to store the relationship between the current target being built (the $(if) ensures that $@ is defined) and its prerequisites.
For example, when building foo a call will be made to bom-run with $@ set to foo and $^ (the list of all prerequisites) set to baz. bom-run will set the value of bom-prereq-foo to baz. Later the values of these bom-prereq-X variables is used to dump out the XML tree.
Line 5 shows the definition of the pattern rule that handles the bom-% target. Since its prerequisite is % this has the affect of building the target matching the % and then building bom-%. In the example above, running make bom-all matches against this pattern rule to build all and then run the commands associated with the bom-% with %* set to all.
bom-%'s commands first delete the bom-file and then recursively dump out the XML starting from $*. In the example, above where the user did make bom-all the bom-% commands calls bom-dump with the argument all.
Line 11 shows the definition of bom-dump. It's fairly routine, it uses a helper function (bom-write) to echo fragments of XML to the bom-file
and calls itself for each of the targets in the prerequisites of the
target it is dumping. Prerequisites are extracted from the bom-prereq-X variables created by bom-run.
Gotchas
There are a few gotchas with this technique.
Firstly, it can end up producing enormous amounts of output. That's because it
will print the entire tree above any target. If a target appears multiple times in the tree then a large tree can be repeated many times in the output. Even for small projects this can make the dump time for the XML very long.
To workaround that it's possible to change the definition of bom-dump to just dump the prerequisite information once for each target. This has much faster than the original above, and could be processed by a script to understand the structure of the Make.
bom-%: %
@$(shell rm -f $(bom-file))$(call bom-write,<bom>)$(call bom-dump,$*)$(call bom-write,</bom>)
bom-write = $(shell echo '$1' >> $(bom-file))
bom-dump
= $(if $(bom-prereq-$1),$(call bom-write,<rule
target="$1">)$(call bom-write,<prereq>)$(foreach
p,$(bom-prereq-$1),$(call bom-write,<rule target="$p" />))$(call
bom-write,</prereq>)$(call bom-write,</rule>),$(call
bom-write,<rule target="$1" />))$(foreach
p,$(bom-prereq-$1),$(call bom-dump,$p))$(eval bom-prereq-$1 := )
For the example Makefile the XML document now looks like:
<bom>
<rule target="all">
<prereq>
<rule target="foo" />
<rule target="bar" />
</prereq>
</rule>
<rule target="foo">
<prereq>
<rule target="baz" />
</prereq>
</rule>
<rule target="baz" />
<rule target="bar" />
</bom>
Another gotcha is that if the Makefile includes rules with no commands those rules will cause a break in the tree output by this technique. For example, if the example Makefile is:
all: foo bar
@echo Making $@
foo: baz
bar:
@echo Making $@
baz:
@echo Making $@
The resulting XML will not mention baz at all because since the rule for foo didn't have any commands SHELL was not expanded and hence the hack didn't work. Here's the XML in that case:
<bom>
<rule target="all">
<prereq>
<rule target="foo" />
<rule target="bar" />
</prereq>
</rule>
<rule target="foo" />
<rule target="bar" />
</bom>
To workaround this the foo: baz can be modified to include a useful command:
foo: baz ; @true
and the correct results will be generated.
Conclusion
This is a simple technique that can provide useful information from the Make. It isn't as flexible as some commercial tools that can provide detailed information about the commands run, and even timing information, but as a Makefile debugging aid it's simple to implement and provides insight that hard to obtain otherwise.
