[linux-elitists] Fun* with inotify and make

[Don Marti]

Shell function, "makewatch" to re-run make when
the relevant files change. Surprisingly useful
for refreshing the HTML version of a page when the
Markdown source changes, especially if you're into the
Auto Reload extension. I also plan to use this for
"make test".

Andrew Cowie has something similar...
http://blogs.operationaldynamics.com/andrew/software/haskell/rebuilding-via-inotify
...but I like mine better because it doesn't have to
special-case the irrelevant files--it actually looks
for what "make" needs.

I think I can get rid of the "cut" in the helper
function if I learn more about grep.

make_prereqs() {
# finds all prerequisites mentioned in a Makefile
# that are actual files.
for f in `egrep '^\S.*:\S*' Makefile | \
cut -d ':' -f 2 | tr -s ' ' '\n' | \
sort -u`; do
[ -e $f ] && echo -n "$f ";
done
echo
}

makewatch() {
# re-run make, with the supplied arguments, when a
# Makefile prerequisite changes. Example:
# makewatch test
# Fun to use with Auto Reload and Pandoc
# https://addons.mozilla.org/en-US/firefox/addon/auto-reload/?src=api
# http://johnmacfarlane.net/pandoc/
if [ ! -e Makefile ]; then
echo "No Makefile in this directory." >&2
return
fi
make $*
while true; do
inotifywait -e close_write -e moved_to `make_prereqs`
make $*
done
}

* actual fun achived depends on Makefile content.

[Teh Entar-Nick]

Shell function, "makewatch" to re-run make when the relevant files
change. Surprisingly useful for refreshing the HTML version of a page
when the Markdown source changes, especially if you're into the Auto
Reload extension.

Neat, but nearly all of my Makefiles use patterns like this:

MARKDOWNS=$(wildcard *.md)
all: $(patsubst %.md,%.html,$(MARKDOWNS))
%.html: %.md
pandoc -t html5 --ascii --section-divs --standalone --smart -o
$@ $<
%.pdf: %.md
pandoc --chapters --variable=papersize:'a5paper' -o $@ $<

It doesn't seem that this kind of automatic pattern-driven makefile is
supported by your script.

I also plan to use this for "make test".

I did this in Python for yardbird a few years ago:

https://bazaar.launchpad.net/~spacehobo/yardbird/trunk/view/head:/autotest.py

Note that it uses pyinotify (for inotify) as well as pynotify (for
dbus-driven desktop notification popups).

I think I can get rid of the "cut" in the helper
function if I learn more about grep.
make_prereqs() {
# finds all prerequisites mentioned in a Makefile
# that are actual files.
for f in `egrep '^\S.*:\S*' Makefile | \
cut -d ':' -f 2 | tr -s ' ' '\n' | \
sort -u`; do
[ -e $f ] && echo -n "$f ";
done
echo
}

MOAR AWK:

gawk -F ':\\s*' '$1 ~ /^\S+$/ && $2 ~ /\S/ {gsub(/\s+/, "\n"); print $2}' Makefile | sort -u | xargs ls 2>/dev/null | xargs echo

It even supports whitespace after the : now. It switched to xargs just
because that's how I roll.

[Don Marti]

Post by Teh Entar-Nick
MARKDOWNS=$(wildcard *.md)
all: $(patsubst %.md,%.html,$(MARKDOWNS))
%.html: %.md
pandoc -t html5 --ascii --section-divs --standalone --smart -o
%.pdf: %.md
It doesn't seem that this kind of automatic pattern-driven makefile is
supported by your script.

Arr, if only we had a program that would parse
Makefiles...

_make_prereqs() {
make -dnr $* | tr ' ' '\n' | \
grep ".*'.$" | grep -o '\w.*\b'
}

make_prereqs() {
# prerequisites mentioned in a Makefile
# that are actual files.
for f in `_make_prereqs $* | sort -u`; do
[ -e $f ] && echo -n "$f ";
done
echo
}

Now it should only watching the files that "make"
thinks are relevant to the target. Still an extra
"grep" in there, probably.

[Teh Entar-Nick]

Post by Don Marti
Arr, if only we had a program that would parse
Makefiles...
_make_prereqs() {
make -dnr $* | tr ' ' '\n' | \
grep ".*'.$" | grep -o '\w.*\b'
}

Good one! You'll want an inotify to refresh this when the Makefile
itself changes, of course.

Only problem I see is that you're pulling out all files, including
*target* files. That could get annoying when e.g. I pull up a .html
file in vim to try something out, and when I save it my hack gets
reverted automatically.

Also, it seems to ignore explicit prerequisites! If I put a
metadata.xml to CC-license my epub files, it never shows up in that
output:

%.epub: %.md %_cover.jpeg metadata.xml
pandoc -S --epub-metadata=metadata.xml --epub-cover-image=$*_cover.jpeg -o $@ $<

Alas:

$ make -dnr all | grep -c xml
0
$ make -dnr foo.epub | grep -c xml
7

Poking around in -p and -B don't seem to help much here.

[Don Marti]

Post by Teh Entar-Nick

Post by Don Marti
Arr, if only we had a program that would parse
Makefiles...
_make_prereqs() {
make -dnr $* | tr ' ' '\n' | \
grep ".*'.$" | grep -o '\w.*\b'
}

Good one! You'll want an inotify to refresh this when the Makefile
itself changes, of course.

Yes, I added Makefile right after I sent the last mail.

_make_prereqs() {
echo "Makefile"
make -dnr $* | tr ' ' '\n' | \
grep ".*'.$" | grep -o '\w.*\b'
}

Post by Teh Entar-Nick
Only problem I see is that you're pulling out all files, including
*target* files. That could get annoying when e.g. I pull up a .html
file in vim to try something out, and when I save it my hack gets
reverted automatically.

I try to always do that kind of modifications on a
copy of the target file so that I don't overwrite it
myself by mistake.

Also, somewhat improved version of the main loop.
Exit status of inotifywait is 0 if the watched-for
event happens, 1 if a different event happens, so this
should only run make if a file has an "interesting"
change.

while true; do
inotifywait -e close_write -e moved_to `make_prereqs $*` && \
make $*

Post by Teh Entar-Nick
Also, it seems to ignore explicit prerequisites! If I put a
metadata.xml to CC-license my epub files, it never shows up in that
%.epub: %.md %_cover.jpeg metadata.xml
$ make -dnr all | grep -c xml
0
$ make -dnr foo.epub | grep -c xml
7
Poking around in -p and -B don't seem to help much here.

Strange. I tried adding a "metadata.xml" prerequisite and got...

$ make_prereqs all
index.html index.md Makefile metadata.xml templates/default.html

$ make -dnr index.html | grep xml
Considering target file `metadata.xml'.
Looking for an implicit rule for `metadata.xml'.
No implicit rule found for `metadata.xml'.
Finished prerequisites of target file `metadata.xml'.
No need to remake target `metadata.xml'.
Prerequisite `metadata.xml' is newer than target `index.html'.

[Teh Entar-Nick]

Post by Don Marti

Post by Teh Entar-Nick
%.epub: %.md %_cover.jpeg metadata.xml

Strange. I tried adding a "metadata.xml" prerequisite and got...
$ make_prereqs all
index.html index.md Makefile metadata.xml templates/default.html

Was it an explicit prereq of an implicit rule, as above?

[Don Marti]

Post by Teh Entar-Nick

Post by Don Marti

Post by Teh Entar-Nick
%.epub: %.md %_cover.jpeg metadata.xml

Strange. I tried adding a "metadata.xml" prerequisite and got...
$ make_prereqs all
index.html index.md Makefile metadata.xml templates/default.html

Was it an explicit prereq of an implicit rule, as above?

This is what I have...

%.html : %.md templates/default.html metadata.xml
pandoc \
--template templates/default.html --to=html5 \
--smart --standalone --toc \
-o $@ $<

(Used it to make this...
http://zgp.org/targeted-advertising-considered-harmful/
...comments welcome.)