Lilac

1. Introduction

The lilac is a flowering woody plant.

Lilac is an addon for Emacs Org mode to make Literate Programming (LP) using Noweb-style [1] references easy. It comes in two parts:

  1. lilac.el: to be loaded and used when weaving [2, p. 98] and tangling [2, p. 98] from your own Org mode files (which use Noweb references), and
  2. lilac.theme: to be sourced by your Org mode files to use Lilac's CSS and JavaScript on top of the HTML export.

This document itself reuses the lilac.theme file, which gives you a preview (if you are viewing the HTML output) of what weaving with Lilac looks like.

NOTE: Do not view this page using GitHub's default Org file viewer, because many sections will appear broken. Instead, please view this page by visiting https://funloop.org/lilac, or by cloning the repository and pointing your browser to the top-level index.html file directly.

1.1. Motivation and differences vs Org defaults

Lilac sacrifices some degree of customization in Org mode in order to make it easier to write literate programs using Noweb-style references. Lilac also focuses on HTML export for weaving.

The following are some examples of how things are done in the default Org mode settings, and how Lilac does things differently.

1.1.1. Source code block captions

1.1.1.1. Org defaults

By default, source code blocks do not get any captions in the HTML output. This can be an annoyance for LP because you have to manually explain the name of each block to keep track of their usage in other blocks (when referring to them with Noweb style references).

1.1.1.2. Lilac

Lilac frees you from having to write a custom #+caption: ... text for every source code block, because it generates them based on the name of the Noweb reference. Lilac links child blocks and parent blocks together so that you can click on code block names to navigate around this parent/child hierarchy.

1.1.4. HTML Table of Contents

1.1.4.1. Org defaults

When exporting to HTML, the Table of Contents (TOC) is at the top of the document.

1.1.4.2. Lilac

Lilac puts the TOC in a sidebar on the left. It also uses some JavaScript to check where the current mouse position is to update the current location in the TOC. That is, the TOC follows you as you scroll down the document.

1.1.5. Side notes, aka "margin notes"

1.1.5.1. Org defaults

Side notes are used to insert notes on the right-hand-side margin, and are inspired by the ones found in Donald Knuth's books.

They are not supported in Org mode.

1.1.5.2. Lilac

Lilac supports the #+begin_sidenote and #+end_sidenote delimiters. Text inside such delimiters are shown on the right hand margin.

2. User Guide

First create an Org mode file that uses LP. Then either clone this repo or add it as a submodule to your project. The point is to get the lilac.el (and its dependencies, themselves submodules) and lilac.theme (and associated CSS/JavaScript files) available locally. Then for the Org mode file you are using, you can load lilac.el and run (lilac-publish) to generate the HTML file, or run (org-babel-tangle) to generate the source code from it. See the Makefile that this project uses as a reference.

The main thing that you need to keep in mind when writing Org mode files for consumption by Lilac is that the source code blocks must use Noweb references using __NREF__ as a prefix. See the discussion below.

2.1. Source code blocks: monoblocks and polyblocks

In Org mode, Noweb-style references by default must be enclosed in double angle brackets << and >>. While this works, it's problematic because it can mean different things syntactically based on the source code language. Your source code block's language might think that the angle brackets are operators or keywords and colorize them differently, for example.

Instead, Lilac expects Noweb-style references in the form __NREF__foo (where the "NREF" stands for "Noweb reference"). Then you are free to name your child block with this same __NREF__foo name. This is better because now you can search for this word __NREF__foo in your raw Org mode document and you'll instantly be able to see where it is used. Contrast this with the default Org mode behavior where you'll have to search for <<foo>> and foo separately (because searching for just foo may collide with other names that are not source code block names).

In the example below, the parent-block refers to 2 other child blocks for its definition.

🔗
#+name: parent-block
#+begin_src bash
echo "Hello from the parent block"
__NREF__child-block-1
__NREF__child-block-2
#+end_src

...

#+name: __NREF__child-block-1
#+begin_src bash
echo "I am child 1"
#+end_src

...

#+header: :noweb-ref __NREF__child-block-2
#+begin_src bash
echo -n "I am "
#+end_src

#+header: :noweb-ref __NREF__child-block-2
#+begin_src bash
echo "child 2"
#+end_src

This example illustrates the two ways to define a child block: either as a single code block with #+name: __NREF__foo, or as multiple blocks with #+header: :noweb-ref __NREF__foo. Lilac calls them monoblocks and polyblocks respectively. Polyblocks are concatenated together in the order they appear in the overall Org file; this final concatenated version is what gets inserted into the Noweb reference in the parent block.

2.1.1. Examples of source code blocks

Below is an example of the usage described above. Notice how the child blocks referenced in the parent block are linked to their definitions.

🔗
echo "Hello from the parent block"
example-child-block-foo
echo ""
example-child-block-bar

Below are the child blocks. The first is a monoblock. Every child block's name links back up to the parent block where it is referenced.

example-child-block-foo🔗
echo "I am child 1"

The blocks below are polyblocks.

example-child-block-bar(1/2) 🔗
echo -n "I am "

Note the fraction after the name which denotes the position of the block in the overall polyblock "chain".

example-child-block-bar(2/2) 🔗
echo "child 2"
2.1.1.1. Multi-parent child blocks

Sometimes a piece of code will be reused in different source code blocks. You can think of it as constants in a programming language, but in LP the concept extends to any arbitrary piece of text.

Below is an example where we define a child block once…

example-child-block-baz2🔗
echo "Hello from child block baz"

…but where we also reuse it (reference it) in different parent blocks.

🔗
echo "Hello from parent 1"
example-child-block-baz
🔗
echo "Hello from parent 2"
example-child-block-baz

Notice that the child block has an additional link named "2", which points to the second parent that references this block (the first parent is linked by the child block name itself, as in the previous example). If it is referenced in additional parent blocks, they will show up as links "3", "4", etc.

2.2. Examples

Here we have some examples of how certain Org mode features look when exported to HTML with Lilac. Each section has an orgmode input and HTML output. The input is what you type into your *.org (plaintext) file, and the output is what you get in HTML after it is rendered with Lilac's publishing function, (lilac-publish).

2.2.1. Code blocks

2.2.1.1. Source blocks
2.2.1.1.1. Plain

Orgmode input: 

#+begin_src python
def foo():
    return 42
#+end_src

Rendered HTML output: 

def foo():
    return 42
2.2.1.1.2. Monoblock

Orgmode input:

Although the ex_monoblock text is linkified here, ignore it. The text that you need to write in the Orgmode input is literally __NREF__ex_monoblock. Lilac currently doesn't understand that the block here is an example and that it should not linkify it. 

#+caption: A parent block.
#+begin_src python
def foo():
    ex_monoblock
    return 42
#+end_src

#+name: ex_monoblock
#+begin_src python
print("hello world")
#+end_src

Rendered HTML output:

🔗
def foo():
    ex_monoblock
    return 42

Similarly, the 2 link is only here because Lilac thinks that this child block is referenced in multiple blocks. In a real example this 2 would not show up.

ex_monoblock2🔗
print("hello world")
2.2.1.1.3. Monoblock (evaluation result's value)

Sometimes it's useful to evaluate code and to use the resulting value. This is in contrast to always inserting the contents of the source code blocks literally.

However you must make sure that the referenced source code block's programming language is added to the list of org-babel-do-load-languages. Otherwise the evaluation will be forbidden.

Orgmode input: 

#+name: __NREF__ex_evaluate_me
#+begin_src python :exports none
return 4 * 10 + 2
#+end_src

#+caption: This uses an evaluation result.
#+begin_src python :noweb yes
print("The answer is <<__NREF__ex_evaluate_me()>>")
#+end_src

Rendered HTML output:

The "42" here is computed by the Python interpreter and then injected into the print statement's string argument.

🔗
print("The answer is 42")
2.2.1.1.4. Monoblock (pass argument to source code block)

Here we have an example of passing in an argument to a source code block.

Orgmode input: 

#+name: __NREF__ex_take_argument
#+begin_src python :exports none :var my_var="foo"
return "Hello, my name is " + my_var
#+end_src

#+caption: Passing string args.
#+begin_src python :noweb yes
print("Call without explicit argument: __NREF__ex_take_argument()")
print("Call with explicit argument: __NREF__ex_take_argument(my_var="bar")")
#+end_src

Rendered HTML output:

🔗
print("Call without explicit argument: Hello, my name is foo")
print("Call with explicit argument: Hello, my name is bar")
2.2.1.1.5. Polyblock

Orgmode input:

Like above for monoblocks, Lilac needlessly linkifies the source code body text of the parent block when it doesn't need to (as this is only an illustrative example). 

#+caption: A parent block.
#+begin_src python
def foo():
    ex_polyblock
    return 42
#+end_src

#+header: :noweb-ref ex_polyblock
#+begin_src python
print("hello world")
#+end_src

Some intervening text.

#+header: :noweb-ref ex_polyblock
#+begin_src python
print("abcdefg")
#+end_src

Rendered HTML output:

🔗
def foo():
    ex_polyblock
    return 42
ex_polyblock2(1/2) 🔗
print("hello world")

Some intervening text.

ex_polyblock2(2/2) 🔗
print("abcdefg")
2.2.1.1.6. Linking to a line in the code block body

The -r flag removes these #ref:... labels from the source code. The -l flag changes the format. In this case we make the format use a leading # character because that's the comment character in Python. This way they appear as comments when we are inserting them into the block while still in Orgmode.

These links are documented in Org's Literal Examples section.

Orgmode input: 

#+begin_src python -r -l "#ref:%s"
print("foo")
print("hi")     #ref:embedded-link
print("bar")
#+end_src

The line at [[(embedded-link)][=embedded-link=]] prints "hi".

Rendered HTML output: 

print("foo")
print("hi")
print("bar")

The line at embedded-link prints "hi".

2.2.1.2. Example blocks

Orgmode input: 

#+begin_example
Hello world.
Foo bar.
#+end_example

Rendered HTML output: 

Hello world.
Foo bar.
2.2.1.3. Quote blocks

Orgmode input: 

#+begin_quote
Hello world.
Foo bar.
#+end_quote

Rendered HTML output:

Hello world. Foo bar.

2.2.2. Side notes

Note that Lilac only adds some special CSS styling to side notes. Orgmode is generous with source code blocks and accepts any #+begin_foo and #+end_foo lines.

Orgmode input: 

#+begin_sidenote
Hello world.
Foo bar.
#+end_sidenote

Rendered HTML output (see right hand margin):

Hello world. Foo bar.

2.2.3. Math (MathJax)

This is not a Lilac feature (it's just part of Org mode), but it's worth demonstrating how it looks.

Orgmode input: 

The equation \( E = mc^2 \) is probably the most famous equation in the world,
perhaps more famous than the Pythagorean theorem

\[
a^2 + b^2 = c^2.
\]

The equation

\[
\frac{\pi^2}{6} = \sum_{n=1}^\infty \frac{1}{n^2} = \
\frac{1}{1^2} + \frac{1}{2^2} + \frac{1}{3^2} + \cdots
\]

was first demonstrated by Leonhard Euler in 1734. Here's another equation,

#+name: eqn:1
\begin{equation}
\pi = \sum_{k = 0}^{\infty}\left[\frac{1}{16^k} \
\left(\frac{4}{8k+1}-\frac{2}{8k+4}-\frac{1}{8k + 5}-\frac{1}{8k+6}\right)\right]
\end{equation}

which is known as the Bailey-Borwein-Plouffe formula and can compute arbitrary
digits of \(\pi\). A variant of this equation is as follows

#+name: eqn:2
\begin{equation}
\pi = \frac1{2^6} \sum_{n=0}^\infty \frac{(-1)^n}{2^{10n}} \
\left(-\frac{2^5}{4n+1} - \frac1{4n+3} + \frac{2^8}{10n+1} - \
\frac{2^6}{10n+3} - \frac{2^2}{10n+5} - \frac{2^2}{10n+7} + \
\frac1{10n+9} \right)
\end{equation}

and is known as the Bellard's formula. Equation [[eqn:2]] is too wide so
MathJax breaks it over multiple lines. Below is a version where we do the line
breaking ourselves.

\begin{equation}
\pi = \frac1{2^6} \sum_{n=0}^\infty \frac{(-1)^n}{2^{10n}} \
\left(-\frac{2^5}{4n+1} - \frac1{4n+3} + \frac{2^8}{10n+1} - \\
\frac{2^6}{10n+3} - \frac{2^2}{10n+5} - \frac{2^2}{10n+7} + \
\frac1{10n+9} \right)
\end{equation}

Rendered HTML output:

The equation \( E = mc^2 \) is probably the most famous equation in the world, perhaps more famous than the Pythagorean theorem

\[ a^2 + b^2 = c^2. \]

The equation

\[ \frac{\pi^2}{6} = \sum_{n=1}^\infty \frac{1}{n^2} = \ \frac{1}{1^2} + \frac{1}{2^2} + \frac{1}{3^2} + \cdots \]

was first demonstrated by Leonhard Euler in 1734. Here's another equation,

\begin{equation} \label{org000004a} \pi = \sum_{k = 0}^{\infty}\left[\frac{1}{16^k} \ \left(\frac{4}{8k+1}-\frac{2}{8k+4}-\frac{1}{8k + 5}-\frac{1}{8k+6}\right)\right] \end{equation}

which is known as the Bailey-Borwein-Plouffe formula and can compute arbitrary digits of \(\pi\). A variant of this equation is as follows

\begin{equation} \label{org000004c} \pi = \frac1{2^6} \sum_{n=0}^\infty \frac{(-1)^n}{2^{10n}} \ \left(-\frac{2^5}{4n+1} - \frac1{4n+3} + \frac{2^8}{10n+1} - \ \frac{2^6}{10n+3} - \frac{2^2}{10n+5} - \frac{2^2}{10n+7} + \ \frac1{10n+9} \right) \end{equation}

and is known as the Bellard's formula. Equation \eqref{org000004c} is too wide, though. Below is a version where we insert manual line breaks.

\begin{equation} \pi = \frac1{2^6} \sum_{n=0}^\infty \frac{(-1)^n}{2^{10n}} \ \left(-\frac{2^5}{4n+1} - \frac1{4n+3} + \frac{2^8}{10n+1} - \\ \frac{2^6}{10n+3} - \frac{2^2}{10n+5} - \frac{2^2}{10n+7} + \ \frac1{10n+9} \right) \end{equation}

3. Known limitations

These limitations stem from the fact that they are not natively supported by Org mode.

Source code blocks must reference other blocks in the same file
If you're using source code blocks, and they refer to other blocks, those referenced blocks must be defined in the same Org file. In practice this isn't a big problem.

4. Developer Guide

If you are interested in contributing back to this project, or if you are curious to see how everything works, have a look at the Developer Guide.

5. Glossary

literate document
A file or collection of files that include both source code and prose to explain it. Well-known formats include Noweb files (*.nw) and Org mode files (*.org).
monoblock
an Org mode source code block with a #+name: ... field. This block is an independent block and there are no other blocks with the same name.
Noweb
A literate programming tool from 1989 that still works and from which Org mode borrows heavily using Noweb-style references. See Wikipedia.
noweb-ref
aka "Noweb-style reference". A Noweb-style reference is just a name (string) that refers to a monoblock or polyblock. See the Org manual.
Org mode
An Emacs major mode for *.org files, where "major mode" means that it provides things like syntax highlighting and keyboard shortcuts for *.org text files if you are using Emacs. For Lilac, the important thing is that we use Org mode as a literate programming tool. See Org mode.
polyblock
an Org mode source code block without a #+name: ... field, but which has a #+header: :noweb-ref ... field. Other blocks with the same Noweb-ref name are concatenated together when they are tangled. Polyblocks are used in cases where we would like to break up a single block into smaller pieces for explanatory purposes. In all other cases, monoblocks are preferable, unless the source code block is not to be tangled and is only for explanatory purposes in the woven output.
source code block
An Org mode facility that allows you to enclose a multiline text (typically source code) with #+begin_src ... and #+end_src lines. They are enclosed in a separate background color in the HTML output, and are often used for illustrating source code listings. The format is #+begin_src LANGUAGE_NAME where LANGUAGE_NAME is the name of the programming language used for the listing. If the name is a recognized name, it will get syntax highlighting in the output automatically.
tangling
The act of extracting source code from a raw literate document.
weaving
The act of converting a raw literate document to a richer format such as PDF or HTML. This allows fancier output, such as for mathematical formulas, which are easier to read versus the original literate document.

6. Projects using Lilac

7. References

[1]
N. Ramsey, “Literate programming simplified,” IEEE Software, vol. 11, no. 5, pp. 97–105, Sep. 1994, doi: 10.1109/52.311070.
[2]
D. E. Knuth, “Literate Programming,” The Computer Journal, vol. 27, no. 2, pp. 97–111, Feb. 1984, doi: 10.1093/comjnl/27.2.97.