Document build

I do all my publish project, and general orgmode configuration in publish.el.

It consists of:

• loading supporting libraries (mostly ./lisp/drislox*.el)
• setting publishing directories
• setting document metadata
• setting names and descriptions of sub-documents
• orgmode configuration (mostly publishing/export preferences)
• listing all the publishing projects and their associated properties

A project defines a set of files or directory to publish. It has some properties that control how it is published, and it defines the function used to publish it.

For simple formats, that generally breaks down to a couple projects. For example, I have a 'tex' target, and 'tex-static'. The former converts org documents to LaTeX, and the latter copies static assets from the base directory to the publishing directory.

For HTML I wanted to produce multiple index pages so I broke the main publish project into one per sub-directory, and then defined a meta project. The meta-project in my case doesn't need any special setup, it just has one property, :components, that is a list of sub-projects.

I also publish some RSS feeds, but they are a special case. Discussed in RSS output.

Orgmode goes a long way, but doesn't produce all the final formats I want. For PDF I rely on latexmk, and a pre-amble document pkb.tex that includes the org generated index, which subsequently includes all the other files as subfiles.

For epub, org produces an XHTML output that I then package using my own ebook.py script which is based on ebooklib.

This is just categorized by which source file the function is in. Some things, like library of babel functions, aren't located by functionality.

Here is the export process summarized:

• Process temporary copy of the source Org buffer :

  1. Execute org-export-before-processing-functions
  2. Expand #+include keywords in the whole buffer
  3. Remove commented subtrees in the whole buffer
  4. Replace macros in the whole buffer
  5. Process code blocks

  This is where #+CALL: functions happen. They can modify the temporary buffer which is handy if you want to place or modify org markup. I also use them for external side effects (image modification).
  

• Parse the temporary buffer, creating AST (Abstract Syntax Tree):

  1. Execute org-export-before-parsing-functions. The hook functions may still modify the buffer

  Used for customizing ID of headlines to be a nice string. Has the downside that files are published in-turn so inter-linking can still end up producing org-generated IDs.
  

  1. Calculate export option values according to subtree-specific export settings, in-buffer keywords, #+BIND keywords, and buffer-local and global customization. The whole buffer is considered;
  2. When org-org-with-cite-processors is non-nil (default), determine contributing bibliographies and record them into export options. The whole buffer is considered;
  3. Execute org-export-filter-options-functions;
  4. Parse the accessible portion of the temporary buffer to generate an AST. The AST is a nested list of lists representing Org syntax elements.

Past this point, modifications to the temporary buffer no longer affect the export; Org export works only with the AST;

1. Remove elements that are not exported from the AST:
   • Headings, Comments, Table width/alignment rows, Table recalc columns, Clocks, drawers, fixed-width environments, footnotes, LaTeX environments, and fragments, node properties, planning lines, property drawers, statistics cookies, timestamps, etc. according to #+OPTIONS keyword
2. Expand environment variables in file link AST nodes
3. Execute org-export-filter-parse-tree-functions. These functions can modify the AST by side effects;
4. Replace citation AST nodes and #+print_bibliography keyword AST nodes

• Convert the AST to text by traversing the AST nodes, depth-first:
  1. Convert the leaf nodes (without children) to text
  2. Pass the converted nodes through the corresponding export filters
  3. Concatenate all the converted child nodes to produce parent node contents;
  4. Convert the nodes with children to text, passing the nodes themselves and their exported contents to the corresponding transcoders and then to the export filters.

• Post-process the exported text:
  1. Post-process the converted AST, as prescribed by the export backend. This step usually adds generated content (like Table of Contents) to the exported text;
  2. Execute org-export-filter-body-functions;
  3. Add the necessary metadata to the final document, as prescribed by the export backend. Examples: Document author/title; HTML headers/footers; LaTeX preamble;
  4. Add bibliography metadata, as prescribed by the citation export processor;
  5. Execute org-export-filter-final-output-functions.

I use ox-rss for the final RSS export.

Preparing things for RSS export is mostly just collecting all the entries into one buffer in a suitable format. Then sorting them anti-chronologically.

The first part happens in lib-babel.org functions. The rest is in drislox-rss.el or drislox-blog.el.

Blog has the extra feature of drafts. If a file has #+FILETAGS: :draft: then it is still exported, but won't be linked from the blog, nor included in the RSS.

TODO write some more, link the file names above