Chapter 9: Metaprogramming

Where we go meta. For real.

Metaprogramming is the art of making programs... that make programs.

There are many techniques and tools for achieving this, but one that is very familiar to Lisp fans is to use macros to generate code at compile-time.

However, we're not going to talk about macros on this chapter.

Instead, I'll reveal the infrastructure that makes macros possible, and we'll discuss macros on the next chapter.

The Compiler Type

The Lux compiler was designed to integrate very well with the language itself.

Most compilers are just programs that take source code and emit some binary executable or some byte-code. But the Lux compiler opens itself for usage within Lux programs and provides Lux programmers with a wealth of information.

The Compiler type enters the stage.

(type: Compiler
  {#info Compiler-Info
   #source Source
   #cursor Cursor
   #modules (List [Text Module])
   #envs (List Scope)
   #type-vars (Bindings Int Type)
   #expected (Maybe Type)
   #seed Int
   #scope-type-vars (List Int)
   #host Void})

By the way, the Compiler type and other weird types you may not recognize there are all defined in the lux module. Check the documentation in the Standard Library for more details.

The Compiler type represents the state of the Lux compiler at any given point.

It's not a reflection of that state, or a subset of it. It is the state of the Lux compiler; and, as you can see, it contains quite a lot of information about compiled modules, the state of the type-checker, the lexical and global environments, and more.

Heck, you can even access the yet-unprocessed source code of a module at any given time.

That's pretty neat.

You can actually write computations that can read and even modify (careful with that one) the state of the compiler. This turns out to be massively useful when implementing a variety of powerful macros.

For example, remember the open and :: macros from chapter 7?

They actually look up the typing information for the structures you give them to figure out the names of members and generate the code necessary to get that functionality going.

And that is just the tip of the iceberg.

The module: macro uses module information to generate all the necessary code for locally importing foreign definitions, and some macros for doing host interop analyze the annotations of local definitions to help you write shorter code when importing Java classes/methods/fields or defining your own.

The possibilities are really vast when it comes to using the information provided by the Compiler state.

The Lux Type

But, how do I use it?

Well, that is where the Lux type and the lux/compiler module come into play.

Yeah, I'm aware that it's weird there's a type with the same name as the language, but I couldn't figure out a better name.

The lux/compiler module houses many functions for querying the Compiler state for information, and even to change it a little bit (in safe ways).

I won't go into detail about what's available, but you'll quickly get an idea of what you can do if you read the documentation for it in the Standard Library.

However, one thing I will say is that those functions rely heavily on the Lux type, which is defined thus:

(type: (Lux a)
  (-> Compiler (Either Text [Compiler a])))

The Lux type is defined in the lux module, although most functions that deal with it are in the lux/compiler module. Also, lux/compiler contains Functor<Lux> and Monad<Lux>.

The Lux type is a functor, and a monad, but it is rather complicated.

You saw some functor/applicative/monad examples in the last chapter, but this is more colorful.

Lux instances are functions that given an instance of the Compiler state, will perform some calculations which may fail (with an error message); but if they succeed, they return a value, plus a (possibly updated) instance of the Compiler.

Lux metaprogramming is based heavily on the Lux type, and macros themselves rely on it for many of their functionalities, as you'll see in the next chapter.

Where do Compiler instances come from?

Clearly, Compiler instances are data, but the compiler is not available at all times.

The compiler is only ever present during... well... compilation.

And that is precisely when all of your Compiler-dependant code will be run.

Basically, in order for you to get your hands on that sweet compiler information, your code must be run at compile-time. But only macro code can ever do that, so you will have to wait until the next chapter to learn how this story ends.

Definition Annotations

Another important piece of information you should be aware of is that definitions don't just have values and types associated with them, but also arbitrary meta-data which you can customize as much as you want.

The relevant types in the lux module are

(type: #rec Ann-Value
  (#BoolM Bool)
  (#IntM Int)
  (#RealM Real)
  (#CharM Char)
  (#TextM Text)
  (#IdentM Ident)
  (#ListM (List Ann-Value))
  (#DictM (List [Text Ann-Value])))

and

(type: Anns
  (List [Ident Ann-Value]))

You can add annotations to definitions in the many definition macros offered in the standard library. All you need to do is pass in some record syntax, with tags being the Ident part of the annotations, and the associated value being either an explicit variant Ann-Value, or some function or macro call that would produce such a value.

Here's an example from lux:

(def: #export (is left right)
  {#;doc (doc "Tests whether the 2 values are identical (not just \"equal\")."
              "This one should succeed:"
              (let [value 5]
                (is 5 5))

              "This one should fail:"
              (is 5 (+ 2 3)))}
  (All [a] (-> a a Bool))
  (_lux_proc ["lux" "=="] [left right]))

The (optional) annotations always goes after the declaration or name of the thing being defined.

Note that all tag usage within annotation records should be prefixed, to avoid potential confusions, as different modules could be using annotation tags with similar names.

The lux/compiler module contains various functions for reading and exploring the definition annotations, and some modules in the standard library (for example, the lux/host module) make heavy use of annotations to figure out properties of definitions which may be useful during code-generation and parsing in macros.

And also, as you can appreciate from the previous example, some macros may be designed to be used during annotation specification.

This chapter feels a little empty because the topic only makes sense within the context of macros. But macros by themselves are a huge subject, and involve more machinery than you've seen so far.

However, I wanted to give you a taste of what's possible in order to whet your appetite, while keeping the chapter focused.

In the next chapter, I'll complete this puzzle, and you'll be given access to a power greater than you've ever known (unless you've already been a lisper for a while).

See you in the next chapter!