Debugging 017_structs.es

The next steps are implementing the basics of the runtime and garbage collection. I hope to make some good progress before I start with my diploma thesis.

• global variables
• function pointers using the syntax function(typeArg1, typeArg2, typeArg3, ...) as returnType
• debug data for use with gdb / ddd featuring
  • source code based debugging with “next” / “step” etc.
  • local variable type and content can be viewed for the basic data types

Here’s a screenshot showing nbody.es debugged by ddd:

Debugging nbody.es in ddd

I’m currently working on extending the debugging data further, so that variables of derived types (especially structs) can be viewed in the debugger. After that I’ll probably start the runtime with a basic garbage collector.

As I’m using LLVM as a backend for Exoself I expect to get similar performance. So far I’ve only ported the nbody benchmark from The Computer Language Benchmarks Game and it was slower by a factor of 1.5 to 2 than gcc depending on the CPU.

The goal of testing is to make sure that the generated programs do what was specified in the source code. Here we need to make sure that we don’t introduce any errors on several stages during compilation: Lexing / parsing, desugaring, type checking / AST annotation and finally code generation.

The ideas presented here are generally applicable to non-interactive command line programs. Even if you are writing a GUI application it can be a good idea to write first a shared library with a command line interface. That makes testing much easier. Later you can still write a GUI that wraps the functionality in a simple way for the end user.

Test Driven Development

This post will refer to TDD as writing tests first and only then writing new code. Of course in practice there are situations where you’ll not write tests or write them after you discover bugs. The strictness of this approach depends on the needs of your project and its need for correctness.

TDD is an iterative approach to writing code: First write some new tests which will fail due to missing functionality. Then write code until the new tests pass and finally check if all other tests still pass. To make this testing as simple as possible it’s important to have an automated way of running the tests.

In Exoself the tests are integrated into the build system. It’s trivially easy to either run the whole test suite or just run a few selected tests. The build system even knows when parts of the compiler implementation changed and automatically rebuilds the tests on demand.

Running only a subset of all tests gets more important when your project grows and the test suite takes to long to complete. Even one ore two minutes might be too long to get feedback regarding changes. So often you’ll want to run the whole test suite on a dedicated machine after source code checkins. A tool like Buildbot can automate this.

Of course there are many ways to write tests. I’ll present two techniques here: unit testing and behavior testing.

Unit Testing

This method tries to make sure that individual units of your program work correctly. Often this means a unit test per class, source file or module. Ideally it’s a very small or even elementary part of you application which you test.

When testing classes you write test cases for the public interface of the class, not the implementation details. When writing tests first this also defines the public interface of the class from a user perspective instead of the class implementor perspective which makes sure that the interface is usable.

Most classes depend on other classes which provide file / database / network access which should not be tested. To avoid this dependence often mock objects are introduced which implement the same interface as the original class, but only make sure that methods are called in the right order and with correct parameters.

I don’t like this way of writing tests: It’s too much additional work and too low level for me.

Behavior Testing

Testing the behavior means that I verify that the output of my program is correct and not any part or unit. In the case of a compiler that’s very simple: Write code in the source language, compile the program and run it. During runtime check with assert statements if runtime calculations match the expected results. Instead of runtime checking you can also of course compare the output of your compiler / program with a reference file.

If you’ve participated in programming competitions like ACM or TopCoder you’ll already know how it works. In these competitions you write a small program and submit it to an automated judge. Then it’s compiled and run for every test case which is a plain text input. The output of your program is then compared to a reference output and you’ll either pass or fail. The major difference to our situation is, that the test inputs and expected outputs are unknown as are the test case which failed.

The behavior testing approach has several benefits compared to unit testing: The program as a whole is tested, not any unit. Of course that means when a test fails it’s more difficult to find the bug in the code but on the other hand you know that certain inputs work correctly. You don’t need to provide mock objects and it’s often less work than unit testing: Just writing programs in the source language of your compiler should be easy.

Making sure that compiled programs work is difficult with a new programming language: No input / output and maybe even missing control flow / assert makes it hard. But for the first tests you can always construct the AST of the source by hand and check that or use the return value of the program.

Test Coverage

Especially with unit testing it can be tempting to get 100% test coverage. Coverage may be defined in several ways, for example functions / statement / path coverage. Path coverage is the most interesting one to guarantee correctness but means you’ll need a really big test suite – in practice that’s either impossible or just too much work for any non trivial program. And a compiler has the added problem that there are infinitely many valid inputs which makes things not really easier…

Behavior testing is easier: You have some kind of definition of the input language, which yields a finite number of basic tests which define the grammar and semantics. Adding some composite tests to the test suite should provide a good enough indicator for correctness of the compiler.

Testing just provides a lower bound of correctness: Everything which you checked works – everything also may or may not work. You can always fool your test suite by hard coding all inputs, then all tests work but nothing else.

Automatic Testing

Instead of writing many behavior tests by hand, this task can also be automated to some degree. For example to test operator precedence you could write a program that generates random mathematical expressions. The generated source code consists of the expression and the expected result.

Of course to compute the expected result you have to duplicate the language semantics in the expression generator. That’s easy if the expression generator is written in a language that has the same semantics for these expressions. Writing this program in your new language is probably a bad idea, since you can not trust the expected values this way.

As I’ll most probably use Python to write the random expression generator it’s not that trivial: Even simple expressions behave different, due to overflow of basic types like int32 or int64 which can not occur in Python. I’ll probably make the generator generic, so that I can generate code in different languages. As I expect the C / C++ implementation to be correct I’ll just use it to calculate the expected values. Exoself will often have the same semantics – at least in regard to the mathematical expressions.

This idea can be extended even to whole programs including control flow: Volatile-Testing Tools. This program generates random programs to check whether a C compiler handles volatiles correctly.

Conclusion

The Exoself test suite consists of many behavior tests, that prove constantly to be invaluable. It’s so much easier to make changes, small or large, and know that everything still works. The test suite is also a replacement, at least for now, for a language specification.

It will be interesting to see how I can improve my test suite by adding random generated programs in the future. If you know about any tools in this area, I’d be interested to hear about them.

Read more here: Compiler Writing Series

This is a technology preview without support and not an end user release.

Here’s a list of features working so far:

• basic mathematical operators
• user defined functions
• control flow
• binary functions as operators
• augmented assign, list assign
• very basic module system: “from module import *”
• type system supporting basic data types like int32 or float64, pointers and structs
• limited function overloading
• accessing C functions
• ASCII raw strings (prefix: ar)
• basic integration into the build system waf
• local type inference

Still somewhat problematic is output, since I don’t have support for variadic functions like printf. A runtime is also missing, so no garbage collection at the moment.

Platforms:
Linux: should just work in most cases as Ubuntu is my development platform. Gentoo needs some help when compiling llvm-py. See llvm-py issue 14.

Mac OS: The compiler itself seems to work. You’ll probably have to remove the hacks and runtime libraries from the build system in the root wscript file. Just remove them at the top of the file in the “dirs = ‘compiler runtime hacks tests’.split()” line.
(if you know where to get a free or reasonably cheap developer account with shell access please write a comment / an email)

Windows: not tested; I have no idea how much work it will be to get it to work

Due to an important exam I won’t have much time to work on Exoself or write blog posts in the next two weeks.

Single Static Assignment

LLVM uses a SSA form to represent variables. In SSA form a variable can be assigned a value only once. This makes it easier to write optimizations, since analyzing when a variable changes is not necessary. But while generating code this is problematic: Instead of changing the value of a variable we have to add a new variable. When using control flow like if or while this gets even more complicated. We need to add PHI nodes which carry information about predecessors of basic blocks.

Instead of doing things manually there’s a simpler way: LLVM does not use the SSA form for memory, so instead of using normal variables we use memory to store our data. To avoid any memory management issues we’ll allocate the memory on the stack using alloca. So when the current function returns the stack is popped and the memory automatically freed.

The great advantage of this solution is that it’s really easy to generate code this way. But there’s a catch: Every access to a variable now involves a load from / store to memory and that’s really inefficient compared to storing data in registers. Fortunately LLVM provides an optimization pass that can promote memory accesses to register variables and inserts the necessary PHI nodes.

This optimization pass is called “mem2reg” and has some limitations:

• it only works with alloca, so heap based memory or global variables are not transformed
• it only looks for allocas in the entry basic block of functions
• it can only promote allocas which are used for plain stores and loads – no address taking or pointer arithmetic is allowed
• it does only work for simple data types such as scalars, vectors and pointers. Especially structs or arrays are not supported.

So in summary the most important one is that allocas must be placed in the entry block of a function for promotion. Structs and arrays are not relevant for this example, but there are more powerful optimization passes which can promote also structs and arrays to registers.

Extending the compiler

To work with variables we’ll need an assignment operator and variable names. So I added two entries to the TreeType class: ASSIGN and VARIABLE_NAME:

class TreeType(object):
    MODULE = 1

    PLUS = 10
    MINUS = 11
    STAR = 12
    SLASH = 13

    INTEGER_CONSTANT = 100

    VARIABLE_NAME = 200

    ASSIGN = 300

In the ASTWalker class the dispatch function must also support these two new AST node types:

        # ...
        elif ast.type == tt.INTEGER_CONSTANT:
            callee = self._onIntegerConstant
            kwargs['value'] = int(ast.text)
        elif ast.type == tt.ASSIGN:
            callee = self._onAssign
            kwargs['variable'] = ast.children[0]
            kwargs['expression'] = ast.children[1]
        elif ast.type == tt.VARIABLE_NAME:
            callee = self._onVariable
            kwargs['variable'] = ast
        else:
            assert(0 and 'dead code path')
        # ...

Since we don’t have a semantic phase in this compiler we have to manage the symbol table manually. So I’ve added it to the constructor of the CodeGen class:

    def __init__(self, *k, *kw):
        ASTWalker.__init__(self, *k, **kw)

        self._symbols = {}

We also need to keep a reference to the current function accessible during an assignment to create the alloca in the entry block of the function instead of the current block. Otherwise the mem2reg pass won’t work.

        # ...
        func = self._module.add_function(funcType, 'main')
        self._currentFunction = func
        # ...

Now to the more interesting part: Generating code for an assignment.

    def _onAssign(self, ast, variable, expression):
        self._dispatch(expression)

        assert(variable.type == TreeType.VARIABLE_NAME)

        varName = variable.text
        if varName not in self._symbols:
            # create a new variable
            # since we want to avoid SSA form we'll put the variable on the stack and let the optimizer do the transformation to SSA

            # alloca must be in the entry block otherwise the optimizer will not transform it
            # get current function and the entry block (the name 'entry' above was only for human readers, it's the first basic block of a function)
            llvmFunc = self._currentFunction
            entryBB = llvmFunc.get_entry_basic_block()

            # create a builder for the entry block and insert new instructions at the beginning --> before the branch instruction
            # order of the alloca's does not really matter
            entryBuilder = Builder.new(entryBB)
            if entryBB.instructions: # avoid segfaults if empty
                entryBuilder.position_at_beginning(entryBB)

            # reserve memory on the stack
            llvmRef = entryBuilder.alloca(Type.int(32), varName)

            # store reference to variable in symbol table
            self._symbols[varName] = llvmRef
        else:
            # use existing variable
            llvmRef = self._symbols[varName]

        # store data to memory location
        self._currentBuilder.store(expression.llvmValue, llvmRef)

        # we also have to return a value
        ast.llvmValue = expression.llvmValue

After checking if the variable already exists either insert an alloca into the entry block or retrieve the reference from the symbol table and then store the result of the expression to that memory location.

And finally the handler for variables inside expressions:

    def _onVariable(self, ast, variable):
        varName = variable.text

        # look up symbol
        llvmRef = self._symbols[varName]

        # load value and return it
        ast.llvmValue = self._currentBuilder.load(llvmRef)

This just looks up the reference in the symbol table and then loads the data from memory.

That’s it. Here’s the complete program:

#!/usr/bin/python
import sys

sys.path.append('/home/fnoeding/exoself/3rdparty/pylibs')

from llvm.core import *
import copy

class Tree(object):
	def __init__(self, type, text, children=None):
		self.type = type
		self.text = text

		if children:
			self.children = children
		else:
			self.children = [] # do not use a default value of children = [] in the function param list!

	def copy(self, withChildren):
		if withChildren:
			return copy.deepcopy(self)
		else:
			c = copy.copy(self)
			c.children = []
			return c

class TreeType(object):
	MODULE = 1

	PLUS = 10
	MINUS = 11
	STAR = 12
	SLASH = 13

	INTEGER_CONSTANT = 100

	VARIABLE_NAME = 200

	ASSIGN = 300

class ASTWalker(object):
	def __init__(self):
		self._nodes = []

	def walkAST(self, ast):
		raise NotImplementedError('subclasses must implement walkAST')

	def _dispatch(self, ast):
		tt = TreeType

		kwargs = {}
		kwargs['ast'] = ast

		if ast.type == tt.MODULE:
			callee = self._onModule
			kwargs['statements'] = ast.children
		elif ast.type in [tt.PLUS, tt.MINUS, tt.STAR, tt.SLASH]:
			callee = self._onOperator

			op = ast.type
			if len(ast.children) == 1:
				arg1 = ast.children[0]
				arg2 = None
			elif len(ast.children) == 2:
				arg1 = ast.children[0]
				arg2 = ast.children[1]
			else:
				assert(0 and 'dead code path')

			kwargs['op'] = op
			kwargs['arg1'] = arg1
			kwargs['arg2'] = arg2
		elif ast.type == tt.INTEGER_CONSTANT:
			callee = self._onIntegerConstant
			kwargs['value'] = int(ast.text)
		elif ast.type == tt.ASSIGN:
			callee = self._onAssign
			kwargs['variable'] = ast.children[0]
			kwargs['expression'] = ast.children[1]
		elif ast.type == tt.VARIABLE_NAME:
			callee = self._onVariable
			kwargs['variable'] = ast
		else:
			assert(0 and 'dead code path')

		callee(**kwargs)

class CodeGen(ASTWalker):
	def __init__(self, *k, **kw):
		ASTWalker.__init__(self, *k, **kw)

		self._symbols = {}

	def walkAST(self, ast):
		self._dispatch(ast)

		return self._module

	def _addHelpers(self):
		# you might want to skip this function on a first read. It adds essentially a function to print integers to stdout.

		# add a prototype for printf
		# int printf(char*, ...)
		funcType = Type.function(Type.int(32), [Type.pointer(Type.int(8))], True)
		printf = self._module.add_function(funcType, 'printf')

		# add a function to print integers to stdout using printf
		# void printInt(int x) { printf("%d\n", x); }
		funcType = Type.function(Type.void(), [Type.int(32)])
		printInt = self._module.add_function(funcType, 'printInt')
		self._printInt = printInt # save for later use in _onModule

		# create a block and a builder for printInt
		bb = printInt.append_basic_block('bb')
		b = Builder.new(bb)

		# create a global constant to hold the first argument of printf
		stringConst = Constant.stringz('%d\n') # zero terminated --> stringz instead of string
		string = self._module.add_global_variable(stringConst.type, '__internalGlobalConst')
		string.initializer = stringConst
		string.global_constant = True
		string.linkage = LINKAGE_INTERNAL # not strictly necessary here, but this global should only be available during link time in the current module

		# address calculation
		# every index traverses a pointer without derefencing.
		# gep (get element pointer) does only address calculation, no memony accesses!
		idx = [Constant.int(Type.int(32), 0), Constant.int(Type.int(32), 0)] # the first index get's us past the global variable (which is a pointer) to the string; the second index is the offset inside the string we want to access
		realAddr = string.gep(idx) # get real address

		# call printf
		b.call(printf, [realAddr, printInt.args[0]])
		b.ret_void()

	def _onModule(self, ast, statements):
		# create a new LLVM module, a container for global variables and functions
		self._module = Module.new('mymodule')

		# add some helpers
		self._addHelpers()

		# add a function: 'int main()'
		funcType = Type.function(Type.int(32), [])
		func = self._module.add_function(funcType, 'main')
		self._currentFunction = func

		# create an 'entry' basic block, if we want to introduce variables we'll need it anyway
		# execution of the function starts here
		entryBB = func.append_basic_block('entry')
		entryBuilder = Builder.new(entryBB)

		# normal code should be inserted into the second block
		bb = func.append_basic_block('bb')

		# also jump to this block from the entry block
		entryBuilder.branch(bb)

		# add the code of the function
		self._currentBuilder = Builder.new(bb)
		for x in statements:
			self._dispatch(x)
			self._currentBuilder.call(self._printInt, [x.llvmValue])

		# main should return 0
		self._currentBuilder.ret(Constant.int(Type.int(32), 0))

		# verify the module
		self._module.verify()

	def _onOperator(self, ast, op, arg1, arg2):
		tt = TreeType

		self._dispatch(arg1)
		if arg2:# some operators are unary
			self._dispatch(arg2)

		cb = self._currentBuilder
		if op == TreeType.PLUS:
			if arg2:
				ast.llvmValue = cb.add(arg1.llvmValue, arg2.llvmValue)
			else:
				ast.llvmValue = arg1.llvmValue # +NUMBER == NUMBER
		elif op == TreeType.MINUS:
			if arg2:
				ast.llvmValue = cb.sub(arg1.llvmValue, arg2.llvmValue)
			else:
				ast.llvmValue = cb.sub(Constant.int(Type.int(32), 0), arg1.llvmValue) # -NUMBER == 0 - NUMBER
		elif op == TreeType.STAR:
			ast.llvmValue = cb.mul(arg1.llvmValue, arg2.llvmValue)
		elif op == TreeType.SLASH:
			ast.llvmValue = cb.sdiv(arg1.llvmValue, arg2.llvmValue)
		else:
			assert(0 and 'dead code path')

	def _onIntegerConstant(self, ast, value):
		ast.llvmValue = Constant.int(Type.int(32), value)

	def _onAssign(self, ast, variable, expression):
		self._dispatch(expression)

		assert(variable.type == TreeType.VARIABLE_NAME)

		varName = variable.text
		if varName not in self._symbols:
			# create a new variable
			# since we want to avoid SSA form we'll put the variable on the stack and let the optimizer do the transformation to SSA

			# alloca must be in the entry block otherwise the optimizer will not transform it
			# get current function and the entry block (the name 'entry' above was only for human readers, it's the first basic block of a function)
			llvmFunc = self._currentFunction
			entryBB = llvmFunc.get_entry_basic_block()

			# create a builder for the entry block and insert new instructions at the beginning --> before the branch instruction
			# order of the alloca's does not really matter
			entryBuilder = Builder.new(entryBB)
			if entryBB.instructions: # avoid segfaults if empty
				entryBuilder.position_at_beginning(entryBB)

			# reserve memory on the stack
			llvmRef = entryBuilder.alloca(Type.int(32), varName)

			# store reference to variable in symbol table
			self._symbols[varName] = llvmRef
		else:
			# use existing variable
			llvmRef = self._symbols[varName]

		# store data to memory location
		self._currentBuilder.store(expression.llvmValue, llvmRef)

		# we also have to return a value
		ast.llvmValue = expression.llvmValue

	def _onVariable(self, ast, variable):
		varName = variable.text

		# look up symbol
		llvmRef = self._symbols[varName]

		# load value and return it
		ast.llvmValue = self._currentBuilder.load(llvmRef)

def createSampleAST():
	# do not reuse any variables! overwrite them first!
	exprs = []

	# 9 - 3 * 3
	int9 = Tree(TreeType.INTEGER_CONSTANT, '9')
	int3a = Tree(TreeType.INTEGER_CONSTANT, '3')
	int3b = Tree(TreeType.INTEGER_CONSTANT, '3')

	star = Tree(TreeType.STAR, '*', [int3a, int3b])
	minus = Tree(TreeType.MINUS, '-', [int9, star])
	exprs.append(minus)

	# 4 + 76 / 2
	int4 = Tree(TreeType.INTEGER_CONSTANT, '4')
	int76 = Tree(TreeType.INTEGER_CONSTANT, '76')
	int2 = Tree(TreeType.INTEGER_CONSTANT, '2')

	slash = Tree(TreeType.SLASH, '/', [int76, int2])
	plus = Tree(TreeType.PLUS, '+', [int4, slash])
	exprs.append(plus)

	# ---21
	int21 = Tree(TreeType.INTEGER_CONSTANT, '21')
	minus = Tree(TreeType.MINUS, '-', [int21])
	minus = Tree(TreeType.MINUS, '-', [minus])
	minus = Tree(TreeType.MINUS, '-', [minus])
	exprs.append(minus)

	# a = 7 * 42
	int7 = Tree(TreeType.INTEGER_CONSTANT, '7')
	int42 = Tree(TreeType.INTEGER_CONSTANT, '42')
	star = Tree(TreeType.STAR, '*', [int7, int42])
	varA = Tree(TreeType.VARIABLE_NAME, 'a')
	assign = Tree(TreeType.ASSIGN, '=', [varA, star])
	exprs.append(assign)

	# a / 6
	varA = Tree(TreeType.VARIABLE_NAME, 'a')
	int6 = Tree(TreeType.INTEGER_CONSTANT, '6')
	slash = Tree(TreeType.SLASH, '/', [varA, int6])
	exprs.append(slash)

	module = Tree(TreeType.MODULE, '', exprs)

	return module

def main():
	# get an AST, should be replaced by a lexer + parser frontend
	ast = createSampleAST()

	codegen = CodeGen()
	module = codegen.walkAST(ast)
	print module

	# to run the generated code do:
	#     ./minicompiler.py | llvm-as | lli
	# or
	#     ./minicompiler.py > out.ll
	#     llvm-as out.ll
	#     lli out.bc
	# and to generate native code skip the lli above and then
	#     llc out.bc
	#     gcc out.s
	#     ./a.out

	# running the optimizer (only mem2reg pass) and print it human readable
	#     ./minicompiler | llvm-as | opt -mem2reg | llvm-dis

if __name__ == '__main__':
	main()

Results of mem2reg

Without the mem2reg pass the output should look like this:

; ModuleID = 'mymodule'
@__internalGlobalConst = internal constant [4 x i8] c"%dA0"         ;  [#uses=1]

declare i32 @printf(i8*, ...)

define void @printInt(i32) {
bb:
        %1 = call i32 (i8*, ...)* @printf(i8* getelementptr ([4 x i8]* @__internalGlobalConst, i32 0, i32 0), i32 %0)           ;  [#uses=0]
        ret void
}

define i32 @main() {
entry:
        %a = alloca i32         ;  [#uses=2]
        br label %bb

bb:             ; preds = %entry
        call void @printInt(i32 0)
        call void @printInt(i32 42)
        call void @printInt(i32 -21)
        store i32 294, i32* %a
        call void @printInt(i32 294)
        %0 = load i32* %a               ;  [#uses=1]
        %1 = sdiv i32 %0, 6             ;  [#uses=1]
        call void @printInt(i32 %1)
        ret i32 0
}

As you can see there’s an alloca in the entry block and store / load instructions. These should be replaced after an mem2reg pass:

; ModuleID = ''
@__internalGlobalConst = internal constant [4 x i8] c"%dA0"         ;  [#uses=1]

declare i32 @printf(i8*, ...)

define void @printInt(i32) {
bb:
        %1 = call i32 (i8*, ...)* @printf(i8* getelementptr ([4 x i8]* @__internalGlobalConst, i32 0, i32 0), i32 %0)           ;  [#uses=0]
        ret void
}

define i32 @main() {
entry:
        br label %bb

bb:             ; preds = %entry
        call void @printInt(i32 0)
        call void @printInt(i32 42)
        call void @printInt(i32 -21)
        call void @printInt(i32 294)
        %0 = sdiv i32 294, 6            ;  [#uses=1]
        call void @printInt(i32 %0)
        ret i32 0
}

Looks better!

In the next post I’ll show how the semantic and code generation phase in Exoself interact to solve some more advanced problems.

Read more here: Compiler Writing Series

Walking the AST

In the last post I’ve already described that the AST is traversed in postorder. The concrete implementation can be done in several ways. One of them is the visitor pattern. Here an external class (or class hierarchy) implements “visit” methods. These are overloaded depending on the type of AST node they should process. Since overloading is not possible in this way in Python I’m using a small variation of this pattern.

As there is no compiler I have to do the dispatching myself using a type – method mapping. The next step is unpacking the AST nodes. This could be done by using specialized AST classes for every node type or just implementing this functionality in a AST walker class. I choose the latter one.

The concrete idea is to call a dispatch method when you need to process a node. This dispatch method determines the right method to call, unpacks the AST and passes all needed arguments to the callee. The dispatch method also maintains a list of parent nodes. This makes accessing things like symbol tables easier and is simpler to implement than maintaining a parent attribute in each node – which would make AST transformation more difficult.

A minimalistic compiler for mathematical expressions

After trying to explain the code generation using my own compiler for Exoself I realized that a small self contained example would be better to get a start. So here is a compiler for simple mathematical expressions.

In the source code you have to manually define an AST that the code generator should transform to executable code. Every expression is evaluated and the result printed to stdout. I’ve added many comments explaining what I’m doing.

Since the code is a bit too wide for the current page layout click on ‘view plain’ or copy it into your favorite text editor to read it.

#!/usr/bin/python
import sys

# sys.path.append('path/to/llvm-py') # you may need to setup the path to llvm-py explicitly.

from llvm.core import *
import copy

# Tree is used for AST storage
class Tree(object):
	def __init__(self, type, text, children=None):
		self.type = type
		self.text = text

		if children:
			self.children = children
		else:
			self.children = [] # do not use a default value of children = [] in the function param list!

	def copy(self, withChildren):
		if withChildren:
			return copy.deepcopy(self)
		else:
			c = copy.copy(self)
			c.children = []
			return c

# IDs for the different AST types
class TreeType(object):
	MODULE = 1

	PLUS = 10
	MINUS = 11
	STAR = 12
	SLASH = 13

	INTEGER_CONSTANT = 100

# base class for traversing an AST. Implements dispatch for the above defined node types
class ASTWalker(object):
	def __init__(self):
            pass

	def walkAST(self, ast):
		raise NotImplementedError('subclasses must implement walkAST')

        # dispatch the node to the right method after unpacking it
	def _dispatch(self, ast):
		tt = TreeType

		kwargs = {}
		kwargs['ast'] = ast

		if ast.type == tt.MODULE:
			callee = self._onModule
			kwargs['statements'] = ast.children
		elif ast.type in [tt.PLUS, tt.MINUS, tt.STAR, tt.SLASH]:
			callee = self._onOperator

			op = ast.type
			if len(ast.children) == 1:
				arg1 = ast.children[0]
				arg2 = None
			elif len(ast.children) == 2:
				arg1 = ast.children[0]
				arg2 = ast.children[1]
			else:
				assert(0 and 'dead code path')

			kwargs['op'] = op
			kwargs['arg1'] = arg1
			kwargs['arg2'] = arg2
		elif ast.type == tt.INTEGER_CONSTANT:
			callee = self._onIntegerConstant
			kwargs['value'] = int(ast.text)
		else:
			assert(0 and 'dead code path')

		callee(**kwargs)

class CodeGen(ASTWalker):
	def __init__(self, *k, **kw):
		ASTWalker.__init__(self, *k, **kw)

	def walkAST(self, ast):
		self._dispatch(ast)

		return self._module

	def _addHelpers(self):
		# you might want to skip this function on a first read. It adds essentially a function to print integers to stdout.

		# add a prototype for printf
		# int printf(char*, ...)
		funcType = Type.function(Type.int(32), [Type.pointer(Type.int(8))], True)
		printf = self._module.add_function(funcType, 'printf')

		# add a function to print integers to stdout using printf
		# void printInt(int x) { printf("%d\n", x); }
		funcType = Type.function(Type.void(), [Type.int(32)])
		printInt = self._module.add_function(funcType, 'printInt')
		self._printInt = printInt # save for later use in _onModule

		# create a block and a builder for printInt
		bb = printInt.append_basic_block('bb')
		b = Builder.new(bb)

		# create a global constant to hold the first argument of printf
		stringConst = Constant.stringz('%d\n') # zero terminated --> stringz instead of string
		string = self._module.add_global_variable(stringConst.type, '__internalGlobalConst')
		string.initializer = stringConst
		string.global_constant = True
		string.linkage = LINKAGE_INTERNAL # not strictly necessary here, but this global should only be available during link time in the current module

		# address calculation
		# every index traverses a pointer without derefencing.
		# gep (get element pointer) does only address calculation, no memony accesses!
		idx = [Constant.int(Type.int(32), 0), Constant.int(Type.int(32), 0)] # the first index get's us past the global variable (which is a pointer) to the string; the second index is the offset inside the string we want to access
		realAddr = string.gep(idx) # get real address

		# call printf
		b.call(printf, [realAddr, printInt.args[0]])
		b.ret_void()

	def _onModule(self, ast, statements):
		# create a new LLVM module, a container for global variables and functions
		self._module = Module.new('mymodule')

		# add some helpers
		self._addHelpers()

		# add a function: 'int main()'
		funcType = Type.function(Type.int(32), [])
		func = self._module.add_function(funcType, 'main')

		# create an 'entry' basic block, if we want to introduce variables we'll need it anyway
		# execution of the function starts here
		entryBB = func.append_basic_block('entry')
		entryBuilder = Builder.new(entryBB)

		# normal code should be inserted into the second block
		bb = func.append_basic_block('bb')

		# also jump to this block from the entry block
		entryBuilder.branch(bb)

		# add the code of the function
		self._currentBuilder = Builder.new(bb)
		for x in statements:
			self._dispatch(x)
			self._currentBuilder.call(self._printInt, [x.llvmValue])

		# main should return 0
		self._currentBuilder.ret(Constant.int(Type.int(32), 0))

		# verify the module
		self._module.verify()

	def _onOperator(self, ast, op, arg1, arg2):
		tt = TreeType

		self._dispatch(arg1)
		if arg2:# some operators are unary
			self._dispatch(arg2)

		cb = self._currentBuilder
		if op == TreeType.PLUS:
			if arg2:
				ast.llvmValue = cb.add(arg1.llvmValue, arg2.llvmValue)
			else:
				ast.llvmValue = arg1.llvmValue # +NUMBER == NUMBER
		elif op == TreeType.MINUS:
			if arg2:
				ast.llvmValue = cb.sub(arg1.llvmValue, arg2.llvmValue)
			else:
				ast.llvmValue = cb.sub(Constant.int(Type.int(32), 0), arg1.llvmValue) # -NUMBER == 0 - NUMBER
		elif op == TreeType.STAR:
			ast.llvmValue = cb.mul(arg1.llvmValue, arg2.llvmValue)
		elif op == TreeType.SLASH:
			ast.llvmValue = cb.sdiv(arg1.llvmValue, arg2.llvmValue)
		else:
			assert(0 and 'dead code path')

	def _onIntegerConstant(self, ast, value):
		ast.llvmValue = Constant.int(Type.int(32), value)

def createSampleAST():
	# do not reuse any variables! overwrite them first!
	exprs = []

	# 9 - 3 * 3
	int9 = Tree(TreeType.INTEGER_CONSTANT, '9')
	int3a = Tree(TreeType.INTEGER_CONSTANT, '3')
	int3b = Tree(TreeType.INTEGER_CONSTANT, '3')

	star = Tree(TreeType.STAR, '*', [int3a, int3b])
	minus = Tree(TreeType.MINUS, '-', [int9, star])
	exprs.append(minus)

	# 4 + 76 / 2
	int4 = Tree(TreeType.INTEGER_CONSTANT, '4')
	int76 = Tree(TreeType.INTEGER_CONSTANT, '76')
	int2 = Tree(TreeType.INTEGER_CONSTANT, '2')

	slash = Tree(TreeType.SLASH, '/', [int76, int2])
	plus = Tree(TreeType.PLUS, '+', [int4, slash])
	exprs.append(plus)

	# ---21
	int21 = Tree(TreeType.INTEGER_CONSTANT, '21')
	minus = Tree(TreeType.MINUS, '-', [int21])
	minus = Tree(TreeType.MINUS, '-', [minus])
	minus = Tree(TreeType.MINUS, '-', [minus])
	exprs.append(minus)

	module = Tree(TreeType.MODULE, '', exprs)

	return module

def main():
	# get an AST, should be replaced by a lexer + parser frontend
	ast = createSampleAST()

	codegen = CodeGen()
	module = codegen.walkAST(ast)
	print module

	# to run the generated code do:
	#     ./minicompiler.py | llvm-as | lli
	# or
	#     ./minicompiler.py > out.ll
	#     llvm-as out.ll
	#     lli out.bc
	# and to generate native code skip the lli above and then
	#     llc out.bc
	#     gcc out.s

	#     ./a.out

if __name__ == '__main__':
	main()

So the basic idea of code generation is:

• dispatch node X
• dispatch all dependencies of X
• emit code for X
• and if applicable store the value of the computation in X for access by parent nodes

When you directly look at the output of the compiler you’ll see that all expressions are evaluated even before the code is executed. Since we are using only constants LLVM uses a constant folder to optimize these operations away. There is a “nofolder” implementation available in LLVM 2.4, but I don’t know how to access it through Python or the C wrapper.

As you have seen this compiler consists only of a single phase: Code generation. Try to add a lexer and parser to avoid hard coding the AST and post a comment with your solution.
If you don’t want to use ANTLR or try something new I’d love to hear about your experiences with waxeye.

Read more here: Compiler Writing Series

I don’t want to start a flame war here. Obviously it’s how I feel about the following languages. Take it with a grain of salt, it probably sounds much worse than it really is.

Existing languages

The core languages of my tool set are C, C++ and Python. Python when it’s not performance critical, C++ if performance matters and for kernel modules obviously C. Some other languages I’ve used are D, Javascript, Java, PHP. I’ve tried out many other languages, to name a few: Lua, Boo, Ruby, Haskell, Erlang, io and even x86 assembler.

So ideally I want a language that is as easy to use as Python but with the performance of C / C++. That’s not really possible since dynamic typing incurs a certain overhead. Using a JIT would be sometimes an option, but Psyco is not available for x86_64. I know there are many alternatives to Psyco such as Shed Skin and some other tools that allow me to embed C / C++ or even annotate types for static compilation. Still no real solution for me.

Why don’t I just extend Python with C++? When you have to access the raw data on both sides of the language barrier that’s bad. Copying it constantly both ways is just too expensive.

(There are designs were this is not an issue, like the application interface of Horde3D – you should really take a look at it; but trying to split the data processing and visualization implementation across the language barrier didn’t work out for me.)

A language which tries to combine the best features of static and dynamic typing is Boo. In theory it would be my dream language since it’s a mix of Python and C++. The main problem again was performance. One part being Mono and another the implementation of Boo itself.

Without going into too much detail I don’t like Java with it’s very verbose style for even simple tasks and why did they forbid operator overloading?

Another very interesting language is D. Or I probably should say languages, since D 1.0 is “stable” and D 2.0 a moving target. I tried D 1.0 and realized that there are just to many quirks with it for me. I’ll pick a few examples: You have two, yes two, standard libraries. And why provide a unicode character type (a builtin) when it’s indices are not character indices but bytes?

Functional languages like Haskell and Erlang don’t suit my thinking model. The so called benefits of side effect free functions and pattern matching are my major deal breaker. Instead I’ll try out D 2.0, when it’s finally stable, to work with transitive const / invariant.

Regarding C / C++: These are old(er) languages and I see several problems with them. The module system is far from optimal as I have to maintain headers which should really be the job of the compiler. The type system in C is too weak and C++ has template programming with unimaginable complexity. Another problem with C++ are the very long compile times even without optimization for template heavy programs. Many many more reasons – but I still feel at home when using them as C++ was my first programming language.

In conclusion there is no perfect language for me. I don’t know if that will ever change since Exoself will certainly have some quirks too – but I at least have the option to make it a better fit.

Other reasons

The short version is that I’m fascinated by how computers work. Be it the inner workings of a CPU or even how a transistor works or on a higher level what happens during a “simple” open call to the glibc. Ever read the code from the headers right down to the syscall implementation? Or have you written a kernel module?

After learning about LLVM more than a year ago I constantly followed it’s progress but without any concrete plans to write my own language. But that changed as you can see.

So my primary motivation for writing a new programming language is having some fun constructing a compiler and runtime. That involves solving some complex problems, but that’s all the more intriguing. If anyone is interested in the results of this project, even better.

Choosing a name

The name originates in a book I’ve read: Diaspora, by Greg Egan. There the term Exoself describes the systems that work together to extend mind (and body) but are not necessarily part of either. A programming language should exactly do that: Extend your mind – but is not part of it in any way.

Another reason for this name was that you can search for it using your favorite search engine. Try that with D or any other one letter language.

Design goals / buzz word compliance

• ease of use, productivity
• garbage collected by default, but opt out by simply using C malloc
• type inferencing for variables, type information available outside of compiler
• statically typed, stronger typed than C++
• performance similar to C
• fast compile times
• C style syntax, maybe later a mode for significant whitespace
• must be able to call C libraries directly, ideally with no overhead
• module system
• multi core aware; no concrete ideas so far, but many options how it could work:
  • Cilk++
  • OpenMP
  • TBB
  • transitive const / invariant
• make parsing easy enough, especially for IDE developers
• no preprocessor

I have some more ideas like a REPL or compile time function evaluation.
Regarding the type information outside the compiler: The compiler itself should be able to provide IDEs with type information. Without incremental parsing that’s not really fast but will have to do for now. It’s better than nothing and is IDE independant.

These design goals enforce several limitations to what Exoself will become. As an example the ability to call C directly comes at the cost of using the same base types as C, no actor concurrency like Erlang, no accurate or moving garbage collection and many more details.

Programming model

The perfect language would support an do whatever you want style. That’s not feasible to implement, so for now Exoself will be an imperative / procedural programming language similar to C. The main idea is to get a working programming language and adding objects makes this right now more complicated than is strictly necessary.

I’ll definitely add objects later. Maybe after a 1.0 release, maybe before. It really depends on what I learn about language design while writing Exoself. For now I’m only planning how to add objects to the language without influencing other parts of the language too much. Essentially this means the language will consist of two seamlessly cooperating parts. One for procedural programming, the other for object oriented programming. This is similar to D’s struct and class distinction but taken a step further.

The idea behind this split is that raw performance is only guaranteed by the procedural part. If you need it, it’s available for you. But if execution speed does not matter that much prefer ease of use. For many tasks it’s good enough to have an Integer class that supports arbitrary integers while being slow(er) instead of having a lightning fast int / long. Just look at Python – it really works!

The documentation for Exoself will include a detailed language specification. But writing that takes much time and is not as interesting as writing code to implement new features. For now I have only many notes where I tried out different ideas and discuss things like effects of garbage collection on the ability to call C functions…

Let’s start simple. What is wrong with the following top level code?

def f(x as int32) as int32
{
    if x == 0
    {
        break;
    }
    return 42;
}

What does the break statement there? It’s not contained in a loop (or switch) so by using it in this context we are violating the rule that a break must be associated with a loop. But nevertheless this code will parse without any problems.

Another important job of this phase, at least in statically compiled languages, is to gather type information for the code generator. Also the type checking is implemented in this phase.

x as int32;
x = 4.0;

While traversing the AST we see that a symbol named x is defined as an int32. After inserting this information into a symbol table we continue with the next statement.

Now we should assign the value 4.0, which is of type float64, to the symbol x. After a lookup in the symbol table we know that x is of type int32. Now the type system has to decide if this is a valid language construct. In Exoself this construct is not valid: There are no implicit conversions from any floating point type to an integer type. As a result the type checker will emit an error.

So what we have done here is walking the AST in postorder: Before evaluating any node all its children must be evaluated. In Exoself after determining the type of an AST node I add the attribute “esType” to that node. So every parent node can easily access the type information.

Here’s the code I’m using for the while statement. A while statement consists of it’s own AST node (ast), the expression to evaluate (expression) and a block to execute. The expression must be of type bool or there must exist an implicit cast to bool.

    def _onWhile(self, ast, expression, block):
        self._dispatch(expression)

        esType = expression.esType
        bool = self._findSymbol(name=u'bool', type_=ESType)
        if not esType.isEquivalentTo(bool, False):
            # types did not match, try implicit cast
            if not estypesystem.canImplicitlyCast(esType, bool):
                self._raiseException(RecoverableCompileError, tree=expression, inlineText='incompatible type, expected bool')

The first thing I’m doing is calling the correct handler for the expression using dispatch. Then I’m retrieving the type from the expression AST and look up the bool type in the symbol table.

The next step is more interesting: I’m trying to find out if the type of the expression is identical to bool. In this case I’m doing a name based comparison instead of a structure based comparison.

This check will very often fail since programmers are too lazy to cast the expression manually to bool in while statements. Therefore my code tries to insert an implicit cast to bool. If that fails we raise an exception which in turn will print an error message what exactly went wrong and why.

An example that is a bit more difficult is the assignment statement. Lets look at its code, too:

    def _onAssignHelper(self, assigneeExpr, exprNode):
        self._dispatch(exprNode)
        esType = exprNode.esType

        # FIXME make assigneeExpr eval more general and move it to astwalker!
        if assigneeExpr.type == TreeType.VARIABLE:
            varNameNode = assigneeExpr.children[0]
            var = self._findSymbol(fromTree=varNameNode, type_=ESVariable, mayFail=True)

            if not var:
                # create new variable with type of expression
                var = ESVariable(varNameNode.text, esType)
                self._addSymbol(fromTree=varNameNode, symbol=var)
            else:
                if not var.esType.isEquivalentTo(esType, False):
                    self._insertImplicitCastNode(exprNode, var.esType)
        elif assigneeExpr.type == TreeType.DEREFERENCE:
            self._dispatch(assigneeExpr)

            if not assigneeExpr.esType.isEquivalentTo(esType, False):
                self._insertImplicitCastNode(exprNode, assigneeExpr.esType)
        else:
            print assigneeExpr.text
            raise NotImplementedError('TODO')

As you can see, still work in progress. (_onAssign, the real handler for simple assignments, is just a thin wrapper around this function to avoid code duplication)

I start again by determining the type of the expression. Then we have to be careful: Variables can be defined by assignment in Exoself, but this makes no sense for pointer expressions.

If the assignee is a variable we try to find the associated symbol, but that may fail. If the variable does not yet exist we add an entry to the symbol table with the type of the expression. As you can see type inferencing is trivial on this level and more languages should provide it. Otherwise, if the variable was already defined check if types match and if not try to insert an implicit cast. In case of the pointer expression just dispatch it to determine its type and compare it in the same manner.

Type system requirements

You now know how to type check – at least in principle. But how can wee determine if two types are the same? There are two ways to type checking ignoring any details.

The first way is to use name based equivalence. The types of two structs are only considered equal in C if they have the same name. The contents are completely ignored:

struct X {int a};
struct Y {int b};
struct X x;
struct Y y;
x = y;// error, x not of type struct Y

The other way to compare data types is by their structure. Would C use structural equivalence for structs instead, the above assignment would be legal. That’s quite confusing since the type system would not prevent many errors. It’s important to understand that the names of variables do not matter for structural equivalence, only their types and order in structs.

All primitive data types of C are compared using structural equivalence. If you introduce a typedef for int called MyInt there’s no way to enforce anyone to use the type MyInt instead of a plain int. Let’s call this an alias.

Regarding the type system of Exoself, which is similar to that of D, I want to use name based equivalence by default but with the option to introduce an alias. The alias statement introduces another name which is equivalent to the old one (even with name based comparison) and the typedef statement introduces a new type which is only structurally equivalent to the old one.

Another requirement for the type system of Exoself was to be able to represent const / invariable data, even if I’m not using that in the near future. Pointers and structs make this again a bit complicated but there’s an elegant solution to this problem.

Type representation

The basic idea is to represent any type by a tree.

x as int32

x as int32*

alias int as int32

typedef byte as uint8

By introducing the part with the typedef I’m breaking structural equivalence to the original type.

struct X {a as int32; b as float32}

Again the struct part breaks structural equivalence and order of the fields is of course relevant.

So we need to define a class which can store this relationship: On a first read you might want to skip over it.

class ESType(object):
	''' represents types of data, not variables! '''

	def __init__(self, parents, payload):
		''' do not call directly! use construction methods '''
		assert(isinstance(parents, list))
		for x in parents:
			assert(isinstance(x, ESType))
		self.parents = parents
		self.payload = payload

	def derivePointer(self):
		return ESType([self], ('pointer', None))

	def dereference(self):
		assert(self.isPointer())

		return self.parents[0]

	def deriveTypedef(self, name):
		# break structural equivalence
		return ESType([self], ('typedef', name))

	@staticmethod
	def createStruct(name, parts):
		return ESType(parts, ('struct', name))

	@staticmethod
	def createFunction(returnTypes, paramTypes):
		assert(len(returnTypes) >= 1)
		parts = []
		parts.extend(returnTypes)
		parts.extend(paramTypes)
		return ESType(parts, ('function', len(returnTypes)))

	@staticmethod
	def createSelfPointer():
		''' only valid inside structs! '''
		return ESType([], ('selfpointer', None))

	def isEquivalentTo(self, other, structural):
		# structural equivalence: Skip any typedefs and ignore different struct names

		t1 = self
		t2 = other

		if structural:
			while t1.payload[0] == 'typedef':
				assert(len(t1.parents) == 1)
				t1 = t1.parents[0]

			while t2.payload[0] == 'typedef':
				assert(len(t2.parents) == 1)
				t2 = t2.parents[0]

		if structural and t1.payload[0] == 'struct' and t2.payload[0] == 'struct':
			pass
		elif t1.payload != t2.payload:
			return False

		if len(t1.parents) != len(t2.parents):
			return False

		for i in range(len(t1.parents)):
			if not t1.parents[i].isEquivalentTo(t2.parents[i], structural):
				return False

		return True

	def __eq__(self, other):
		raise NotImplementedError('use isEquivalentTo')

	def __ne__(self, other):
		raise NotImplementedError('use isEquivalentTo')

	def isPointer(self):
		p = self
		while p.payload[0] == 'typedef':
			p = p.parents[0]

		return p.payload[0] == 'pointer'

Essentially I’ve defined a class that has two members. A ordered list of parents and a payload. The payload is a tuple exactly as in the images above denoting the type of the current node. The parents member is a list containing references to other ESType instances to represent the tree.

The deriveX methods can create new types from existing ones. By calling derivePointer on an existing instance a new instance of ESType will be created with the payload (‘pointer’, None) and the parent of this new instance will be the current one. That way we can represent a pointer to an arbitrary type. That’s only almost correct as we’ll see in a moment.

Since structs need multiple parents I’ve added a static method to ESType to do this work for me. Just pass all elements in the order they should be defined. But what happens with structs containing pointers to themselves? That would create a loop in the tree and that’s not a good idea.

I’m using a trick here: I provide a special type for self pointers. These are valid only inside structs and get later translated to the real type.

Now checking if two types are equal is easy: Just compare the payloads and all parents recursively until something doesn’t match. Under certain circumstances you’ll still need structural equivalence. For example when assigning a variable of type byte a uint8 constant using an explicit cast. The cast will do nothing since the structures are equivalent. So while comparing just skip typedef nodes and anything containing names.

more AST annotation

Now we have an AST with type information but there are still some things missing. An important data structure that I already mentioned is the symbol table. In Exoself every pair of curly braces introduces a new scope. Variables defined inside a scope are not available outside a scope just as in C. Some other AST nodes like the module itself, function definitions and for loops also introduce a scope.

So all these AST nodes must have a symbol table. This could be as simple as a dict mapping symbol names to instances of variable / function classes. But the lookup of symbols is a bit more complicated. You have to search all symbol tables recursively.

My idea to solve this problem was to maintain a list of all parents of the current node. By traversing through this list starting at the end we move with every step towards the root node of the AST. Using the hasattr function in Python we can simply check if the current AST node has a symbol table or not. If there is one, search it for the requested symbol. If the symbol was found we can stop the search and return the result, otherwise continue.

Adding symbols is done by searching for the nearest symbol table. Additionally Exoself forbids shadowing of variables, so an additional search is done to check if that name is already used.

In the next post I’ll describe the code generation phase using LLVM and llvm-py.

Read more here: Compiler Writing Series