======================================================= Generating Python Bytecode with ``peak.util.assembler`` ======================================================= ``peak.util.assembler`` is a simple bytecode assembler module that handles most low-level bytecode generation details like jump offsets, stack size tracking, line number table generation, constant and variable name index tracking, etc. That way, you can focus your attention on the desired semantics of your bytecode instead of on these mechanical issues. In addition to a low-level opcode-oriented API for directly generating specific bytecodes, the module also offers an extensible mini-AST framework for generating code from high-level specifications. This framework does most of the work needed to transform tree-like structures into linear bytecode instructions, and includes the ability to do compile-time constant folding. .. contents:: Table of Contents -------------- Programmer API -------------- Code Objects ============ To generate bytecode, you create a ``Code`` instance and perform operations on it. For example, here we create a ``Code`` object representing lines 15 and 16 of some input source:: >>> from peak.util.assembler import Code >>> c = Code() >>> c.set_lineno(15) # set the current line number (optional) >>> c.LOAD_CONST(42) >>> c.set_lineno(16) # set it as many times as you like >>> c.RETURN_VALUE() You'll notice that most ``Code`` methods are named for a CPython bytecode operation, but there also some other methods like ``.set_lineno()`` to let you set the current line number. There's also a ``.code()`` method that returns a Python code object, representing the current state of the ``Code`` you've generated:: >>> from dis import dis >>> dis(c.code()) 15 0 LOAD_CONST 1 (42) 16 3 RETURN_VALUE As you can see, ``Code`` instances automatically generate a line number table that maps each ``set_lineno()`` to the corresponding position in the bytecode. And of course, the resulting code objects can be run with ``eval()`` or ``exec``, or used with ``new.function`` to create a function:: >>> eval(c.code()) 42 >>> exec c.code() # exec discards the return value, so no output here >>> import new >>> f = new.function(c.code(), globals()) >>> f() 42 Opcodes, Jumps, and Labels ========================== ``Code`` objects have methods for all of CPython's symbolic opcodes. Generally speaking, each method accepts either zero or one argument, depending on whether the opcode accepts an argument. But while Python bytecode always encodes arguments as 16 or 32-bit integers, you will generally pass actual names or values to ``Code`` methods, and the ``Code`` object will take care of maintaining the necessary lookup tables and translation to integer bytecode arguments. Labels and backpatching forward references:: >>> c = Code() >>> ref = c.JUMP_ABSOLUTE() # jump w/unspecified target >>> c.LOAD_CONST(1) >>> ref() # resolve the forward reference >>> c.RETURN_VALUE() >>> dis(c.code()) 0 0 JUMP_ABSOLUTE 6 3 LOAD_CONST 1 (1) >> 6 RETURN_VALUE >>> c = Code() >>> lbl = c.label() # create a label at this point in the code >>> c.LOAD_CONST(1) >>> ref = c.JUMP_ABSOLUTE(lbl) # and jump to it >>> dis(c.code()) 0 >> 0 LOAD_CONST 1 (1) 3 JUMP_ABSOLUTE 0 Code Generation API =================== Code generation from tuples, lists, dicts, and local variable names:: >>> from peak.util.assembler import Const, Call, Global, Local >>> c = Code() >>> c( [Local('x'), (Local('y'),Local('z'))] ) # push a value on the stack >>> dis(c.code()) 0 0 LOAD_FAST 0 (x) 3 LOAD_FAST 1 (y) 6 LOAD_FAST 2 (z) 9 BUILD_TUPLE 2 12 BUILD_LIST 2 And with constants, dictionaries, globals, and calls:: >>> c = Code() >>> c.return_( [Global('type'), Const(27)] ) # push and RETURN_VALUE >>> dis(c.code()) 0 0 LOAD_GLOBAL 0 (type) 3 LOAD_CONST 1 (27) 6 BUILD_LIST 2 9 RETURN_VALUE >>> c = Code() >>> c( {Const('x'): Const(123)} ) >>> dis(c.code()) 0 0 BUILD_MAP 0 3 DUP_TOP 4 LOAD_CONST 1 ('x') 7 LOAD_CONST 2 (123) 10 ROT_THREE 11 STORE_SUBSCR >>> c = Code() >>> c(Call(Global('type'), (Const(1),))) >>> dis(c.code()) 0 0 LOAD_GLOBAL 0 (type) 3 LOAD_CONST 1 (1) 6 CALL_FUNCTION 1 >>> c = Code() >>> c(Call(Global('getattr'), (Const(1), Const('__class__')))) >>> dis(c.code()) 0 0 LOAD_GLOBAL 0 (getattr) 3 LOAD_CONST 1 (1) 6 LOAD_CONST 2 ('__class__') 9 CALL_FUNCTION 2 ``Call`` objects take 1-4 arguments: the expression to be called, a sequence of positional arguments, a sequence of keyword/value pairs for explicit keyword arguments, an "*" argument, and a "**" argument. To omit any of the optional arguments, just pass in an empty sequence in its place:: >>> c = Code() >>> c.return_( ... Call(Global('foo'), [Local('q')], [('x',Const(1))], ... Local('starargs'), Local('kwargs')) ... ) >>> dis(c.code()) 0 0 LOAD_GLOBAL 0 (foo) 3 LOAD_FAST 0 (q) 6 LOAD_CONST 1 ('x') 9 LOAD_CONST 2 (1) 12 LOAD_FAST 1 (starargs) 15 LOAD_FAST 2 (kwargs) 18 CALL_FUNCTION_VAR_KW 257 21 RETURN_VALUE Code generation is extensible: any 1-argument callables passed in will be passed the code object during generation. (The return value, if any, is ignored.) You can even use ``Code`` methods, if they don't have any required arguments:: >>> c = Code() >>> c.LOAD_GLOBAL('foo') >>> c(Call(Code.DUP_TOP, ())) >>> dis(c.code()) 0 0 LOAD_GLOBAL 0 (foo) 3 DUP_TOP 4 CALL_FUNCTION 0 This basically means you can create a simple AST of callable objects to drive code generation, with a lot of the grunt work automatically handled for you. Setting the Code's Calling Signature ==================================== The simplest way to set up the calling signature for a ``Code`` instance is to clone an existing function or code object's signature, using the ``Code.from_function()`` or ``Code.from_code()`` classmethods. These methods create a new code object whose calling signature (number and names of arguments) matches that of the original function or code objects:: >>> def f1(a,b,*c,**d): ... pass >>> c1 = Code.from_function(f1) >>> c1.co_argcount 2 >>> c1.co_varnames ['a', 'b', 'c', 'd'] >>> import inspect >>> inspect.getargspec(f1) (['a', 'b'], 'c', 'd', None) >>> f2 = new.function(c1.code(), globals()) >>> inspect.getargspec(f2) (['a', 'b'], 'c', 'd', None) Note that these constructors do not copy any actual *code* from the code or function objects. They simply copy the signature, and, if you set the ``copy_lineno`` keyword argument to a true value, they will also set the created code object's ``co_firstlineno`` to match that of the original code or function object:: >>> c1 = Code.from_function(f1, copy_lineno=True) >>> c1.co_firstlineno 1 If you create a ``Code`` instance from a function that has nested positional arguments, the returned code object will include a prologue to unpack the arguments properly:: >>> def f3(a, (b,c), (d,(e,f))): ... pass >>> f4 = new.function(Code.from_function(f3).code(), globals()) >>> dis(f4) 0 0 LOAD_FAST 1 (.1) 3 UNPACK_SEQUENCE 2 6 STORE_FAST 3 (b) 9 STORE_FAST 4 (c) 12 LOAD_FAST 2 (.2) 15 UNPACK_SEQUENCE 2 18 STORE_FAST 5 (d) 21 UNPACK_SEQUENCE 2 24 STORE_FAST 6 (e) 27 STORE_FAST 7 (f) This is roughly the same code that Python would generate to do the same unpacking process, and is designed so that the ``inspect`` module will recognize it as an argument unpacking prologue:: >>> inspect.getargspec(f3) (['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None) >>> inspect.getargspec(f4) (['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None) Code Attributes =============== ``Code`` instances have a variety of attributes corresponding to either the attributes of the Python code objects they generate, or to the current state of code generation. For example, the ``co_argcount`` and ``co_varnames`` attributes correspond to those used in creating the code for a Python function. If you want your code to be a function, you can set them as follows:: >>> c = Code() >>> c.co_argcount = 3 >>> c.co_varnames = ['a','b','c'] >>> c.LOAD_CONST(42) >>> c.RETURN_VALUE() >>> f = new.function(c.code(), globals()) >>> f(1,2,3) 42 >>> import inspect >>> inspect.getargspec(f) (['a', 'b', 'c'], None, None, None) Although Python code objects want ``co_varnames`` to be a tuple, ``Code`` instances use a list, so that names can be added during code generation. The ``.code()`` method automatically creates tuples where necessary. Here are all of the ``Code`` attributes you may want to read or write: co_filename A string representing the source filename for this code. If it's an actual filename, then tracebacks that pass through the generated code will display lines from the file. The default value is ``''``. co_name The name of the function, class, or other block that this code represents. The default value is ``''``. co_argcount Number of positional arguments a function accepts; defaults to 0 co_varnames A list of strings naming the code's local variables, beginning with its positional argument names, followed by its ``*`` and ``**`` argument names, if applicable, followed by any other local variable names. These names are used by the ``LOAD_FAST`` and ``STORE_FAST`` opcodes, and invoking the ``.LOAD_FAST(name)`` and ``.STORE_FAST(name)`` methods of a code object will automatically add the given name to this list, if it's not already present. co_flags The flags for the Python code object. This defaults to ``CO_OPTIMIZED | CO_NEWLOCALS``, which is the correct value for a function using "fast" locals. This value is automatically or-ed with ``CO_NOFREE`` when generating a code object, if the ``co_cellvars`` and ``co_freevars`` attributes are empty. And if you use the ``LOAD_NAME()``, ``STORE_NAME()``, or ``DELETE_NAME()`` methods, the ``CO_OPTIMIZED`` bit is automatically reset, since these opcodes can only be used when the code is running with a real (i.e. not virtualized) ``locals()`` dictionary. If you need to change any other flag bits besides the above, you'll need to set or clear them manually. For your convenience, the ``peak.util.assembler`` module exports all the ``CO_`` constants used by Python. For example, you can use ``CO_VARARGS`` and ``CO_VARKEYWORDS`` to indicate whether a function accepts ``*`` or ``**`` arguments, as long as you extend the ``co_varnames`` list accordingly. (Assuming you don't have an existing function or code object with the desired signature, in which case you could just use the ``from_function()`` or ``from_code()`` classmethods instead of messing with these low-level attributes and flags.) stack_size The predicted height of the runtime value stack, as of the current opcode. Its value is automatically updated by most opcodes, but you may want to save and restore it for things like try/finally blocks. co_freevars A tuple of strings naming a function's "cell" variables. Defaults to an empty tuple. A function's free variables are the variables it "inherits" from its surrounding scope. If you're going to use this, you should set it only once, before generating any code that references any free *or* cell variables. co_cellvars A tuple of strings naming a function's "cell" variables. Defaults to an empty tuple. A function's cell variables are the variables that are "inherited" by one or more of its nested functions. If you're going to use this, you should set it only once, before generating any code that references any free *or* cell variables. These other attributes are automatically generated and maintained, so you'll probably never have a reason to change them: co_consts A list of constants used by the code; the first (zeroth?) constant is always ``None``. Normally, this is automatically maintained; the ``.LOAD_CONST(value)`` method checks to see if the constant is already present in this list, and adds it if it is not there. co_names A list of non-optimized or global variable names. It's automatically updated whenever you invoke a method to generate an opcode that uses such names. co_code A byte array containing the generated code. Don't mess with this. co_firstlineno The first line number of the generated code. It automatically gets set if you call ``.set_lineno()`` before generating any code; otherwise it defaults to zero. co_lnotab A byte array containing a generated line number table. It's automatically generated, so don't mess with it. co_stacksize The maximum amount of stack space the code will require to run. This value is usually updated automatically as you generate code. ---------------------- Internals and Doctests ---------------------- Line number tracking:: >>> def simple_code(flno, slno, consts=1, ): ... c = Code() ... c.set_lineno(flno) ... for i in range(consts): c.LOAD_CONST(None) ... c.set_lineno(slno) ... c.RETURN_VALUE() ... return c.code() >>> dis(simple_code(1,1)) 1 0 LOAD_CONST 0 (None) 3 RETURN_VALUE >>> simple_code(1,1).co_stacksize 1 >>> dis(simple_code(13,414)) # FAILURE EXPECTED IN PYTHON 2.3 13 0 LOAD_CONST 0 (None) 414 3 RETURN_VALUE >>> dis(simple_code(13,14,100)) 13 0 LOAD_CONST 0 (None) 3 LOAD_CONST 0 (None) ... 14 300 RETURN_VALUE >>> simple_code(13,14,100).co_stacksize 100 >>> dis(simple_code(13,572,120)) # FAILURE EXPECTED IN Python 2.3 13 0 LOAD_CONST 0 (None) 3 LOAD_CONST 0 (None) ... 572 360 RETURN_VALUE Stack size tracking:: >>> c = Code() >>> c.LOAD_CONST(1) >>> c.POP_TOP() >>> c.LOAD_CONST(2) >>> c.LOAD_CONST(3) >>> c.co_stacksize 2 >>> c.BINARY_ADD() >>> c.LOAD_CONST(4) >>> c.co_stacksize 2 >>> c.LOAD_CONST(5) >>> c.LOAD_CONST(6) >>> c.co_stacksize 4 >>> c.POP_TOP() >>> c.stack_size 3 Stack underflow detection/recovery, and global/local variable names:: >>> c = Code() >>> c.LOAD_GLOBAL('foo') >>> c.stack_size 1 >>> c.STORE_ATTR('bar') # drops stack by 2 Traceback (most recent call last): ... AssertionError: Stack underflow >>> c.co_names # 'bar' isn't added unless success ['foo'] >>> c.LOAD_ATTR('bar') >>> c.co_names ['foo', 'bar'] >>> c.DELETE_FAST('baz') >>> c.co_varnames ['baz'] >>> dis(c.code()) 0 0 LOAD_GLOBAL 0 (foo) 3 LOAD_ATTR 1 (bar) 6 DELETE_FAST 0 (baz) Sequence operators and stack tracking: Function calls and raise:: >>> c = Code() >>> c.LOAD_GLOBAL('locals') >>> c.CALL_FUNCTION() # argc/kwargc default to 0 >>> c.POP_TOP() >>> c.LOAD_GLOBAL('foo') >>> c.LOAD_CONST(1) >>> c.LOAD_CONST('x') >>> c.LOAD_CONST(2) >>> c.CALL_FUNCTION(1,1) # argc, kwargc >>> c.POP_TOP() >>> dis(c.code()) 0 0 LOAD_GLOBAL 0 (locals) 3 CALL_FUNCTION 0 6 POP_TOP 7 LOAD_GLOBAL 1 (foo) 10 LOAD_CONST 1 (1) 13 LOAD_CONST 2 ('x') 16 LOAD_CONST 3 (2) 19 CALL_FUNCTION 257 22 POP_TOP >>> c = Code() >>> c.LOAD_GLOBAL('foo') >>> c.LOAD_CONST(1) >>> c.LOAD_CONST('x') >>> c.LOAD_CONST(2) >>> c.BUILD_MAP(0) >>> c.stack_size 5 >>> c.CALL_FUNCTION_KW(1,1) >>> c.POP_TOP() >>> c.stack_size 0 >>> c = Code() >>> c.LOAD_GLOBAL('foo') >>> c.LOAD_CONST(1) >>> c.LOAD_CONST('x') >>> c.LOAD_CONST(1) >>> c.BUILD_TUPLE(1) >>> c.CALL_FUNCTION_VAR(0,1) >>> c.POP_TOP() >>> c.stack_size 0 >>> c = Code() >>> c.LOAD_GLOBAL('foo') >>> c.LOAD_CONST(1) >>> c.LOAD_CONST('x') >>> c.LOAD_CONST(1) >>> c.BUILD_TUPLE(1) >>> c.BUILD_MAP(0) >>> c.CALL_FUNCTION_VAR_KW(0,1) >>> c.POP_TOP() >>> c.stack_size 0 >>> c = Code() >>> c.RAISE_VARARGS(0) >>> c.RAISE_VARARGS(1) Traceback (most recent call last): ... AssertionError: Stack underflow >>> c.LOAD_CONST(1) >>> c.RAISE_VARARGS(1) >>> dis(c.code()) 0 0 RAISE_VARARGS 0 3 LOAD_CONST 1 (1) 6 RAISE_VARARGS 1 Sequence building, unpacking, dup'ing:: >>> c = Code() >>> c.LOAD_CONST(1) >>> c.LOAD_CONST(2) >>> c.BUILD_TUPLE(3) Traceback (most recent call last): ... AssertionError: Stack underflow >>> c.BUILD_LIST(3) Traceback (most recent call last): ... AssertionError: Stack underflow >>> c.BUILD_TUPLE(2) >>> c.stack_size 1 >>> c.UNPACK_SEQUENCE(2) >>> c.stack_size 2 >>> c.DUP_TOPX(3) Traceback (most recent call last): ... AssertionError: Stack underflow >>> c.DUP_TOPX(2) >>> c.stack_size 4 >>> c.LOAD_CONST(3) >>> c.BUILD_LIST(5) >>> c.stack_size 1 >>> c.UNPACK_SEQUENCE(5) >>> c.BUILD_SLICE(3) >>> c.stack_size 3 >>> c.BUILD_SLICE(3) >>> c.stack_size 1 >>> c.BUILD_SLICE(2) Traceback (most recent call last): ... AssertionError: Stack underflow >>> dis(c.code()) 0 0 LOAD_CONST 1 (1) 3 LOAD_CONST 2 (2) 6 BUILD_TUPLE 2 9 UNPACK_SEQUENCE 2 12 DUP_TOPX 2 15 LOAD_CONST 3 (3) 18 BUILD_LIST 5 21 UNPACK_SEQUENCE 5 24 BUILD_SLICE 3 27 BUILD_SLICE 3 XXX Need tests for MAKE_CLOSURE/MAKE_FUNCTION Labels and backpatching forward references:: >>> c = Code() >>> ref = c.JUMP_ABSOLUTE() >>> c.LOAD_CONST(1) >>> ref() >>> c.RETURN_VALUE() >>> dis(c.code()) 0 0 JUMP_ABSOLUTE 6 3 LOAD_CONST 1 (1) >> 6 RETURN_VALUE >>> c = Code() >>> ref = c.JUMP_FORWARD() >>> c.LOAD_CONST(1) >>> ref() >>> c.RETURN_VALUE() >>> dis(c.code()) 0 0 JUMP_FORWARD 3 (to 6) 3 LOAD_CONST 1 (1) >> 6 RETURN_VALUE >>> c = Code() >>> lbl = c.label() >>> c.LOAD_CONST(1) >>> c.JUMP_IF_TRUE(lbl) Traceback (most recent call last): ... AssertionError: Relative jumps can't go backwards >>> c = Code() >>> lbl = c.label() >>> c.LOAD_CONST(1) >>> ref = c.JUMP_ABSOLUTE(lbl) >>> dis(c.code()) 0 >> 0 LOAD_CONST 1 (1) 3 JUMP_ABSOLUTE 0 "Call" combinations:: >>> c = Code() >>> c.set_lineno(1) >>> c(Call(Global('foo'), [Local('q')], ... [('x',Const(1))], Local('starargs')) ... ) >>> c.RETURN_VALUE() >>> dis(c.code()) 1 0 LOAD_GLOBAL 0 (foo) 3 LOAD_FAST 0 (q) 6 LOAD_CONST 1 ('x') 9 LOAD_CONST 2 (1) 12 LOAD_FAST 1 (starargs) 15 CALL_FUNCTION_VAR 257 18 RETURN_VALUE >>> c = Code() >>> c.set_lineno(1) >>> c(Call(Global('foo'), [Local('q')], [('x',Const(1))], ... None, Local('kwargs')) ... ) >>> c.RETURN_VALUE() >>> dis(c.code()) 1 0 LOAD_GLOBAL 0 (foo) 3 LOAD_FAST 0 (q) 6 LOAD_CONST 1 ('x') 9 LOAD_CONST 2 (1) 12 LOAD_FAST 1 (kwargs) 15 CALL_FUNCTION_KW 257 18 RETURN_VALUE Cloning:: >>> c = Code.from_function(lambda (x,y):1, True) >>> dis(c.code()) 1 0 LOAD_FAST 0 (.0) 3 UNPACK_SEQUENCE 2 6 STORE_FAST 1 (x) 9 STORE_FAST 2 (y) >>> c = Code.from_function(lambda x,(y,(z,a,b)):1, True) >>> dis(c.code()) 1 0 LOAD_FAST 1 (.1) 3 UNPACK_SEQUENCE 2 6 STORE_FAST 2 (y) 9 UNPACK_SEQUENCE 3 12 STORE_FAST 3 (z) 15 STORE_FAST 4 (a) 18 STORE_FAST 5 (b) TODO ==== * Test free/cell ops (LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF) * Test MAKE_FUNCTION/MAKE_CLOSURE * Test code flags generation/cloning