[Subversion] / BytecodeAssembler / peak / util / assembler.txt  

View of /BytecodeAssembler/peak/util/assembler.txt

Parent Directory | Revision Log
Revision: 2188 - (download)
Thu Jun 15 06:19:17 2006 UTC (17 years, 9 months ago) by pje
File size: 23122 byte(s)
Mark tests that fail under Python 2.3, and explain why.
=======================================================
Generating Python Bytecode with ``peak.util.assembler``
=======================================================

``peak.util.assembler`` is a simple bytecode assembler module that handles most
low-level bytecode generation details like jump offsets, stack size tracking,
line number table generation, constant and variable name index tracking, etc.
That way, you can focus your attention on the desired semantics of your
bytecode instead of on these mechanical issues.

In addition to a low-level opcode-oriented API for directly generating specific
bytecodes, the module also offers an extensible mini-AST framework for
generating code from high-level specifications.  This framework does most of
the work needed to transform tree-like structures into linear bytecode
instructions, and includes the ability to do compile-time constant folding.


.. contents:: Table of Contents


--------------
Programmer API
--------------


Code Objects
============

To generate bytecode, you create a ``Code`` instance and perform operations
on it.  For example, here we create a ``Code`` object representing lines
15 and 16 of some input source::

    >>> from peak.util.assembler import Code
    >>> c = Code()
    >>> c.set_lineno(15)   # set the current line number (optional)
    >>> c.LOAD_CONST(42)

    >>> c.set_lineno(16)   # set it as many times as you like
    >>> c.RETURN_VALUE()

You'll notice that most ``Code`` methods are named for a CPython bytecode
operation, but there also some other methods like ``.set_lineno()`` to let you
set the current line number.  There's also a ``.code()`` method that returns
a Python code object, representing the current state of the ``Code`` you've
generated::

    >>> from dis import dis
    >>> dis(c.code())
      15          0 LOAD_CONST               1 (42)
      16          3 RETURN_VALUE

As you can see, ``Code`` instances automatically generate a line number table
that maps each ``set_lineno()`` to the corresponding position in the bytecode.

And of course, the resulting code objects can be run with ``eval()`` or
``exec``, or used with ``new.function`` to create a function::

    >>> eval(c.code())
    42

    >>> exec c.code()   # exec discards the return value, so no output here

    >>> import new
    >>> f = new.function(c.code(), globals())
    >>> f()
    42


Opcodes, Jumps, and Labels
==========================

``Code`` objects have methods for all of CPython's symbolic opcodes.  Generally
speaking, each method accepts either zero or one argument, depending on whether
the opcode accepts an argument.

But while Python bytecode always encodes arguments as 16 or 32-bit integers,
you will generally pass actual names or values to ``Code`` methods, and the
``Code`` object will take care of maintaining the necessary lookup tables and
translation to integer bytecode arguments.



Labels and backpatching forward references::

    >>> c = Code()
    >>> ref = c.JUMP_ABSOLUTE()     # jump w/unspecified target
    >>> c.LOAD_CONST(1)
    >>> ref()                       # resolve the forward reference
    >>> c.RETURN_VALUE()
    >>> dis(c.code())
      0           0 JUMP_ABSOLUTE            6
                  3 LOAD_CONST               1 (1)
            >>    6 RETURN_VALUE


    >>> c = Code()
    >>> lbl = c.label()     # create a label at this point in the code
    >>> c.LOAD_CONST(1)
    >>> ref = c.JUMP_ABSOLUTE(lbl)  # and jump to it
    >>> dis(c.code())
      0     >>    0 LOAD_CONST               1 (1)
                  3 JUMP_ABSOLUTE            0


Code Generation API
===================

Code generation from tuples, lists, dicts, and local variable names::

    >>> from peak.util.assembler import Const, Call, Global, Local

    >>> c = Code()
    >>> c( [Local('x'), (Local('y'),Local('z'))] )  # push a value on the stack
    >>> dis(c.code())
      0           0 LOAD_FAST                0 (x)
                  3 LOAD_FAST                1 (y)
                  6 LOAD_FAST                2 (z)
                  9 BUILD_TUPLE              2
                 12 BUILD_LIST               2

And with constants, dictionaries, globals, and calls::


    >>> c = Code()
    >>> c.return_( [Global('type'), Const(27)] )     # push and RETURN_VALUE
    >>> dis(c.code())
      0           0 LOAD_GLOBAL              0 (type)
                  3 LOAD_CONST               1 (27)
                  6 BUILD_LIST               2
                  9 RETURN_VALUE

    >>> c = Code()
    >>> c( {Const('x'): Const(123)} )
    >>> dis(c.code())
      0           0 BUILD_MAP                0
                  3 DUP_TOP
                  4 LOAD_CONST               1 ('x')
                  7 LOAD_CONST               2 (123)
                 10 ROT_THREE
                 11 STORE_SUBSCR

    >>> c = Code()
    >>> c(Call(Global('type'), (Const(1),)))
    >>> dis(c.code())
      0           0 LOAD_GLOBAL              0 (type)
                  3 LOAD_CONST               1 (1)
                  6 CALL_FUNCTION            1

    >>> c = Code()
    >>> c(Call(Global('getattr'), (Const(1), Const('__class__'))))
    >>> dis(c.code())
      0           0 LOAD_GLOBAL              0 (getattr)
                  3 LOAD_CONST               1 (1)
                  6 LOAD_CONST               2 ('__class__')
                  9 CALL_FUNCTION            2

``Call`` objects take 1-4 arguments: the expression to be called, a sequence
of positional arguments, a sequence of keyword/value pairs for explicit keyword
arguments, an "*" argument, and a "**" argument.  To omit any of the optional
arguments, just pass in an empty sequence in its place::

    >>> c = Code()
    >>> c.return_(
    ...     Call(Global('foo'), [Local('q')], [('x',Const(1))],
    ...          Local('starargs'), Local('kwargs'))
    ... )
    >>> dis(c.code())
      0           0 LOAD_GLOBAL              0 (foo)
                  3 LOAD_FAST                0 (q)
                  6 LOAD_CONST               1 ('x')
                  9 LOAD_CONST               2 (1)
                 12 LOAD_FAST                1 (starargs)
                 15 LOAD_FAST                2 (kwargs)
                 18 CALL_FUNCTION_VAR_KW   257
                 21 RETURN_VALUE

Code generation is extensible: any 1-argument callables passed in will
be passed the code object during generation.  (The return value, if any, is
ignored.)  You can even use ``Code`` methods, if they don't have any required
arguments::

    >>> c = Code()
    >>> c.LOAD_GLOBAL('foo')
    >>> c(Call(Code.DUP_TOP, ()))
    >>> dis(c.code())
      0           0 LOAD_GLOBAL              0 (foo)
                  3 DUP_TOP
                  4 CALL_FUNCTION            0

This basically means you can create a simple AST of callable objects to drive
code generation, with a lot of the grunt work automatically handled for you.


Setting the Code's Calling Signature
====================================

The simplest way to set up the calling signature for a ``Code`` instance is
to clone an existing function or code object's signature, using the
``Code.from_function()`` or ``Code.from_code()`` classmethods.  These methods
create a new code object whose calling signature (number and names of
arguments) matches that of the original function or code objects::

    >>> def f1(a,b,*c,**d):
    ...     pass

    >>> c1 = Code.from_function(f1)
    >>> c1.co_argcount
    2
    >>> c1.co_varnames
    ['a', 'b', 'c', 'd']

    >>> import inspect
    >>> inspect.getargspec(f1)
    (['a', 'b'], 'c', 'd', None)

    >>> f2 = new.function(c1.code(), globals())
    >>> inspect.getargspec(f2)
    (['a', 'b'], 'c', 'd', None)

Note that these constructors do not copy any actual *code* from the code
or function objects.  They simply copy the signature, and, if you set the
``copy_lineno`` keyword argument to a true value, they will also set the
created code object's ``co_firstlineno`` to match that of the original code or
function object::

    >>> c1 = Code.from_function(f1, copy_lineno=True)
    >>> c1.co_firstlineno
    1

If you create a ``Code`` instance from a function that has nested positional
arguments, the returned code object will include a prologue to unpack the
arguments properly::

    >>> def f3(a, (b,c), (d,(e,f))):
    ...     pass

    >>> f4 = new.function(Code.from_function(f3).code(), globals())
    >>> dis(f4)
      0           0 LOAD_FAST                1 (.1)
                  3 UNPACK_SEQUENCE          2
                  6 STORE_FAST               3 (b)
                  9 STORE_FAST               4 (c)
                 12 LOAD_FAST                2 (.2)
                 15 UNPACK_SEQUENCE          2
                 18 STORE_FAST               5 (d)
                 21 UNPACK_SEQUENCE          2
                 24 STORE_FAST               6 (e)
                 27 STORE_FAST               7 (f)

This is roughly the same code that Python would generate to do the same
unpacking process, and is designed so that the ``inspect`` module will
recognize it as an argument unpacking prologue::

    >>> inspect.getargspec(f3)
    (['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)

    >>> inspect.getargspec(f4)
    (['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)


Code Attributes
===============

``Code`` instances have a variety of attributes corresponding to either the
attributes of the Python code objects they generate, or to the current state
of code generation.

For example, the ``co_argcount`` and ``co_varnames`` attributes
correspond to those used in creating the code for a Python function.  If you
want your code to be a function, you can set them as follows::

    >>> c = Code()
    >>> c.co_argcount = 3
    >>> c.co_varnames = ['a','b','c']

    >>> c.LOAD_CONST(42)
    >>> c.RETURN_VALUE()

    >>> f = new.function(c.code(), globals())
    >>> f(1,2,3)
    42

    >>> import inspect
    >>> inspect.getargspec(f)
    (['a', 'b', 'c'], None, None, None)

Although Python code objects want ``co_varnames`` to be a tuple, ``Code``
instances use a list, so that names can be added during code generation.  The
``.code()`` method automatically creates tuples where necessary.

Here are all of the ``Code`` attributes you may want to read or write:

co_filename
    A string representing the source filename for this code.  If it's an actual
    filename, then tracebacks that pass through the generated code will display
    lines from the file.  The default value is ``'<generated code>'``.

co_name
    The name of the function, class, or other block that this code represents.
    The default value is ``'<lambda>'``.

co_argcount
    Number of positional arguments a function accepts; defaults to 0

co_varnames
    A list of strings naming the code's local variables, beginning with its
    positional argument names, followed by its ``*`` and ``**`` argument names,
    if applicable, followed by any other local variable names.  These names
    are used by the ``LOAD_FAST`` and ``STORE_FAST`` opcodes, and invoking
    the ``.LOAD_FAST(name)`` and ``.STORE_FAST(name)`` methods of a code object
    will automatically add the given name to this list, if it's not already
    present.

co_flags
    The flags for the Python code object.  This defaults to
    ``CO_OPTIMIZED | CO_NEWLOCALS``, which is the correct value for a function
    using "fast" locals.  This value is automatically or-ed with ``CO_NOFREE``
    when generating a code object, if the ``co_cellvars`` and ``co_freevars``
    attributes are empty.  And if you use the ``LOAD_NAME()``,
    ``STORE_NAME()``, or ``DELETE_NAME()`` methods, the ``CO_OPTIMIZED`` bit
    is automatically reset, since these opcodes can only be used when the
    code is running with a real (i.e. not virtualized) ``locals()`` dictionary.

    If you need to change any other flag bits besides the above, you'll need to
    set or clear them manually.  For your convenience, the
    ``peak.util.assembler`` module exports all the ``CO_`` constants used by
    Python.  For example, you can use ``CO_VARARGS`` and ``CO_VARKEYWORDS`` to
    indicate whether a function accepts ``*`` or ``**`` arguments, as long as
    you extend the ``co_varnames`` list accordingly.  (Assuming you don't have
    an existing function or code object with the desired signature, in which
    case you could just use the ``from_function()`` or ``from_code()``
    classmethods instead of messing with these low-level attributes and flags.)

stack_size
    The predicted height of the runtime value stack, as of the current opcode.
    Its value is automatically updated by most opcodes, but you may want to
    save and restore it for things like try/finally blocks.

co_freevars
    A tuple of strings naming a function's "cell" variables.  Defaults to an
    empty tuple.  A function's free variables are the variables it "inherits"
    from its surrounding scope.  If you're going to use this, you should set
    it only once, before generating any code that references any free *or* cell
    variables.

co_cellvars
    A tuple of strings naming a function's "cell" variables.  Defaults to an
    empty tuple.  A function's cell variables are the variables that are
    "inherited" by one or more of its nested functions.  If you're going to use
    this, you should set it only once, before generating any code that
    references any free *or* cell variables.

These other attributes are automatically generated and maintained, so you'll
probably never have a reason to change them:

co_consts
    A list of constants used by the code; the first (zeroth?) constant is
    always ``None``.  Normally, this is automatically maintained; the
    ``.LOAD_CONST(value)`` method checks to see if the constant is already
    present in this list, and adds it if it is not there.

co_names
    A list of non-optimized or global variable names.  It's automatically
    updated whenever you invoke a method to generate an opcode that uses
    such names.

co_code
    A byte array containing the generated code.  Don't mess with this.

co_firstlineno
    The first line number of the generated code.  It automatically gets set
    if you call ``.set_lineno()`` before generating any code; otherwise it
    defaults to zero.

co_lnotab
    A byte array containing a generated line number table.  It's automatically
    generated, so don't mess with it.

co_stacksize
    The maximum amount of stack space the code will require to run.  This
    value is usually updated automatically as you generate code.



----------------------
Internals and Doctests
----------------------

Line number tracking::

    >>> def simple_code(flno, slno, consts=1, ):
    ...     c = Code()
    ...     c.set_lineno(flno)
    ...     for i in range(consts): c.LOAD_CONST(None)
    ...     c.set_lineno(slno)
    ...     c.RETURN_VALUE()
    ...     return c.code()

    >>> dis(simple_code(1,1))
      1           0 LOAD_CONST               0 (None)
                  3 RETURN_VALUE

    >>> simple_code(1,1).co_stacksize
    1

    >>> dis(simple_code(13,414))    # FAILURE EXPECTED IN PYTHON 2.3
     13           0 LOAD_CONST               0 (None)
    414           3 RETURN_VALUE

    >>> dis(simple_code(13,14,100))
     13           0 LOAD_CONST               0 (None)
                  3 LOAD_CONST               0 (None)
    ...
     14         300 RETURN_VALUE

    >>> simple_code(13,14,100).co_stacksize
    100

    >>> dis(simple_code(13,572,120))    # FAILURE EXPECTED IN Python 2.3
     13           0 LOAD_CONST               0 (None)
                  3 LOAD_CONST               0 (None)
    ...
    572         360 RETURN_VALUE


Stack size tracking::

    >>> c = Code()
    >>> c.LOAD_CONST(1)
    >>> c.POP_TOP()
    >>> c.LOAD_CONST(2)
    >>> c.LOAD_CONST(3)
    >>> c.co_stacksize
    2
    >>> c.BINARY_ADD()
    >>> c.LOAD_CONST(4)
    >>> c.co_stacksize
    2
    >>> c.LOAD_CONST(5)
    >>> c.LOAD_CONST(6)
    >>> c.co_stacksize
    4
    >>> c.POP_TOP()
    >>> c.stack_size
    3

Stack underflow detection/recovery, and global/local variable names::

    >>> c = Code()
    >>> c.LOAD_GLOBAL('foo')
    >>> c.stack_size
    1
    >>> c.STORE_ATTR('bar')     # drops stack by 2
    Traceback (most recent call last):
      ...
    AssertionError: Stack underflow

    >>> c.co_names  # 'bar' isn't added unless success
    ['foo']

    >>> c.LOAD_ATTR('bar')
    >>> c.co_names
    ['foo', 'bar']

    >>> c.DELETE_FAST('baz')
    >>> c.co_varnames
    ['baz']

    >>> dis(c.code())
      0           0 LOAD_GLOBAL              0 (foo)
                  3 LOAD_ATTR                1 (bar)
                  6 DELETE_FAST              0 (baz)

Sequence operators and stack tracking:



Function calls and raise::

    >>> c = Code()
    >>> c.LOAD_GLOBAL('locals')
    >>> c.CALL_FUNCTION()   # argc/kwargc default to 0
    >>> c.POP_TOP()
    >>> c.LOAD_GLOBAL('foo')
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST('x')
    >>> c.LOAD_CONST(2)
    >>> c.CALL_FUNCTION(1,1)    # argc, kwargc
    >>> c.POP_TOP()

    >>> dis(c.code())
      0           0 LOAD_GLOBAL              0 (locals)
                  3 CALL_FUNCTION            0
                  6 POP_TOP
                  7 LOAD_GLOBAL              1 (foo)
                 10 LOAD_CONST               1 (1)
                 13 LOAD_CONST               2 ('x')
                 16 LOAD_CONST               3 (2)
                 19 CALL_FUNCTION          257
                 22 POP_TOP

    >>> c = Code()
    >>> c.LOAD_GLOBAL('foo')
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST('x')
    >>> c.LOAD_CONST(2)
    >>> c.BUILD_MAP(0)
    >>> c.stack_size
    5
    >>> c.CALL_FUNCTION_KW(1,1)
    >>> c.POP_TOP()
    >>> c.stack_size
    0

    >>> c = Code()
    >>> c.LOAD_GLOBAL('foo')
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST('x')
    >>> c.LOAD_CONST(1)
    >>> c.BUILD_TUPLE(1)
    >>> c.CALL_FUNCTION_VAR(0,1)
    >>> c.POP_TOP()
    >>> c.stack_size
    0

    >>> c = Code()
    >>> c.LOAD_GLOBAL('foo')
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST('x')
    >>> c.LOAD_CONST(1)
    >>> c.BUILD_TUPLE(1)
    >>> c.BUILD_MAP(0)
    >>> c.CALL_FUNCTION_VAR_KW(0,1)
    >>> c.POP_TOP()
    >>> c.stack_size
    0

    >>> c = Code()
    >>> c.RAISE_VARARGS(0)
    >>> c.RAISE_VARARGS(1)
    Traceback (most recent call last):
      ...
    AssertionError: Stack underflow
    >>> c.LOAD_CONST(1)
    >>> c.RAISE_VARARGS(1)

    >>> dis(c.code())
      0           0 RAISE_VARARGS            0
                  3 LOAD_CONST               1 (1)
                  6 RAISE_VARARGS            1

Sequence building, unpacking, dup'ing::

    >>> c = Code()
    >>> c.LOAD_CONST(1)
    >>> c.LOAD_CONST(2)
    >>> c.BUILD_TUPLE(3)
    Traceback (most recent call last):
      ...
    AssertionError: Stack underflow

    >>> c.BUILD_LIST(3)
    Traceback (most recent call last):
      ...
    AssertionError: Stack underflow

    >>> c.BUILD_TUPLE(2)
    >>> c.stack_size
    1

    >>> c.UNPACK_SEQUENCE(2)
    >>> c.stack_size
    2
    >>> c.DUP_TOPX(3)
    Traceback (most recent call last):
      ...
    AssertionError: Stack underflow

    >>> c.DUP_TOPX(2)
    >>> c.stack_size
    4
    >>> c.LOAD_CONST(3)
    >>> c.BUILD_LIST(5)
    >>> c.stack_size
    1
    >>> c.UNPACK_SEQUENCE(5)
    >>> c.BUILD_SLICE(3)
    >>> c.stack_size
    3
    >>> c.BUILD_SLICE(3)
    >>> c.stack_size
    1
    >>> c.BUILD_SLICE(2)
    Traceback (most recent call last):
      ...
    AssertionError: Stack underflow

    >>> dis(c.code())
      0           0 LOAD_CONST               1 (1)
                  3 LOAD_CONST               2 (2)
                  6 BUILD_TUPLE              2
                  9 UNPACK_SEQUENCE          2
                 12 DUP_TOPX                 2
                 15 LOAD_CONST               3 (3)
                 18 BUILD_LIST               5
                 21 UNPACK_SEQUENCE          5
                 24 BUILD_SLICE              3
                 27 BUILD_SLICE              3

    XXX Need tests for MAKE_CLOSURE/MAKE_FUNCTION


Labels and backpatching forward references::

    >>> c = Code()
    >>> ref = c.JUMP_ABSOLUTE()
    >>> c.LOAD_CONST(1)
    >>> ref()
    >>> c.RETURN_VALUE()
    >>> dis(c.code())
      0           0 JUMP_ABSOLUTE            6
                  3 LOAD_CONST               1 (1)
            >>    6 RETURN_VALUE

    >>> c = Code()
    >>> ref = c.JUMP_FORWARD()
    >>> c.LOAD_CONST(1)
    >>> ref()
    >>> c.RETURN_VALUE()
    >>> dis(c.code())
      0           0 JUMP_FORWARD             3 (to 6)
                  3 LOAD_CONST               1 (1)
            >>    6 RETURN_VALUE

    >>> c = Code()
    >>> lbl = c.label()
    >>> c.LOAD_CONST(1)
    >>> c.JUMP_IF_TRUE(lbl)
    Traceback (most recent call last):
      ...
    AssertionError: Relative jumps can't go backwards

    >>> c = Code()
    >>> lbl = c.label()
    >>> c.LOAD_CONST(1)
    >>> ref = c.JUMP_ABSOLUTE(lbl)
    >>> dis(c.code())
      0     >>    0 LOAD_CONST               1 (1)
                  3 JUMP_ABSOLUTE            0


"Call" combinations::


    >>> c = Code()
    >>> c.set_lineno(1)
    >>> c(Call(Global('foo'), [Local('q')],
    ...        [('x',Const(1))], Local('starargs'))
    ... )
    >>> c.RETURN_VALUE()
    >>> dis(c.code())
      1           0 LOAD_GLOBAL              0 (foo)
                  3 LOAD_FAST                0 (q)
                  6 LOAD_CONST               1 ('x')
                  9 LOAD_CONST               2 (1)
                 12 LOAD_FAST                1 (starargs)
                 15 CALL_FUNCTION_VAR      257
                 18 RETURN_VALUE


    >>> c = Code()
    >>> c.set_lineno(1)
    >>> c(Call(Global('foo'), [Local('q')], [('x',Const(1))],
    ...        None, Local('kwargs'))
    ... )
    >>> c.RETURN_VALUE()
    >>> dis(c.code())
      1           0 LOAD_GLOBAL              0 (foo)
                  3 LOAD_FAST                0 (q)
                  6 LOAD_CONST               1 ('x')
                  9 LOAD_CONST               2 (1)
                 12 LOAD_FAST                1 (kwargs)
                 15 CALL_FUNCTION_KW       257
                 18 RETURN_VALUE


Cloning::

    >>> c = Code.from_function(lambda (x,y):1, True)
    >>> dis(c.code())
      1           0 LOAD_FAST                0 (.0)
                  3 UNPACK_SEQUENCE          2
                  6 STORE_FAST               1 (x)
                  9 STORE_FAST               2 (y)

    >>> c = Code.from_function(lambda x,(y,(z,a,b)):1, True)
    >>> dis(c.code())
      1           0 LOAD_FAST                1 (.1)
                  3 UNPACK_SEQUENCE          2
                  6 STORE_FAST               2 (y)
                  9 UNPACK_SEQUENCE          3
                 12 STORE_FAST               3 (z)
                 15 STORE_FAST               4 (a)
                 18 STORE_FAST               5 (b)

TODO
====

* Test free/cell ops (LOAD_CLOSURE, LOAD_DEREF, STORE_DEREF)
* Test MAKE_FUNCTION/MAKE_CLOSURE
* Test code flags generation/cloning




cvs-admin@eby-sarna.com

Powered by ViewCVS 1.0-dev

ViewCVS and CVS Help