View of /BytecodeAssembler/peak/util/assembler.txtView of /BytecodeAssembler/peak/util/assembler.txt
Parent Directory
| 
Revision Log
Major doc upgrade and new features: Return() nodes, code() accepts multiple args, and forward references can be used as code generation targets. Booleans are now treated as constants. Local() vars can now be references to cell/free variables, and free/cell index lookup is fixed. ``curry()`` is now ``ast_curry()``, and handles ``None`` arguments by replacing them with special ``nil`` objects, correcting the problem of naively using ``None`` for AST child nodes. And the documentation now covers virtually all of the package's current functionality.
=======================================================
Generating Python Bytecode with ``peak.util.assembler``
=======================================================
``peak.util.assembler`` is a simple bytecode assembler module that handles most
low-level bytecode generation details like jump offsets, stack size tracking,
line number table generation, constant and variable name index tracking, etc.
That way, you can focus your attention on the desired semantics of your
bytecode instead of on these mechanical issues.
In addition to a low-level opcode-oriented API for directly generating specific
bytecodes, the module also offers an extensible mini-AST framework for
generating code from high-level specifications. This framework does most of
the work needed to transform tree-like structures into linear bytecode
instructions, and includes the ability to do compile-time constant folding.
.. contents:: Table of Contents
--------------
Programmer API
--------------
Code Objects
============
To generate bytecode, you create a ``Code`` instance and perform operations
on it. For example, here we create a ``Code`` object representing lines
15 and 16 of some input source::
>>> from peak.util.assembler import Code
>>> c = Code()
>>> c.set_lineno(15) # set the current line number (optional)
>>> c.LOAD_CONST(42)
>>> c.set_lineno(16) # set it as many times as you like
>>> c.RETURN_VALUE()
You'll notice that most ``Code`` methods are named for a CPython bytecode
operation, but there also some other methods like ``.set_lineno()`` to let you
set the current line number. There's also a ``.code()`` method that returns
a Python code object, representing the current state of the ``Code`` you've
generated::
>>> from dis import dis
>>> dis(c.code())
15 0 LOAD_CONST 1 (42)
16 3 RETURN_VALUE
As you can see, ``Code`` instances automatically generate a line number table
that maps each ``set_lineno()`` to the corresponding position in the bytecode.
And of course, the resulting code objects can be run with ``eval()`` or
``exec``, or used with ``new.function`` to create a function::
>>> eval(c.code())
42
>>> exec c.code() # exec discards the return value, so no output here
>>> import new
>>> f = new.function(c.code(), globals())
>>> f()
42
Opcodes and Arguments
=====================
``Code`` objects have methods for all of CPython's symbolic opcodes. Generally
speaking, each method accepts either zero or one argument, depending on whether
the opcode accepts an argument.
Python bytecode always encodes opcode arguments as 16 or 32-bit integers, but
sometimes these numbers are actually offsets into a sequence of names or
constants. ``Code`` objects take care of maintaining these sequences for you,
allowing you to just pass in a name or value directly, instead of needing to
keep track of what numbers map to what names or values.
The name or value you pass in to such methods will be looked up in the
appropriate table (see `Code Attributes`_ below for a list), and if not found,
it will be added::
>>> c = Code()
>>> c.co_consts, c.co_varnames, c.co_names
([None], [], [])
>>> c.LOAD_CONST(42)
>>> c.LOAD_FAST('x')
>>> c.LOAD_GLOBAL('y')
>>> c.LOAD_NAME('z')
>>> c.co_consts, c.co_varnames, c.co_names
([None, 42], ['x'], ['y', 'z'])
The one exception to this automatic addition feature is that opcodes referring
to "free" or "cell" variables will not automatically add new names, because the
names need to be defined first::
>>> c.LOAD_DEREF('q')
Traceback (most recent call last):
...
NameError: ('Undefined free or cell var', 'q')
In general, opcode methods take the same arguments as their Python bytecode
equivalent. But there are a few special cases.
Call Arguments
--------------
First, the ``CALL_FUNCTION()``, ``CALL_FUNCTION_VAR()``, ``CALL_FUNCTION_KW()``,
and ``CALL_FUNCTION_VAR_KW()`` methods all take *two* arguments, both of which
are optional. (The ``_VAR`` and ``_KW`` suffixes in the method names indicate
whether or not a ``*args`` or ``**kwargs`` or both are also present on the
stack, in addition to the explicit positional and keyword arguments.)
The first argument of each of these methods, is the number of positional
arguments on the stack, and the second is the number of keyword/value pairs on
the stack (to be used as keyword arguments). Both default to zero if not
supplied::
>>> c = Code()
>>> c.LOAD_CONST(type)
>>> c.LOAD_CONST(27)
>>> c.CALL_FUNCTION(1) # 1 positional, no keywords
>>> c.RETURN_VALUE()
>>> eval(c.code()) # computes type(27)
<type 'int'>
>>> c = Code()
>>> c.LOAD_CONST(dict)
>>> c.LOAD_CONST('x')
>>> c.LOAD_CONST(42)
>>> c.CALL_FUNCTION(0,1) # no positional, 1 keyword
>>> c.RETURN_VALUE()
>>> eval(c.code()) # computes dict(x=42)
{'x': 42}
Jumps, Labels, and Blocks
-------------------------
Opcodes that perform jumps or refer to addresses (which includes block-setup
opcodes like ``SETUP_LOOP`` and ``SETUP_FINALLY``) can be invoked in one of
two ways. First, if you are jumping backwards (with ``JUMP_ABSOLUTE`` or
``CONTINUE_LOOP``), you can obtain the target bytecode offset using the
``.label()`` method, and then later pass that label into the appropriate
method::
>>> c = Code()
>>> my_label = c.label() # create a label at the start of the code
>>> c.LOAD_CONST(42)
>>> c.JUMP_ABSOLUTE(my_label) # now jump back to it
<...>
>>> dis(c.code())
0 >> 0 LOAD_CONST 1 (42)
3 JUMP_ABSOLUTE 0
But if you are jumping *forward* (or setting up a loop or a try block), you
will need to call the jump or setup method without any arguments, and save the
return value. The return value of all jump and block-setup methods is a
"forward reference" object that can be called later to indicate that the
desired jump target has been reached::
>>> c = Code()
>>> forward = c.JUMP_ABSOLUTE() # create a jump and a forward reference
>>> c.LOAD_CONST(42) # this is what we want to skip over
>>> forward() # calling the reference changes the jump to point here
>>> c.LOAD_CONST(23)
>>> c.RETURN_VALUE()
>>> dis(c.code())
0 0 JUMP_ABSOLUTE 6
3 LOAD_CONST 1 (42)
>> 6 LOAD_CONST 2 (23)
9 RETURN_VALUE
>>> eval(c.code())
23
Note that ``Code`` objects do not currently implement any special handling
for "block" operations like ``SETUP_EXCEPT`` and ``END_FINALLY``. You will
probably need to manually adjust the predicted stack size when working with
these opcodes. See the section below on `Code Attributes`_ for information
on the ``stack_size`` and ``co_stacksize`` attributes of ``Code`` objects.
Other Special Opcodes
---------------------
The ``MAKE_CLOSURE`` method takes an argument for the number of default values
on the stack, just like the "real" Python opcode. However, it also has an
an additional required argument: the number of closure cells on the stack.
The Python interpreter normally gets this number from a code object that's on
the stack, but ``Code`` objects need this value in order to update the
current stack size, for purposes of computing the required total stack size::
>>> def x(a,b): # a simple closure example
... def y():
... return a+b
... return y
>>> c = Code()
>>> c.co_cellvars = ('a','b')
>>> c.LOAD_CLOSURE('a')
>>> c.LOAD_CLOSURE('b')
>>> c.LOAD_CONST(None) # in real code, this'd be a Python code constant
>>> c.MAKE_CLOSURE(0,2) # no defaults, 2 free vars in the new function
High-Level Code Generation
==========================
Typical real-life code generation use cases call for transforming tree-like
data structures into bytecode, rather than linearly outputting instructions.
``Code`` objects provide for this using a simple but high-level transformation
API.
``Code`` objects may be *called*, passing in one or more arguments. Each
argument will have bytecode generated for it, according to its type:
Simple Constants
----------------
If an argument is an integer, long, float, complex, string, unicode, boolean,
``None``, or Python code object, it is treated as though it was passed to
the ``LOAD_CONST`` method directly::
>>> c = Code()
>>> c(1, 2L, 3.0, 4j+5, "6", u"7", False, None, c.code())
>>> dis(c.code())
0 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (2L)
6 LOAD_CONST 3 (3.0)
9 LOAD_CONST 4 ((5+4j))
12 LOAD_CONST 5 ('6')
15 LOAD_CONST 6 (u'7')
18 LOAD_CONST 7 (False)
21 LOAD_CONST 0 (None)
24 LOAD_CONST 8 (<code object <lambda> at ...>)
Simple Containers
-----------------
If an argument is a tuple, list, or dictionary, code is generated to
reconstruct the given data, recursively::
>>> c = Code()
>>> c({1:(2,"3"), 4:[5,6]})
>>> dis(c.code())
0 0 BUILD_MAP 0
3 DUP_TOP
4 LOAD_CONST 1 (1)
7 LOAD_CONST 2 (2)
10 LOAD_CONST 3 ('3')
13 BUILD_TUPLE 2
16 ROT_THREE
17 STORE_SUBSCR
18 DUP_TOP
19 LOAD_CONST 4 (4)
22 LOAD_CONST 5 (5)
25 LOAD_CONST 6 (6)
28 BUILD_LIST 2
31 ROT_THREE
32 STORE_SUBSCR
Arbitrary Constants
-------------------
The ``Const`` wrapper allows you to treat any object as a literal constant,
regardless of its type::
>>> from peak.util.assembler import Const
>>> c = Code()
>>> c( Const( (1,2,3) ) )
>>> dis(c.code())
0 0 LOAD_CONST 1 ((1, 2, 3))
As you can see, the above creates code that references an actual tuple as
a constant, rather than generating code to recreate the tuple using a series of
``LOAD_CONST`` operations followed by a ``BUILD_TUPLE``.
Local and Global Names
----------------------
The ``Local`` and ``Global`` wrappers take a name, and load either a local or
global variable, respectively::
>>> from peak.util.assembler import Global, Local
>>> c( Local('x'), Global('y') )
>>> dis(c.code())
0 0 LOAD_CONST 1 ((1, 2, 3))
3 LOAD_FAST 0 (x)
6 LOAD_GLOBAL 0 (y)
As with simple constants and ``Const`` wrappers, these objects can be used to
construct more complex expressions, like ``{a:(b,c)}``::
>>> c = Code()
>>> c( {Local('a'): (Local('b'), Local('c'))} )
>>> dis(c.code())
0 0 BUILD_MAP 0
3 DUP_TOP
4 LOAD_FAST 0 (a)
7 LOAD_FAST 1 (b)
10 LOAD_FAST 2 (c)
13 BUILD_TUPLE 2
16 ROT_THREE
17 STORE_SUBSCR
If the code object is not using "fast locals" (i.e. ``CO_OPTIMIZED`` isn't
set), local variables will be dereferenced using ``LOAD_NAME`` instead of
``LOAD_FAST``, and if the referenced local name is a "cell" or "free"
variable, ``LOAD_DEREF`` is used instead::
>>> from peak.util.assembler import CO_OPTIMIZED
>>> c = Code()
>>> c.co_flags &= ~CO_OPTIMIZED
>>> c.co_cellvars = ('y',)
>>> c.co_freevars = ('z',)
>>> c( Local('x'), Local('y'), Local('z') )
>>> dis(c.code())
0 0 LOAD_NAME 0 (x)
3 LOAD_DEREF 0 (y)
6 LOAD_DEREF 1 (z)
Calling Functions and Methods
-----------------------------
>>> from peak.util.assembler import Call
The ``Call`` wrapper takes 1-4 arguments: the expression to be called, a
sequence of positional arguments, a sequence of keyword/value pairs for
explicit keyword arguments, an "*" argument, and a "**" argument. To omit any
of the optional arguments, just pass in an empty sequence in its place::
>>> c = Code()
>>> c( Call(Global('type'), [Const(27)]) )
>>> dis(c.code()) # type(27)
0 0 LOAD_GLOBAL 0 (type)
3 LOAD_CONST 1 (27)
6 CALL_FUNCTION 1
>>> c = Code()
>>> c(Call(Global('dict'), (), [('x', 42)]))
>>> dis(c.code()) # dict(x=42)
0 0 LOAD_GLOBAL 0 (dict)
3 LOAD_CONST 1 ('x')
6 LOAD_CONST 2 (42)
9 CALL_FUNCTION 256
>>> c = Code()
>>> c(Call(Global('foo'), (), (), Local('args'), Local('kw')))
>>> dis(c.code()) # foo(*args, **kw)
0 0 LOAD_GLOBAL 0 (foo)
3 LOAD_FAST 0 (args)
6 LOAD_FAST 1 (kw)
9 CALL_FUNCTION_VAR_KW 0
Returning Values
----------------
The ``Return(target)`` wrapper generates code for its target, followed by
a ``RETURN_VALUE`` opcode::
>>> from peak.util.assembler import Return
>>> c = Code()
>>> c( Return(1) )
>>> dis(c.code())
0 0 LOAD_CONST 1 (1)
3 RETURN_VALUE
``Code`` objects also have a ``return_()`` method that provides a more compact
spelling of the same thing::
>>> c = Code()
>>> c.return_((1,2))
>>> dis(c.code())
0 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (2)
6 BUILD_TUPLE 2
9 RETURN_VALUE
Both ``Return`` and ``return_()`` can be used with no argument, in which case
``None`` is returned::
>>> c = Code()
>>> c.return_()
>>> c( Return() )
>>> dis(c.code())
0 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
4 LOAD_CONST 0 (None)
7 RETURN_VALUE
Using Forward References As Targets
-----------------------------------
The forward reference callbacks returned by jump operations are also usable
as code generation targets, indicating that the jump should go to the
current location. For example::
>>> c = Code()
>>> forward = c.JUMP_FORWARD()
>>> c( 1, 2, forward, Return(3) )
>>> dis(c.code())
0 0 JUMP_FORWARD 6 (to 9)
3 LOAD_CONST 1 (1)
6 LOAD_CONST 2 (2)
>> 9 LOAD_CONST 3 (3)
12 RETURN_VALUE
Custom Code Generation
======================
Code generation is extensible: you can use any callable as a code-generation
target. It will be called with exactly one argument: the code object. It can
then perform whatever operations are desired.
In the most trivial case, you can use any unbound ``Code`` method as a code
generation target, e.g.::
>>> c = Code()
>>> c.LOAD_GLOBAL('foo')
>>> c(Call(Code.DUP_TOP, ()))
>>> dis(c.code())
0 0 LOAD_GLOBAL 0 (foo)
3 DUP_TOP
4 CALL_FUNCTION 0
As you can see, the ``Code.DUP_TOP()`` is called on the code instance, causing
a ``DUP_TOP`` opcode to be output. This is sometimes a handy trick for
accessing values that are already on the stack. More commonly, however, you'll
want to implement more sophisticated callables, perhaps something like::
>>> from peak.util.assembler import ast_curry
>>> def TryFinally(block1, block2, code=None):
... if code is None:
... return ast_curry(TryFinally, block1, block2)
... fwd = code.SETUP_FINALLY()
... code(block1, Code.POP_BLOCK, None, fwd, block2, Code.END_FINALLY)
>>> def Stmt(value, code=None):
... if code is None:
... return ast_curry(Stmt, value)
... code( value, Code.POP_TOP )
>>> c = Code()
>>> c( TryFinally(Stmt(1), Stmt(2)) )
>>> dis(c.code())
0 0 SETUP_FINALLY 8 (to 11)
3 LOAD_CONST 1 (1)
6 POP_TOP
7 POP_BLOCK
8 LOAD_CONST 0 (None)
>> 11 LOAD_CONST 2 (2)
14 POP_TOP
15 END_FINALLY
The ``ast_curry()`` utility function returns an ``instancemethod`` chain that
binds the given arguments to the given function, creating a hashable and
comparable data structure -- a trivial sort of "AST node". Just follow the
code pattern above, using a ``code=None`` final argument, and returning a
curried version of the function if ``code is None``. Otherwise, your function
should simply do whatever is needed to "generate" the arguments.
This is exactly the same way that ``Const``, ``Call``, ``Local``, etc. are
implemented within ``peak.util.assembler``.
The ``ast_curry()`` utility function isn't quite perfect; due to a quirk of the
``instancemethod`` type, it can't save arguments whose value is ``None``: if
you pass a ``None`` argument to ``ast_curry()``, it will be replaced with a
special ``nil`` object that tests as false, and generates a ``None`` constant
when code is generated for it. If your function accepts any arguments that
might have a value of ``None``, you must correctly handle the cases where you
receive a value of ``nil`` (found in ``peak.util.assembler``) instead of
``None``.
However, if you can use ``ast_curry()`` to generate your AST nodes, you will
have objects that are hashable and comparable by default, as long as none of
your child nodes are unhashable or incomparable. This can be useful for
algorithms that require comparing AST subtrees, such as common subexpression
elimination.
Setting the Code's Calling Signature
====================================
The simplest way to set up the calling signature for a ``Code`` instance is
to clone an existing function or code object's signature, using the
``Code.from_function()`` or ``Code.from_code()`` classmethods. These methods
create a new ``Code`` instance whose calling signature (number and names of
arguments) matches that of the original function or code objects::
>>> def f1(a,b,*c,**d):
... pass
>>> c = Code.from_function(f1)
>>> f2 = new.function(c.code(), globals())
>>> import inspect
>>> inspect.getargspec(f1)
(['a', 'b'], 'c', 'd', None)
>>> inspect.getargspec(f2)
(['a', 'b'], 'c', 'd', None)
Note that these constructors do not copy any actual *code* from the code
or function objects. They simply copy the signature, and, if you set the
``copy_lineno`` keyword argument to a true value, they will also set the
created code object's ``co_firstlineno`` to match that of the original code or
function object::
>>> c1 = Code.from_function(f1, copy_lineno=True)
>>> c1.co_firstlineno
1
If you create a ``Code`` instance from a function that has nested positional
arguments, the returned code object will include a prologue to unpack the
arguments properly::
>>> def f3(a, (b,c), (d,(e,f))):
... pass
>>> f4 = new.function(Code.from_function(f3).code(), globals())
>>> dis(f4)
0 0 LOAD_FAST 1 (.1)
3 UNPACK_SEQUENCE 2
6 STORE_FAST 3 (b)
9 STORE_FAST 4 (c)
12 LOAD_FAST 2 (.2)
15 UNPACK_SEQUENCE 2
18 STORE_FAST 5 (d)
21 UNPACK_SEQUENCE 2
24 STORE_FAST 6 (e)
27 STORE_FAST 7 (f)
This is roughly the same code that Python would generate to do the same
unpacking process, and is designed so that the ``inspect`` module will
recognize it as an argument unpacking prologue::
>>> inspect.getargspec(f3)
(['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)
>>> inspect.getargspec(f4)
(['a', ['b', 'c'], ['d', ['e', 'f']]], None, None, None)
Code Attributes
===============
``Code`` instances have a variety of attributes corresponding to either the
attributes of the Python code objects they generate, or to the current state
of code generation.
For example, the ``co_argcount`` and ``co_varnames`` attributes
correspond to those used in creating the code for a Python function. If you
want your code to be a function, you can set them as follows::
>>> c = Code()
>>> c.co_argcount = 3
>>> c.co_varnames = ['a','b','c']
>>> c.LOAD_CONST(42)
>>> c.RETURN_VALUE()
>>> f = new.function(c.code(), globals())
>>> f(1,2,3)
42
>>> import inspect
>>> inspect.getargspec(f)
(['a', 'b', 'c'], None, None, None)
Although Python code objects want ``co_varnames`` to be a tuple, ``Code``
instances use a list, so that names can be added during code generation. The
``.code()`` method automatically creates tuples where necessary.
Here are all of the ``Code`` attributes you may want to read or write:
co_filename
A string representing the source filename for this code. If it's an actual
filename, then tracebacks that pass through the generated code will display
lines from the file. The default value is ``'<generated code>'``.
co_name
The name of the function, class, or other block that this code represents.
The default value is ``'<lambda>'``.
co_argcount
Number of positional arguments a function accepts; defaults to 0
co_varnames
A list of strings naming the code's local variables, beginning with its
positional argument names, followed by its ``*`` and ``**`` argument names,
if applicable, followed by any other local variable names. These names
are used by the ``LOAD_FAST`` and ``STORE_FAST`` opcodes, and invoking
the ``.LOAD_FAST(name)`` and ``.STORE_FAST(name)`` methods of a code object
will automatically add the given name to this list, if it's not already
present.
co_flags
The flags for the Python code object. This defaults to
``CO_OPTIMIZED | CO_NEWLOCALS``, which is the correct value for a function
using "fast" locals. This value is automatically or-ed with ``CO_NOFREE``
when generating a code object, if the ``co_cellvars`` and ``co_freevars``
attributes are empty. And if you use the ``LOAD_NAME()``,
``STORE_NAME()``, or ``DELETE_NAME()`` methods, the ``CO_OPTIMIZED`` bit
is automatically reset, since these opcodes can only be used when the
code is running with a real (i.e. not virtualized) ``locals()`` dictionary.
If you need to change any other flag bits besides the above, you'll need to
set or clear them manually. For your convenience, the
``peak.util.assembler`` module exports all the ``CO_`` constants used by
Python. For example, you can use ``CO_VARARGS`` and ``CO_VARKEYWORDS`` to
indicate whether a function accepts ``*`` or ``**`` arguments, as long as
you extend the ``co_varnames`` list accordingly. (Assuming you don't have
an existing function or code object with the desired signature, in which
case you could just use the ``from_function()`` or ``from_code()``
classmethods instead of messing with these low-level attributes and flags.)
stack_size
The predicted height of the runtime value stack, as of the current opcode.
Its value is automatically updated by most opcodes, but you may want to
save and restore it for things like try/finally blocks. If you increase
the value of this attribute, you should also update the ``co_stacksize``
attribute if it is less than the new ``stack_size``.
co_freevars
A tuple of strings naming a function's "cell" variables. Defaults to an
empty tuple. A function's free variables are the variables it "inherits"
from its surrounding scope. If you're going to use this, you should set
it only once, before generating any code that references any free *or* cell
variables.
co_cellvars
A tuple of strings naming a function's "cell" variables. Defaults to an
empty tuple. A function's cell variables are the variables that are
"inherited" by one or more of its nested functions. If you're going to use
this, you should set it only once, before generating any code that
references any free *or* cell variables.
These other attributes are automatically generated and maintained, so you'll
probably never have a reason to change them:
co_consts
A list of constants used by the code; the first (zeroth?) constant is
always ``None``. Normally, this is automatically maintained; the
``.LOAD_CONST(value)`` method checks to see if the constant is already
present in this list, and adds it if it is not there.
co_names
A list of non-optimized or global variable names. It's automatically
updated whenever you invoke a method to generate an opcode that uses
such names.
co_code
A byte array containing the generated code. Don't mess with this.
co_firstlineno
The first line number of the generated code. It automatically gets set
if you call ``.set_lineno()`` before generating any code; otherwise it
defaults to zero.
co_lnotab
A byte array containing a generated line number table. It's automatically
generated, so don't mess with it.
co_stacksize
The maximum amount of stack space the code will require to run. This
value is usually updated automatically as you generate code. However, if
you manually set a new ``stack_size`` that is larger than the current
``co_stacksize``, you should increase the ``co_stacksize`` to match, so
that ``co_stacksize`` is always the largest stack size the code will
generate at runtime.
----------------------
Internals and Doctests
----------------------
Line number tracking::
>>> def simple_code(flno, slno, consts=1, ):
... c = Code()
... c.set_lineno(flno)
... for i in range(consts): c.LOAD_CONST(None)
... c.set_lineno(slno)
... c.RETURN_VALUE()
... return c.code()
>>> dis(simple_code(1,1))
1 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
>>> simple_code(1,1).co_stacksize
1
>>> dis(simple_code(13,414)) # FAILURE EXPECTED IN PYTHON 2.3
13 0 LOAD_CONST 0 (None)
414 3 RETURN_VALUE
>>> dis(simple_code(13,14,100))
13 0 LOAD_CONST 0 (None)
3 LOAD_CONST 0 (None)
...
14 300 RETURN_VALUE
>>> simple_code(13,14,100).co_stacksize
100
>>> dis(simple_code(13,572,120)) # FAILURE EXPECTED IN Python 2.3
13 0 LOAD_CONST 0 (None)
3 LOAD_CONST 0 (None)
...
572 360 RETURN_VALUE
Stack size tracking::
>>> c = Code()
>>> c.LOAD_CONST(1)
>>> c.POP_TOP()
>>> c.LOAD_CONST(2)
>>> c.LOAD_CONST(3)
>>> c.co_stacksize
2
>>> c.BINARY_ADD()
>>> c.LOAD_CONST(4)
>>> c.co_stacksize
2
>>> c.LOAD_CONST(5)
>>> c.LOAD_CONST(6)
>>> c.co_stacksize
4
>>> c.POP_TOP()
>>> c.stack_size
3
Stack underflow detection/recovery, and global/local variable names::
>>> c = Code()
>>> c.LOAD_GLOBAL('foo')
>>> c.stack_size
1
>>> c.STORE_ATTR('bar') # drops stack by 2
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> c.co_names # 'bar' isn't added unless success
['foo']
>>> c.LOAD_ATTR('bar')
>>> c.co_names
['foo', 'bar']
>>> c.DELETE_FAST('baz')
>>> c.co_varnames
['baz']
>>> dis(c.code())
0 0 LOAD_GLOBAL 0 (foo)
3 LOAD_ATTR 1 (bar)
6 DELETE_FAST 0 (baz)
Sequence operators and stack tracking:
Function calls and raise::
>>> c = Code()
>>> c.LOAD_GLOBAL('locals')
>>> c.CALL_FUNCTION() # argc/kwargc default to 0
>>> c.POP_TOP()
>>> c.LOAD_GLOBAL('foo')
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST('x')
>>> c.LOAD_CONST(2)
>>> c.CALL_FUNCTION(1,1) # argc, kwargc
>>> c.POP_TOP()
>>> dis(c.code())
0 0 LOAD_GLOBAL 0 (locals)
3 CALL_FUNCTION 0
6 POP_TOP
7 LOAD_GLOBAL 1 (foo)
10 LOAD_CONST 1 (1)
13 LOAD_CONST 2 ('x')
16 LOAD_CONST 3 (2)
19 CALL_FUNCTION 257
22 POP_TOP
>>> c = Code()
>>> c.LOAD_GLOBAL('foo')
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST('x')
>>> c.LOAD_CONST(2)
>>> c.BUILD_MAP(0)
>>> c.stack_size
5
>>> c.CALL_FUNCTION_KW(1,1)
>>> c.POP_TOP()
>>> c.stack_size
0
>>> c = Code()
>>> c.LOAD_GLOBAL('foo')
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST('x')
>>> c.LOAD_CONST(1)
>>> c.BUILD_TUPLE(1)
>>> c.CALL_FUNCTION_VAR(0,1)
>>> c.POP_TOP()
>>> c.stack_size
0
>>> c = Code()
>>> c.LOAD_GLOBAL('foo')
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST('x')
>>> c.LOAD_CONST(1)
>>> c.BUILD_TUPLE(1)
>>> c.BUILD_MAP(0)
>>> c.CALL_FUNCTION_VAR_KW(0,1)
>>> c.POP_TOP()
>>> c.stack_size
0
>>> c = Code()
>>> c.RAISE_VARARGS(0)
>>> c.RAISE_VARARGS(1)
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> c.LOAD_CONST(1)
>>> c.RAISE_VARARGS(1)
>>> dis(c.code())
0 0 RAISE_VARARGS 0
3 LOAD_CONST 1 (1)
6 RAISE_VARARGS 1
Sequence building, unpacking, dup'ing::
>>> c = Code()
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST(2)
>>> c.BUILD_TUPLE(3)
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> c.BUILD_LIST(3)
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> c.BUILD_TUPLE(2)
>>> c.stack_size
1
>>> c.UNPACK_SEQUENCE(2)
>>> c.stack_size
2
>>> c.DUP_TOPX(3)
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> c.DUP_TOPX(2)
>>> c.stack_size
4
>>> c.LOAD_CONST(3)
>>> c.BUILD_LIST(5)
>>> c.stack_size
1
>>> c.UNPACK_SEQUENCE(5)
>>> c.BUILD_SLICE(3)
>>> c.stack_size
3
>>> c.BUILD_SLICE(3)
>>> c.stack_size
1
>>> c.BUILD_SLICE(2)
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> dis(c.code())
0 0 LOAD_CONST 1 (1)
3 LOAD_CONST 2 (2)
6 BUILD_TUPLE 2
9 UNPACK_SEQUENCE 2
12 DUP_TOPX 2
15 LOAD_CONST 3 (3)
18 BUILD_LIST 5
21 UNPACK_SEQUENCE 5
24 BUILD_SLICE 3
27 BUILD_SLICE 3
Stack levels for MAKE_FUNCTION/MAKE_CLOSURE::
>>> c = Code()
>>> c.MAKE_FUNCTION(0)
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST(2) # simulate being a function
>>> c.MAKE_FUNCTION(1)
>>> c.stack_size
1
>>> c = Code()
>>> c.MAKE_CLOSURE(0, 0)
Traceback (most recent call last):
...
AssertionError: Stack underflow
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST(2) # simulate being a function
>>> c.MAKE_CLOSURE(1, 0)
>>> c.stack_size
1
>>> c = Code()
>>> c.LOAD_CONST(1)
>>> c.LOAD_CONST(2)
>>> c.LOAD_CONST(3) # simulate being a function
>>> c.MAKE_CLOSURE(1, 1)
>>> c.stack_size
1
Labels and backpatching forward references::
>>> c = Code()
>>> lbl = c.label()
>>> c.LOAD_CONST(1)
>>> c.JUMP_IF_TRUE(lbl)
Traceback (most recent call last):
...
AssertionError: Relative jumps can't go backwards
"Call" combinations::
>>> c = Code()
>>> c.set_lineno(1)
>>> c(Call(Global('foo'), [Local('q')],
... [('x',Const(1))], Local('starargs'))
... )
>>> c.RETURN_VALUE()
>>> dis(c.code())
1 0 LOAD_GLOBAL 0 (foo)
3 LOAD_FAST 0 (q)
6 LOAD_CONST 1 ('x')
9 LOAD_CONST 2 (1)
12 LOAD_FAST 1 (starargs)
15 CALL_FUNCTION_VAR 257
18 RETURN_VALUE
>>> c = Code()
>>> c.set_lineno(1)
>>> c(Call(Global('foo'), [Local('q')], [('x',Const(1))],
... None, Local('kwargs'))
... )
>>> c.RETURN_VALUE()
>>> dis(c.code())
1 0 LOAD_GLOBAL 0 (foo)
3 LOAD_FAST 0 (q)
6 LOAD_CONST 1 ('x')
9 LOAD_CONST 2 (1)
12 LOAD_FAST 1 (kwargs)
15 CALL_FUNCTION_KW 257
18 RETURN_VALUE
Cloning::
>>> c = Code.from_function(lambda (x,y):1, True)
>>> dis(c.code())
1 0 LOAD_FAST 0 (.0)
3 UNPACK_SEQUENCE 2
6 STORE_FAST 1 (x)
9 STORE_FAST 2 (y)
>>> c = Code.from_function(lambda x,(y,(z,a,b)):1, True)
>>> dis(c.code())
1 0 LOAD_FAST 1 (.1)
3 UNPACK_SEQUENCE 2
6 STORE_FAST 2 (y)
9 UNPACK_SEQUENCE 3
12 STORE_FAST 3 (z)
15 STORE_FAST 4 (a)
18 STORE_FAST 5 (b)
TODO
====
* Constant folding
* ast_type(node): called function, Const, or node.__class__
* tuples are Const if their contents are; no other types are Const
* ast_children(node): tuple of argument values for curried types, const value,
or empty tuple. If node is a tuple, the value must be flattened.
* is_const(node): ast_type(node) is Const
* const_value(node): ast_children(node)[0]
* Call() does the actual folding
* Inline builtins (getattr, operator.getitem, etc.) to opcodes
* Getattr/Op/Unary("symbol", arg1 [, arg2]) node types -> Call() if folding
* Call() translates functions back to Ops if inlining
* Pretty printing and short-naming of ASTs
* Test NAME vs. FAST operators flag checks/sets
* Test code flags generation/cloning
* Document block handling (SETUP_*, POP_BLOCK, END_FINALLY) and make sure the
latter two have correct stack tracking
|
cvs-admin@eby-sarna.com Powered by ViewCVS 1.0-dev |
ViewCVS and CVS Help |