.. _BytecodeAssembler reference manual: http://peak.telecommunity.com/DevCenter/BytecodeAssembler#toc |
.. _BytecodeAssembler reference manual: http://peak.telecommunity.com/DevCenter/BytecodeAssembler#toc |
|
|
|
|
|
Changes since version 0.6: |
|
|
|
* Experimental Python 3 support, including emulation of restored |
|
``BINARY_DIVIDE``, ``UNARY_CONVERT``, and ``SLICE_#`` opcodes. |
|
|
|
Changes since version 0.5.2: |
|
|
|
* Symbolic disassembly with full emulation of backward-compatible |
|
``JUMP_IF_TRUE`` and ``JUMP_IF_FALSE`` opcodes on Python 2.7 -- tests now |
|
run clean on Python 2.7. |
|
|
|
* Support for backward emulation of Python 2.7's ``JUMP_IF_TRUE_OR_POP`` and |
|
``JUMP_IF_FALSE_OR_POP`` instructions on earlier Python versions; these |
|
emulations are also used in BytecodeAssembler's internal code generation, |
|
for maximum performance on 2.7+ (with no change to performance on older |
|
versions). |
|
|
Changes since version 0.5.1: |
Changes since version 0.5.1: |
|
|
* Initial support for Python 2.7's new opcodes and semantics changes, mostly |
* Initial support for Python 2.7's new opcodes and semantics changes, mostly |
that maps each ``set_lineno()`` to the corresponding position in the bytecode. |
that maps each ``set_lineno()`` to the corresponding position in the bytecode. |
|
|
And of course, the resulting code objects can be run with ``eval()`` or |
And of course, the resulting code objects can be run with ``eval()`` or |
``exec``, or used with ``new.function`` to create a function:: |
``exec``, or used with ``new.function``/``types.FunctionType`` to create a |
|
function:: |
|
|
>>> eval(c.code()) |
>>> eval(c.code()) |
42 |
42 |
|
|
>>> exec c.code() # exec discards the return value, so no output here |
>>> exec(c.code()) # exec discards the return value, so no output here |
|
|
>>> import new |
>>> try: |
>>> f = new.function(c.code(), globals()) |
... from new import function |
|
... except ImportError: # Python 3 workarounds |
|
... from types import FunctionType as function |
|
... long = int |
|
... unicode = str |
|
|
|
>>> f = function(c.code(), globals()) |
>>> f() |
>>> f() |
42 |
42 |
|
|
>>> c.RETURN_VALUE() |
>>> c.RETURN_VALUE() |
|
|
>>> eval(c.code()) # computes type(27) |
>>> eval(c.code()) # computes type(27) |
<type 'int'> |
<... 'int'> |
|
|
>>> c = Code() |
>>> c = Code() |
>>> c.LOAD_CONST(dict) |
>>> c.LOAD_CONST(dict) |
``None``, or Python code object, it is treated as though it was passed to |
``None``, or Python code object, it is treated as though it was passed to |
the ``LOAD_CONST`` method directly:: |
the ``LOAD_CONST`` method directly:: |
|
|
|
|
|
|
>>> c = Code() |
>>> c = Code() |
>>> c(1, 2L, 3.0, 4j+5, "6", u"7", False, None, c.code()) |
>>> c(1, long(2), 3.0, 4j+5, "6", unicode("7"), False, None, c.code()) |
>>> dis(c.code()) |
>>> dis(c.code()) |
0 0 LOAD_CONST 1 (1) |
0 0 LOAD_CONST 1 (1) |
3 LOAD_CONST 2 (2L) |
3 LOAD_CONST 2 (2...) |
6 LOAD_CONST 3 (3.0) |
6 LOAD_CONST 3 (3.0) |
9 LOAD_CONST 4 ((5+4j)) |
9 LOAD_CONST 4 ((5+4j)) |
12 LOAD_CONST 5 ('6') |
12 LOAD_CONST 5 ('6') |
15 LOAD_CONST 6 (u'7') |
15 LOAD_CONST 6 (...'7') |
18 LOAD_CONST 7 (False) |
18 LOAD_CONST 7 (False) |
21 LOAD_CONST 0 (None) |
21 LOAD_CONST 0 (None) |
24 LOAD_CONST 8 (<code object <lambda> at ...>) |
24 LOAD_CONST 8 (<code object <lambda>...>) |
|
|
Note that although some values of different types may compare equal to each |
Note that although some values of different types may compare equal to each |
other, ``Code`` objects will not substitute a value of a different type than |
other, ``Code`` objects will not substitute a value of a different type than |
the one you requested:: |
the one you requested:: |
|
|
>>> c = Code() |
>>> c = Code() |
>>> c(1, True, 1.0, 1L) # equal, but different types |
>>> c(1, True, 1.0) # equal, but different types |
>>> dis(c.code()) |
>>> dis(c.code()) |
0 0 LOAD_CONST 1 (1) |
0 0 LOAD_CONST 1 (1) |
3 LOAD_CONST 2 (True) |
3 LOAD_CONST 2 (True) |
6 LOAD_CONST 3 (1.0) |
6 LOAD_CONST 3 (1.0) |
9 LOAD_CONST 4 (1L) |
|
|
|
Simple Containers |
Simple Containers |
----------------- |
----------------- |
|
|
|
|
>>> Getattr(Const(object), '__class__') # const expression, const result |
>>> Getattr(Const(object), '__class__') # const expression, const result |
Const(<type 'type'>) |
Const(<... 'type'>) |
|
|
Or the attribute name can be an expression, in which case a ``getattr()`` call |
Or the attribute name can be an expression, in which case a ``getattr()`` call |
is compiled instead:: |
is compiled instead:: |
AssertionError: Label previously defined |
AssertionError: Label previously defined |
|
|
|
|
|
More Conditional Jump Instructions |
|
---------------------------------- |
|
|
|
In Python 2.7, the traditional ``JUMP_IF_TRUE`` and ``JUMP_IF_FALSE`` |
|
instructions were replaced with four new instructions that either conditionally |
|
or unconditionally pop the value being tested. This was done to improve |
|
performance, since virtually all conditional jumps in Python code pop the |
|
value on one branch or the other. |
|
|
|
To provide better cross-version compatibility, BytecodeAssembler emulates the |
|
old instructions on Python 2.7 by emitting a ``DUP_TOP`` followed by a |
|
``POP_JUMP_IF_FALSE`` or ``POP_JUMP_IF_TRUE`` instruction. |
|
|
|
However, since this decreases performance, BytecodeAssembler *also* emulates |
|
Python 2.7's ``JUMP_IF_FALSE_OR_POP`` and ``JUMP_IF_FALSE_OR_TRUE`` opcodes |
|
on *older* Pythons:: |
|
|
|
>>> c = Code() |
|
>>> l1, l2 = Label(), Label() |
|
>>> c(Local('a'), l1.JUMP_IF_FALSE_OR_POP, Return(27), l1) |
|
>>> c(l2.JUMP_IF_TRUE_OR_POP, Return(42), l2, Code.RETURN_VALUE) |
|
>>> dump(c.code()) |
|
LOAD_FAST 0 (a) |
|
JUMP_IF_FALSE L1 |
|
POP_TOP |
|
LOAD_CONST 1 (27) |
|
RETURN_VALUE |
|
L1: JUMP_IF_TRUE L2 |
|
POP_TOP |
|
LOAD_CONST 2 (42) |
|
RETURN_VALUE |
|
L2: RETURN_VALUE |
|
|
|
This means that you can immediately begin using the "or-pop" variations, in |
|
place of a jump followed by a pop, and BytecodeAssembler will use the faster |
|
single instruction automatically on Python 2.7+. |
|
|
|
BytecodeAssembler *also* supports using Python 2.7's conditional jumps |
|
that do unconditional pops, but currently cannot emulate them on older Python |
|
versions, so at the moment you should use them only when your code requires |
|
Python 2.7. |
|
|
|
(Note: for ease in doctesting across Python versions, the ``dump()`` function |
|
*always* shows the code as if it were generated for Python 2.6 or lower, so |
|
if you need to check the *actual* bytecodes generated, you must use Python's |
|
``dis.dis()`` function instead!) |
|
|
|
|
N-Way Comparisons |
N-Way Comparisons |
----------------- |
----------------- |
|
|
|
|
>>> from peak.util.assembler import const_value |
>>> from peak.util.assembler import const_value |
|
|
>>> simple_values = [1, 2L, 3.0, 4j+5, "6", u"7", False, None, c.code()] |
>>> simple_values = [1, long(2), 3.0, 4j+5, "6", unicode("7"), False, None, c.code()] |
|
|
>>> map(const_value, simple_values) |
>>> list(map(const_value, simple_values)) |
[1, 2L, 3.0, (5+4j), '6', u'7', False, None, <code object <lambda> ...>] |
[1, 2..., 3.0, (5+4j), '6', ...'7', False, None, <code object <lambda>...>] |
|
|
Values wrapped in a ``Const()`` are also returned as-is:: |
Values wrapped in a ``Const()`` are also returned as-is:: |
|
|
>>> map(const_value, map(Const, simple_values)) |
>>> list(map(const_value, map(Const, simple_values))) |
[1, 2L, 3.0, (5+4j), '6', u'7', False, None, <code object <lambda> ...>] |
[1, 2..., 3.0, (5+4j), '6', ...'7', False, None, <code object <lambda>...>] |
|
|
But no other node types produce constant values; instead, ``NotAConstant`` is |
But no other node types produce constant values; instead, ``NotAConstant`` is |
raised:: |
raised:: |
>>> const_value(Local('x')) |
>>> const_value(Local('x')) |
Traceback (most recent call last): |
Traceback (most recent call last): |
... |
... |
NotAConstant: Local('x') |
peak.util.assembler.NotAConstant: Local('x') |
|
|
Tuples of constants are recursively replaced by constant tuples:: |
Tuples of constants are recursively replaced by constant tuples:: |
|
|
>>> const_value( (1,Global('y')) ) |
>>> const_value( (1,Global('y')) ) |
Traceback (most recent call last): |
Traceback (most recent call last): |
... |
... |
NotAConstant: Global('y') |
peak.util.assembler.NotAConstant: Global('y') |
|
|
As do any types not previously described here:: |
As do any types not previously described here:: |
|
|
>>> const_value([1,2]) |
>>> const_value([1,2]) |
Traceback (most recent call last): |
Traceback (most recent call last): |
... |
... |
NotAConstant: [1, 2] |
peak.util.assembler.NotAConstant: [1, 2] |
|
|
Unless of course they're wrapped with ``Const``:: |
Unless of course they're wrapped with ``Const``:: |
|
|
``Const`` node instead of a ``Call`` node:: |
``Const`` node instead of a ``Call`` node:: |
|
|
>>> Call( Const(type), [1] ) |
>>> Call( Const(type), [1] ) |
Const(<type 'int'>) |
Const(<... 'int'>) |
|
|
Thus, you can also take the ``const_value()`` of such calls:: |
Thus, you can also take the ``const_value()`` of such calls:: |
|
|
passed in to another ``Call``:: |
passed in to another ``Call``:: |
|
|
>>> Call(Const(type), [Call( Const(dict), [], [('x',27)] )]) |
>>> Call(Const(type), [Call( Const(dict), [], [('x',27)] )]) |
Const(<type 'dict'>) |
Const(<... 'dict'>) |
|
|
Notice that this folding takes place eagerly, during AST construction. If you |
Notice that this folding takes place eagerly, during AST construction. If you |
want to implement delayed folding after constant propagation or variable |
want to implement delayed folding after constant propagation or variable |
>>> c = Code() |
>>> c = Code() |
>>> c( Call(Const(type), [1]) ) |
>>> c( Call(Const(type), [1]) ) |
>>> dis(c.code()) |
>>> dis(c.code()) |
0 0 LOAD_CONST 1 (<type 'int'>) |
0 0 LOAD_CONST 1 (<... 'int'>) |
|
|
>>> c = Code() |
>>> c = Code() |
>>> c( Call(Const(type), [1], fold=False) ) |
>>> c( Call(Const(type), [1], fold=False) ) |
>>> dis(c.code()) |
>>> dis(c.code()) |
0 0 LOAD_CONST 1 (<type 'type'>) |
0 0 LOAD_CONST 1 (<... 'type'>) |
3 LOAD_CONST 2 (1) |
3 LOAD_CONST 2 (1) |
6 CALL_FUNCTION 1 |
6 CALL_FUNCTION 1 |
|
|
... if const_value(value): |
... if const_value(value): |
... continue # true constants can be skipped |
... continue # true constants can be skipped |
... except NotAConstant: # but non-constants require code |
... except NotAConstant: # but non-constants require code |
... code(value, end.JUMP_IF_FALSE, Code.POP_TOP) |
... code(value, end.JUMP_IF_FALSE_OR_POP) |
... else: # and false constants end the chain right away |
... else: # and false constants end the chain right away |
... return code(value, end) |
... return code(value, end) |
... code(values[-1], end) |
... code(values[-1], end) |
>>> Getattr = nodetype()(Getattr) |
>>> Getattr = nodetype()(Getattr) |
|
|
>>> const_value(Getattr(1, '__class__')) |
>>> const_value(Getattr(1, '__class__')) |
<type 'int'> |
<... 'int'> |
|
|
The ``fold_args()`` function tries to evaluate the node immediately, if all of |
The ``fold_args()`` function tries to evaluate the node immediately, if all of |
its arguments are constants, by creating a temporary ``Code`` object, and |
its arguments are constants, by creating a temporary ``Code`` object, and |
... pass |
... pass |
|
|
>>> c = Code.from_function(f1) |
>>> c = Code.from_function(f1) |
>>> f2 = new.function(c.code(), globals()) |
>>> f2 = function(c.code(), globals()) |
|
|
>>> import inspect |
>>> import inspect |
|
|
>>> def f3(a, (b,c), (d,(e,f))): |
>>> def f3(a, (b,c), (d,(e,f))): |
... pass |
... pass |
|
|
>>> f4 = new.function(Code.from_function(f3).code(), globals()) |
>>> f4 = function(Code.from_function(f3).code(), globals()) |
>>> dis(f4) |
>>> dis(f4) |
0 0 LOAD_FAST 1 (.1) |
0 0 LOAD_FAST 1 (.1) |
3 UNPACK_SEQUENCE 2 |
3 UNPACK_SEQUENCE 2 |
>>> c.LOAD_CONST(42) |
>>> c.LOAD_CONST(42) |
>>> c.RETURN_VALUE() |
>>> c.RETURN_VALUE() |
|
|
>>> f = new.function(c.code(), globals()) |
>>> f = function(c.code(), globals()) |
>>> f(1,2,3) |
>>> f(1,2,3) |
42 |
42 |
|
|
|
|
>>> c = Code() |
>>> c = Code() |
>>> fwd = c.JUMP_FORWARD() |
>>> fwd = c.JUMP_FORWARD() |
>>> print c.stack_size # forward jump marks stack size as unknown |
>>> print(c.stack_size) # forward jump marks stack size as unknown |
None |
None |
|
|
>>> c.LOAD_CONST(42) |
>>> c.LOAD_CONST(42) |
... return cond, then, else_ |
... return cond, then, else_ |
... else_clause = Label() |
... else_clause = Label() |
... end_if = Label() |
... end_if = Label() |
... code(cond, else_clause.JUMP_IF_FALSE, Code.POP_TOP, then) |
... code(cond, else_clause.JUMP_IF_FALSE_OR_POP, then) |
... code(end_if.JUMP_FORWARD, else_clause, Code.POP_TOP, else_) |
... code(end_if.JUMP_FORWARD, else_clause, Code.POP_TOP, else_) |
... code(end_if) |
... code(end_if) |
>>> If = nodetype()(If) |
>>> If = nodetype()(If) |
... return cond, then, else_ |
... return cond, then, else_ |
... else_clause = Label() |
... else_clause = Label() |
... end_if = Label() |
... end_if = Label() |
... code(cond, else_clause.JUMP_IF_FALSE, Code.POP_TOP, then) |
... code(cond, else_clause.JUMP_IF_FALSE_OR_POP, then) |
... if code.stack_size is not None: |
... if code.stack_size is not None: |
... end_if.JUMP_FORWARD(code) |
... end_if.JUMP_FORWARD(code) |
... code(else_clause, Code.POP_TOP, else_, end_if) |
... code(else_clause, Code.POP_TOP, else_, end_if) |
POP_BLOCK |
POP_BLOCK |
JUMP_FORWARD L4 |
JUMP_FORWARD L4 |
L1: DUP_TOP |
L1: DUP_TOP |
LOAD_CONST 2 (<...exceptions.KeyError...>) |
LOAD_CONST 2 (<...KeyError...>) |
COMPARE_OP 10 (exception match) |
COMPARE_OP 10 (exception match) |
JUMP_IF_FALSE L2 |
JUMP_IF_FALSE L2 |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP... |
LOAD_CONST 3 (2) |
LOAD_CONST 3 (2) |
JUMP_FORWARD L5 |
JUMP_FORWARD L5 |
L2: POP_TOP |
L2: POP_TOP |
DUP_TOP |
DUP_TOP |
LOAD_CONST 4 (<...exceptions.TypeError...>) |
LOAD_CONST 4 (<...TypeError...>) |
COMPARE_OP 10 (exception match) |
COMPARE_OP 10 (exception match) |
JUMP_IF_FALSE L3 |
JUMP_IF_FALSE L3 |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP... |
LOAD_CONST 5 (3) |
LOAD_CONST 5 (3) |
JUMP_FORWARD L5 |
JUMP_FORWARD L5 |
L3: POP_TOP |
L3: POP_TOP |
clause, and a loop body:: |
clause, and a loop body:: |
|
|
>>> from peak.util.assembler import For |
>>> from peak.util.assembler import For |
>>> y = Call(Const(range), (3,)) |
>>> y = Call(Const(list), (Call(Const(range), (3,)),)) |
>>> x = LocalAssign('x') |
>>> x = LocalAssign('x') |
>>> body = Suite([Local('x'), Code.PRINT_EXPR]) |
>>> body = Suite([Local('x'), Code.PRINT_EXPR]) |
|
|
>>> c.return_(Function(Return(Local('a')), 'f', ['a'], defaults=[42])) |
>>> c.return_(Function(Return(Local('a')), 'f', ['a'], defaults=[42])) |
>>> dis(c.code()) |
>>> dis(c.code()) |
0 0 LOAD_CONST 1 (42) |
0 0 LOAD_CONST 1 (42) |
3 LOAD_CONST 2 (<... f ..., file "<string>", line -1>) |
3 LOAD_CONST 2 (<... f..., file ...<string>..., line ...>) |
6 MAKE_FUNCTION 1 |
6 MAKE_FUNCTION 1 |
9 RETURN_VALUE |
9 RETURN_VALUE |
|
|
>>> dis(c.code()) |
>>> dis(c.code()) |
0 0 LOAD_CONST 1 (99) |
0 0 LOAD_CONST 1 (99) |
3 LOAD_CONST 2 (66) |
3 LOAD_CONST 2 (66) |
6 LOAD_CONST 3 (<... f ..., file "<string>", line -1>) |
6 LOAD_CONST 3 (<... f..., file ...<string>..., line ...>) |
9 MAKE_FUNCTION 2 |
9 MAKE_FUNCTION 2 |
12 RETURN_VALUE |
12 RETURN_VALUE |
|
|
|
|
>>> dis(f) |
>>> dis(f) |
0 0 LOAD_CLOSURE 0 (a) |
0 0 LOAD_CLOSURE 0 (a) |
... LOAD_CONST 1 (<... <lambda> ..., file "<string>", line -1>) |
... LOAD_CONST 1 (<... <lambda>..., file ...<string>..., line ...>) |
... MAKE_CLOSURE 0 |
... MAKE_CLOSURE 0 |
... RETURN_VALUE |
... RETURN_VALUE |
|
|
>>> c = Code() |
>>> c = Code() |
>>> else_ = Label() |
>>> else_ = Label() |
>>> end = Label() |
>>> end = Label() |
>>> c(99, else_.JUMP_IF_TRUE, Code.POP_TOP, end.JUMP_FORWARD) |
>>> c(99, else_.JUMP_IF_TRUE_OR_POP, end.JUMP_FORWARD) |
>>> c(else_, Code.POP_TOP, end) |
>>> c(else_, Code.POP_TOP, end) |
>>> dump(c.code()) |
>>> dump(c.code()) |
LOAD_CONST 1 (99) |
LOAD_CONST 1 (99) |
>>> c.stack_size |
>>> c.stack_size |
0 |
0 |
>>> if sys.version>='2.7': |
>>> if sys.version>='2.7': |
... print c.stack_history == [0, 1, 1, 1, 2, 1, 1, 1, 0, None, None, 1] |
... print(c.stack_history == [0, 1, 1, 1, 0, 0, 0, None, None, 1]) |
... else: |
... else: |
... print c.stack_history == [0, 1, 1, 1, 1, 1, 1, 0, None, None, 1] |
... print(c.stack_history == [0, 1, 1, 1, 1, 1, 1, 0, None, None, 1]) |
True |
True |
|
|
|
|
>>> c = Code() |
>>> c = Code() |
>>> c.return_(Call(Const(type), [], [], (1,))) |
>>> c.return_(Call(Const(type), [], [], (1,))) |
>>> dis(c.code()) |
>>> dis(c.code()) |
0 0 LOAD_CONST 1 (<type 'int'>) |
0 0 LOAD_CONST 1 (<... 'int'>) |
3 RETURN_VALUE |
3 RETURN_VALUE |
|
|
|
|
POP_BLOCK |
POP_BLOCK |
JUMP_FORWARD L3 |
JUMP_FORWARD L3 |
L1: DUP_TOP |
L1: DUP_TOP |
LOAD_CONST 1 (<...exceptions.AttributeError...>) |
LOAD_CONST 1 (<...AttributeError...>) |
COMPARE_OP 10 (exception match) |
COMPARE_OP 10 (exception match) |
JUMP_IF_FALSE L2 |
JUMP_IF_FALSE L2 |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP |
POP_TOP... |
LOAD_CONST 2 (<type 'type'>) |
LOAD_CONST 2 (<... 'type'>) |
ROT_TWO |
ROT_TWO |
CALL_FUNCTION 1 |
CALL_FUNCTION 1 |
JUMP_FORWARD L3 |
JUMP_FORWARD L3 |
END_FINALLY |
END_FINALLY |
L3: RETURN_VALUE |
L3: RETURN_VALUE |
|
|
>>> type_or_class.func_code = c.code() |
>>> type_or_class.__code__ = type_or_class.func_code = c.code() |
>>> type_or_class(23) |
>>> type_or_class(23) |
<type 'int'> |
<... 'int'> |
|
|
|
|
|
|
>>> c(Switch(Local('x'), [(1,Return(42)),(2,Return("foo"))], Return(27))) |
>>> c(Switch(Local('x'), [(1,Return(42)),(2,Return("foo"))], Return(27))) |
>>> c.return_() |
>>> c.return_() |
|
|
>>> f = new.function(c.code(), globals()) |
>>> f = function(c.code(), globals()) |
>>> f(1) |
>>> f(1) |
42 |
42 |
>>> f(2) |
>>> f(2) |
|
|
>>> dump(c.code()) |
>>> dump(c.code()) |
SETUP_LOOP L2 |
SETUP_LOOP L2 |
LOAD_CONST 1 (<...method get of dict...>) |
LOAD_CONST 1 (<...method ...get of ...>) |
LOAD_FAST 0 (x) |
LOAD_FAST 0 (x) |
CALL_FUNCTION 1 |
CALL_FUNCTION 1 |
JUMP_IF_FALSE L1 |
JUMP_IF_FALSE L1 |
* Exhaustive tests of all opcodes' stack history effects |
* Exhaustive tests of all opcodes' stack history effects |
|
|
* Test wide jumps and wide argument generation in general |
* Test wide jumps and wide argument generation in general |
|
|
|
* Remove/renumber local variables when a local is converted to free/cell |
|
|