entire evoque site in a single printable page
evoque - managed eval-based freeform templating
Evoque is a full-featured and generic text templating system for python using a simple $-substitution syntax and providing support for flow control, nested templates, overlays, inter-template addressing and invocation, cache management, arbitrary python expressions, all python % operator formatting, restricted execution, automatic cross-site scripting protection, advanced encoding guessing algorithm, and more.
Allowing only python expressions and a managed evaluation namespace, Evoque offers a surprising level of simplicity, versatility and performance.
Buzz
- Full-featured pure python templating engine / 972 SLOC
- Automatic input quoting / XSS protection
- Restricted execution
- Dynamic template inheritance
- Every text file is a template
- Template collections
- Unicode
- Simplicity
- Speed
Snapshot
Usage from within an application
Templates are always part of a Collection in a Domain -- a Domain and a default Collection instance are always created, be it explicitly or implicitly.
The preferred way to load/retrieve a Template is via a Domain instance; the following code implicitly creates the default Collection instance:
It may sometimes be more convenient to instantiate directly; the following code implicitly creates the Domain and the default Collection instances:
Basic benchmark
A basic benchmark, consisting of a small template that does a variety of standard templating tasks such as looping, calling nested and external sub-templates, escaping of data, etc. For each run, time is averaged over 2000 renderings, and the best time of 4 runs is retained. All times are in ms. What you should want is automatic quoting and less time.
| quoting | qpy 1.6 | evoque 0.1.2 | mako 0.2.2 | genshi 0.5.1 |
|---|---|---|---|---|
| automatic | 0.17 | 0.55 | 0.60 | 4.10 |
| manual | 0.53 | 0.60 | ||
| none | 0.43 | 0.43 | ||
| restricted | 0.66 | |||
| manual quoting implies no qpy | xml mode |
Feature highlights
Full-featured generic templating engine, with rich features such as overlays and template caching.
Small footprint, coming in at 972 SLOC. And 146 of those SLOC are consumed by the generic and fairly advanced algorithm for guessing a string's encoding (see a separate discussion of the decodeh module). On the other hand this number does not include the code for the unit tests or the benchmarks.
Unicode. All internal processing is done on unicode instances. To facilitate this Evoque includes a sophisticated encoding guessing algorithm. Any and all encoding of the output is left to the liberty of the client application.
Template addressing. Every template, nested or not, is evoque'able from within any other.
Conditionals, loops, nested templates, evoque'ations, comments may all be liberally intermixed and nested.
Automatic input quoting / XSS protection thanks to the h8 quoted-string class from the Qpy Templating package, providing automatic cross-site scripting protection by guaranteeing that all input is always quoted and quoted only once. To benefit from this feature Qpy needs to be installed -- running without Qpy only means that you must take care to do your input quoting manually (which is just what some other text templating systems require you to do anyway). But, if you want to Avoid Being Called a Bozo When Producing XML, then this is one killer feature that you want!
Python expressions only, i.e. no python statements, nothing is ever exec'ed.
Restricted execution. Evaluation is within a managed, and optionally restricted, namespace; templates may be safely exposed to untrusted clients.
Easily introspectable templates, of their signatures and evaluation namespace.
Good template evaluation error handling making template development and debugging easy.
Applications may extend evaluation namespace with any python callable or, naturally, with Qpy Templates.
Every text file is a template. No particular distinction between template and content, i.e. any text source when used as one, is a template. Useful to assemble and evoque bits and pieces of content from any kind of source, and either rendered raw or evaluated.
Extremely dynamic e.g. see how easy it is to parametrize the base template of a template hierarchy chain per user.
Simplicity, with a trivially memorable syntax, just a handful of directives, easy management of different template collections, and general consistency.
Speed. The simplicity of the implementation, and the awesome python builtins and standard library, translate into what may well be the fastest pure python templating system around.
Other notes on the design and philosophy
- Templates are organized in Collections that belong to a Domain.
- Collections have a root folder -- all text files below are templates.
- All collections are explicitly named -- decouples template addressing from deployment paths and protects sensitive path info within templates.
- The creation of the Domain also creates the default Collection, that is named "" (the empty string). The default collection may be given another name, by simply creating it first.
- Every template instance is associated to one and only one collection.
- All templates are assigned a nick name, unique within the collection: memory-based templates are always named explicitly; for file-based templates the default name is their collection-root-relative or c-rel locator.
- The src or location of any collection-qualified template is always specified as c-rel.
- A previously loaded template may be retrieved by its name and the collection.
- The eval globals dict belongs to Evoque and/or the Client Application, and is shared by all templates in a domain -- it is never modified after initialization.
- The eval locals dict belongs to the document being rendered, and is passed down the nested execution scope, possibly cloned and modified each time.
Template syntax
definition: An evoque template is either (a) a textual file or (b) a string in memory or (c) a begin/end delineated block in a textual file or a string, each optionally containing additional evoque-specific markup.
Entire template syntax at a glance
Explanations
Expressions
Expressions have the form ${expr !format} where the "!format" is optional, and defaults to "!s", i.e. default string rendering. The possible values for format are whatever is allowed by the python's string formating operations. The expr itself may be any valid python expression, that will be evaluated within the passed context (that may optionally be restricted). For example, the trivial template:
will produce the following output when rendered with
amount=1.0/3:
Directives
Directives have the form $directive{expr}.
Syntactically they are same as an expression but have an additional
qualifying keyword
inserted between the $ and the opening {.
See dedicated discussion on directives in general and for each
directive in the directives section.
Literals: "$", "{", "}", "!", "\"
A literal "$" must always be escaped.
A single "$" always signals the beginning of a substitution or a directive,
e.g $expr has no specific meaning for Evoque and will
give a SyntaxError
(the "$" used to delineate any expanded svn keywords are exempted from this).
Should always use either $$expr or ${expr} instead,
depending on what is desired.
A literal "\" at end of a line slurps the newline character but a literal "\" followed by white space (only) until the first newline character is put out as is but without the white space, i.e. "\ \n" outputs ""\\n"".
All other literals within the text are never escaped.
Within an expression it is sometimes necessary to use literal braces
e.g. to specify a dict value in place, such as key={"a":1},
or a literal piece of CSS such as "tag.class{display:inline}".
The alternative "{%" and "%}" expression delimeters are provided
to comfortably allow for this.
Whitespace
A "\" at end-of-line consumes the following newline, providing a basic
whitespace control. In addition to this, the
slurpy_directives=True on domain or on
each collection will consume all trailing whitespace
after a directive.
In addition, all leading or trailing whitespace within expressions and directives, i.e. between the curly braces, even if multi-line, is always ignored.
Comments
Single or multi line comments are delineated with
#[ and ]#,
respectively. For the convenience of easily commenting out
a template section (even if that section itself already
contains comments) comments may be nested. This however
necessitates a minimum syntax to be respected, namely that
comment openings and closing are balanced.
Built-in callables
Any callable set on the evaluation context, either with
domain.set_on_globals() or by setting it on a
locals dict that is then passed as a parameter to
template.evoque(locals),
may be called as an expression, e.g.
${my_callable(params)}.
Evoque provides the following built-in callables.
inspect(output=True)
Pretty prints an overview of the evaluation namespace,
namely the context's globals/locals and the template's
expressions needing evaluation.
When output is set to False, the output is written
to domain.log.info() instead of rendered as
part of the template output.
evoque(name, **kw)
This is perfectly equivalent to the directive form of
$evoque{name, **kw}.
The difference is that in the callable variation
it is possible to intercept the output and
process it further via any other callable --
but note that it is recommended that such
processing be specified via the filters
template parameter.
Directives
Directives have the form $directive{expr}.
What expr may be depends on the directive, e.g.
for if and for the difference is that
for $if{expr} the expr
is a normal expression evaluated as a bool, and for
$for{expr in sequence}
it must have this specific form.
A directive clause does not always have an expr,
and in these cases the curly braces are not present,
e.g. $else, $fi, $rof.
Directives must respect their specific context,
e.g. if a $for{} is opened inside a nested template,
then it must be closed with $rof within the
same nested template.
Directives, in general, may be liberally intermixed and nested.
A begin/end block may include other begin/end
blocks, if blocks, for blocks,
an overlay declaration, etc.
However, specific constraints apply,
e.g. only one prefer and overlay
directives are allowed per template, nested or not, and they must be
declared at top level of the template to which they pertain.
There is no significance to whether directives and any
clauses are specified on a single line or on separate lines.
This has benefits such as allowing an inlined if/else
e.g. to calculate the value of an HTML attribute,
just as easily as multi-line if/else blocks
to control which section of a template gets rendered.
conditional if
The conditional if/elif/else construct,
apart from the closing $fi,
behaves just like you would expect i.e. as in python,
with optional (multiple) $elif{}'s and an optional
concluding $else.
The condition expression may be any python expression, that must evaluate without errors in the supplied namespace. As for all directives, being on a single line or spanning many lines has no relevance.
for loops
Again, the for loop directive behaves like you would
expect (like in python) except for additional $else
option for the case of an empty sequence
(note the difference to python's $else
that is executed at end unless loop exited with
break statement).
One should note that the statement of the loop must be made up
of an iterable on the right side of an in keyword,
and whatever is on the left side of the in statement
is the loop variable or variables that get set on the evaluation
context on each iteration. This may be a single atomic variable,
a flat sequence of variables, or even something more complex.
Here's an example of a 2-level tuple as left part of a
for loop statement:
That when rendered will produce:
begin/end blocks: sub-templates
The $begin{label} and the
$end{label} pair of directives
define a nested template, addressable from any other template.
A nested template has all the
characteristics of top-level file or string templates --
the only difference is in how they are addressed.
In fact, for the case of top-level file or string templates,
you may consider that the begin and end
directives are simply just implied by the beginning and end
of the string. The label may be considered
just as a named anchor, a destination inside the
file to which one can point to directly.
Parameters
There is really only one parameter, and that is the label that is what is used to address the template.
Rules
- A label defining a nested template (whatever the nesting level) may be used once and only once in a template file or string.
- Nested templates must be addressed via their label, using a syntax
similar to fragment identifiers of a URL, i.e.
#label. When being addressed locally, i.e. from same file or string, nested templates may be addressed with just#label. From within other templates, the information to identify the top-level template must also be specified, e.g.template_name#labelfor an explicitly named template, or by something liketemplate.html#labelfor a template implicitly named by its file name. - For overlay templates it is possible to explicitly specify what level the lookup for a locally addressed nested template should start from i.e. if the name starts with a single "#" then from current current, if with "##" then from one level down, if with "###" then from two levels down, etc.
- Nesting of templates may be multi-level; addressing is always flat,
i.e. a nested template may further nest other templates, but all templates
irrespective of nesting depth are addressed either (locally) with
#labelor withtemplate_name#label. - If only one of the
beginorenddirectives is present, then the other is implied by, respectively, the beginning or the end of the string. - A nested template is not automatically rendered when the template that defines it is rendered; it must be explicitly evoque'd.
- Everything already established about directives applies, e.g.
both
beginandendmay all be on same line, or each may span many lines; any of the two allowed delimeter pairs may be used; any whitespace around the label is ignored, etc.
Execution
Nested templates are the def, the macro, the callable block, the sub-template of Evoque. To be executed, they are evoque'd:
This will evaluate the nested template using a copy
of the caller's context.
Note that given that a nested template is a template in its own right,
it can specify its own default data (and other preferences)
that is what will be used when the caller's context does not
specify a required variable. If neither caller nor default data
specify a required variable, an evaluation error will occur
and will be reported as per the domain's errors
setting.
evoque: rendering a template
Any Evoque template, be it file-based or
string-based, top-level or nested, is addressable from any
other Evoque template in the domain.
From within a template, $evoque{name, **kw}
is the directive that gets and renders a template --
it combines the full functionality of the two methods you would
need to call to do the same thing from within an application, i.e. the
methods
Collection.get_template() and
Template.evoque().
Parameters
Rules
The only required parameter is name, that must always be
specified first, and most often the call is as simple as:
$evoque{name}.
Note that for convenience the name parameter may be
specified as an unquoted string literal i.e. when the name=
qualifier is not explicitly stated, the string found will be interpreted as
a literal even if it is not enclosed in quotes.
For example, for a template with name my_template, using any of
the following 3 forms is perfectly equivalent:
$evoque{my_template},
$evoque{"my_template"},
$evoque{name="my_template"}.
But, $evoque{name=my_template} will cause the runtime evaluation
of a my_template variable that will be used as the name.
The evoque directive gets the named template from the
specified collection -- if collection=None, then the
calling template's own collection is used, otherwise
collection must name an already existing collection.
Template may still need to be loaded.
If it is already loaded, the template is retrieved from the
cache, subject to cache staleness checking as per the
collections's cache settings.
The src, input_encoding parameters
are only used in the event that the template needs to be loaded:
- if template is not yet loaded then it is assumed to be a file-based template
and the location of the file to load is given by joining the collection's
base path and the value of
src. - if
src=Nonethen name is interpreted as the location of file to load.
All parameters other than name, src, collection,
if not specified, will be set to the collection's default.
The raw parameter determines whether the raw template
string is to be returned, without any evaluation, e.g. for the purpose of
editing the template string itself. Note that if the template is not yet
loaded and raw=True, then it will be loaded but not compiled.
See the raw source howto for more details.
The quoting parameter determines what the default quoting
class for the template is (if the template is not yet loaded) and/or which
quoting class to use for this evoque'ation.
Any specified filters,
that must be either None or a sequence of filter functions,
are executed after evaluation.
Filters must of course be available in the evaluation space.
An evoque'd sub-template will be quoted as per the quoting class of the calling template. This of course will only happen in the event that the two templates use different quoting classes.
Built-in callable
The $evoque{name, **kw} directive is also
provided as the perfectly equivalent built-in callable:
- "name" is now a normal python string so may be evaluated, i.e. if literal, then it must be quoted.
- may be intermingled with other callables, python builtins,
custom filters, etc. For example:
${markdown(evoque("my_mardown_template"))}
overlay
The $overlay{} directive defines a dependency chain of templates,
super-imposing a template on top of another.
Other template systems often call this functionality
template inheritance -- a term that may however be a little
misleading given the very specific connotations
from object-oriented programming,
and it is best to not add to any object-disorientation.
Nonetheless, a great feature of evoque overlays is that they are in many
ways similar in behaviour to the inheritance paradigm of object-oriented
programming, as for example the nice feature that an evoque template does
not care or even know about whether it is being used as a base template
to an overlay.
Layered positive and negative spaces
A simple way to think of an overlay is as a graphic, with layers that may be
transparent, and having positive and negative space. The positive space are
the sub-template objects on the page, i.e. the top-level
begin/end blocks defined by the template.
The negative space is everything else, i.e. all markup that is not
contained in any top-level begin/end block.
An overlay template may declare that it overlays the
space="positive"
or the space="negative" of the overlaid template:
- A positive-space overlay is one that may override any objects in the positive-space, i.e. top-level sub-template objects. Any markup in the negative-space of such a template is ignored. By default, overlays are positive-space.
- A negative-space overlay is required to supply the negative-space for the template. That negative space may evoque sub-templates from the overlay chain below as if they were defined locally. Any sub-template defined below that is not evoque'd from the supplied negative-space will not be rendered. Naturally, such a template may of course also override any top-level sub-templates that make up its positive space.
Examples
The pages below offer some overlay scenario examples. These examples are live, in the sense that the templates are included directly from the Evoque unit tests, and the generated output is included into these pages automatically and with zero touch-up or modification.
Overlay directive parameters
The parameters are the same as for the
evoque directive,
but with an additional space keyword (inserted at
the second position, after the name that is the only
one required) that as we have seen may be
either positive or negative:
Additional notes about overlays
- An overlay directive declares that this template overlays the specified template, in either a positive-space or in a negative-space mode.
- Only one overlay directive per template is allowed.
- Any template may be overlaid, and overlaid templates are just normal templates requiring no modifications.
- An overlaid template may itself overlay another template.
- In an overlay, overlaid templates may be addressed with multiple
leading
#characters, to indicate deeper levels into the overlay chain, i.e.:#labeltells to start lookup from self##labeltells to start lookup from one level below###labeltells to start lookup from two levels below, etc.specified_name#labelcontinues to work as in any other template, of course, irrespective of whether the template given byspecified_nameparticipates in the overlay chain or not.
overlay example - basic
Here is a simple example, showing a frequently occuring case
of having a template overlay another's positive-space.
We use a base template that defines a page layout, and three sub-templates.
We overlay the base.html template and override one of
the three sub-templates, namely #content.
The templates
base.html
- just a normal template, and it is just fine for standalone usage as is
- defines 3 nested templates, and evoques them
- no need to be aware of any overlays over it
overlay.html
- positive overlay template on base.html, own negative space is ignored
- overrides #content, one of the 3 nested templates defined by base.html
- if we add definitons for other nested (except #header and #footer that are defined in base) they would be ignored
- when evoque'd directly, this template is rendered with:
- the -ve space from base.html
- own #content
- the #header and #footer from base.html
Output
Rendering the overlay:
Will give the following output:
overlay example - positive
Let's extend the simple example to add an intermediate template, so we
therefore have an overlay chain conisting of 3 templates, namely
overlay_chain_pos.html over overlay_mid.html
over the unmodified base.html (from simple example).
The templates
overlay_mid.html
- an intermediate positive overlay on base.html
- the -ve space from base.html is imposed
- overrides one +ve block, #footer
overlay_chain_pos.html
- overrides the positive space of overlay_mid.html, redefining #content
- when evoque'd directly, this template is rendered with:
- the -ve space from below
- own #content
- the #header and #footer from below
Output
Rendering the overlay:
Will give the following output (note negative space used is that from base.html):
overlay example - negative
Let's modify the positive chain example a little to show a negative overlay.
We add the overlay_chain_neg.html template that is a negative
overlay on the unmodified overlay_mid.html + base.html
base templates (from previous example). This example also shows how to specify
specific levels in the overlay chain at which to start lookup.
overlay_chain_neg.html
- a negative overlay, thus using own negative space
- redefines #content and #footer
- evoques all 3 #footer templates in the overlay chain, showing how to:
- start lookup from self
- start lookup from "one level below"
- explicitly evoque a named template
Output
Rendering the overlay:
Will give the following output:
autoload preferences
The $prefer{} directive is a
declaration of autoload preferences, from within a template.
Only one per template is allowed.
When template is fetched on the fly e.g. when evoque'd
from another template, it may either have already been
loaded, compiled, and cached or it may still need to be
looked up and compiled -- in this latter case all parameters
that may affect how the template is compiled are needed.
Precisely for such a case, the $prefer{} directive
provides the convenience of specifying default parameter values
that will be used if corresponding values are not specified
when the template is loaded.
Parameters
The parameters are a subset of those of the evoque directive. A parameter of the prefer directive is only taken into consideration if:
- it has not been specified at template load time;
- the preferred value is different than the cascaded default value from the collection.
For details of each of these parameters, see the
evoque directive page.
auto testing a template
The $test{} directive specifies test data
within the template itself, that will be used to evaluate
the template when template.test() is called.
test directives have no effect on output generated
when a template is evoque'd.
Parameters
May specify sample data, as keyword parameters that the template may need. Positional arguments are not supported -- only keyword args.
From an application, any tests are executed with:
Rules
- Specify sample test data for the template, nested or not.
- May specify as many as desired, execution will cascade through them in the specified order, i.e. if a kw arg is not specified in current test, then its last defined value will be used for this test.
- If template defines any default
data, then that data is used as initial data for the tests (see Combined with template's default data, below). - Executed simply with
template.test(), and if any evaluation errors occur they are always raised irrespective of the setting fordomain.errors. -
template.test()returns a list of responses, one for eachtestdirective. If notestdirective is specified, a test will always be attempted with either the template's default data or with no data. -
template.test()on arawtemplate returnsNone. - A
testdirective must occur at the template's top level. - May also be used to prime the
template.evaluator.codescache.
Example
Doing a template.test() from an application will thus evaluate
this template 3 times, one for each test directive specified. Any parameter
not specified in a test directive will inherit its value from
the previous test directive.
In this case the test data provides for coverage of all the possible
logical paths, so every piece of the template is exercised.
Combined with template's default data
Templates may be set-up to define and remember own default data.
This may be done either at initiaization (data init parameter)
or via a $prefer{data=dict(**kw)} directive.
If a template's data attribute is set, the test()
method will use that value as the starting point for the test data cascade.
To illustrate this relation, here's the example above, adjusted with a
$prefer{} directive and still giving identical
test() results:
Usage from within an application
Templates are always part of a Collection in a Domain -- a Domain and a default Collection instance are always created, be it explicitly or implicitly.
The preferred way to load/retrieve a Template is via a Domain instance; the following code implicitly creates the default Collection instance:
It may sometimes be more convenient to instantiate directly; the following code implicitly creates the Domain and the default Collection instances:
Evoque is conceived with safety and security in mind. Under no circumstances, for example, is it ever possible to evoque a template from a file that is not within a declared collection. Given that there is no special distinction between templates and content, template collection root directories are treated in the same way that a web server treats its document-root.
Other notes
Given a template instance, you can always get the associated
collection and domain with:
To create the (default) collection first, thus having an opportunity to name it explicitly, you do:
domain, collection, template
A template is always part of a collection in a domain.
The implications of this are that those related objects are always created, even if we are still free to create them explicitly in any order we prefer. Another implication is consistency and simplicity of the implementation, reducing the need to handle other scenarios.
The preferred usage is to first explicitly create the Domain instance, that will by default create the default empty string named Collection.
Domain
Collection
Template
quoting or from template.qsclass).
If locals is None, use a copy of "own" locals.
If raw or template.raw is True, return raw string
without evaluation.
Any kw args are set onto locals before evaluation.
"""
test() -> either(None, [qsclass])
""" Return a rendering per test directive """
unload() -> None
""" Unloads self from collection """
Note there are no instance attributes that correspond directly to the src, from_string and quoting init parameters.
managing the collection's template cache
Each collection controls its own cache, customizable via two parameters that get their default values from the domain:
cache_size: int = 0
sets the maximum number of loaded templates in a collection, with a value of 0 meaning no limit.auto_reload: int = 60
sets the minimum number of seconds to wait before doing astaton the template file to see if it has been modified. A value of 0 means to do the file system check on each request for the template.
The collection cache is always on.
If you prefer to run like there was no cache, i.e. reloading
every template each time it is requested, you can set
cache_size=1 and auto_reload=0.
If, on the other hand, you would like to run such that
templates are virtually never reloaded, you can set
cache_size=0 and
auto_reload=60*60*24,
i.e. a sufficiently large number of seconds to wait
between checks.
template error handling
Template errors may occur in one of two contexts -- during loading or during evaluation. It is of course possible that a loading error of one template occurs during the evaluation of another, for example when a template evoques another template that is not yet loaded. Thus the big differentiator for errors is not the type of error but when an error happens.
All errors not occurring during evaluation are raised as exceptions, as any other normal application error.
The behaviour of evaluation errors may be controlled via two domain parameters:
errors: int = 3
sets evaluation behaviour for errors, and may be an int between 0 and 4:4 = raise
will re-raise the exception given by the error.3 = render
will render the error as a 1-line descriptive summary of the error.2 = name
will render the error as the indicative string:EvalError[expr], where expr is the actual source of the failed expression.1 = zero
(not yet implemented) assume the zero object for the inferred python type for the failed expression being evaluated. Examples, if astrthen proceed with the empty string, if alistproceed with empty list, if anintproceed with 0, etc.0 = silent
no output is generated for the error, i.e. errors are rendered as the the empty string.
log: logging.getLogger("evoque")
sets the logger to be used for all evaluation errors; independently of the setting oferrors, evaluation errors are always logged. Also, the output of callinginspect(False)from within a template are sent to log.info().
restricted execution
Evoque supports
python expressions only. This means that
python statements are already naturally restricted as
when evaluated these will always raise a
SyntaxError.
The problem of restricting execution is thus reduced
to the problem of making the eval python built-in
safe. This is by no means an easy task, but at least with its
more limited scope it is simpler than attempting to
generically restrict or sandbox the python interpreter.
Running in restricted mode will (supposedly) make it impossible to manipulate tangible resources, such as files and sockets, from within a template. Protection of intangible resources, such as memory and CPU usage, is not (yet) provided (see below).
restricted: bool = False
This is the one domain init parameter that when set
to True will initialize the domain to
run in restricted execution mode.
Setting this to True has therefore the following
consequences:
- Sets a dummy
__builtins__empty dict on the domain-wide globals dict used by eval. - Adds a top-level entry, domain-wide globals dict used by eval, for each builtin deemed safe.
- At runtime scans all expressions before
compiling them for
"__"(double underscore) and if found will raise aLookupError.
Builtins deemed unsafe
All subclasses of BaseException are considered
potentially unsafe and so are not made available as entries on the
globals dict.
Other python 2.5 __builtins__ considered unsafe
are (defined in the list Domain.DISALLOW_BUILTINS):
This may seem overly restrictive, but then templating should not require more than a small subset of python's possibilities, for convenience of doing simple progamming tasks e.g. enumerating over a list.
Is this enough?
Depending on paranoia level, probably not. Besides more testing to validate that the above effectively does not allow the template programmer to access and manipulate any tangible resources, it should still be possible to bring down the interpreter via DOS maliciousness, e.g. evaluating an expression for a very large multiplication to consume all available memory, or to take a very long time to finish. Further runtime analysis of expressions will be required to protect against intangible resources such as memory and CPU usage, and how best to add such protection is future work that is still to be done. Restricted execution mode should for now best be considered experimental.
managing the namespace
Besides turning restricted mode on or off, customization control of the evaluation context is available by means of:
The domain methods allow adding of objects onto the domain-wide globals dict.
no_underscored: bool = True
By default, the set_namespace_on_globals()
methods look for an __all__ attribute on
the source object, and if one is found will only set
the what is listed by it.
If no one is found, it will import all attributes
but by default not those starting with an underscore --
behaviour that may be changed by setting
no_underscored = False.
Passing a prepared mylocals dict to
template.evoque() can additionally expose
specific objects per template evoque'ation.
Howto
Because everything is sometimes not immediately obvious ;-)
Sub-templates in markdown
This amounts to just specifying the markdown filter on the output of the evoque'd template. It is easy to see how it works with an example (coming straight from the unit test data included with the Evoque distribution).
The template
markdown.html
For simplicity, we just use a single file, where the top-level template evoques a locally-nested markdown template, but of course the source of the sub-template could be anything you like. Notice that all the outer template does is evoque the inner template into a <div/>, specifying the filters to use as well as some default data.
Output
We must supply the filter we would like to use, so we need to set it on the globals of the evaluation namespace. We can then render the template:
That will produce the following output:
Notice that automatic quoting of incoming data also just happens for sub-templates in markdown or in any other format.
Alternatives
Declaring preferred filters within template itself
A template may declare itself what filters should be used.
Sometimes we may not have write access to a template source, but in
those cases it would be easy to just read that source and wrap it into
a another template under our control. The template below
issues a $prefer{} directive with
filters=[markdown] as well as some default data,
and gives identical results as above, but relieves the caller
from the need of specifying the filters or any default data on each call.
Using only python callables
Just to get a liitle closer to what is going on underneath...
declaring filters on the $evoque{} call
or the $prefer{} directive of a template is really just
a little syntactic sugar over doing everything
with pure python callables. Here's what such an alternate version would
look like (here we are required to use the callable version of
the evoque directive):
markdown_callables.html
This will produce the exact same output when rendered as above.
Using without qpy and still doing html quoting
Qpy is only used when quoting="xml", thus if you run with
quoting="str" you will not need to have Qpy installed, but you
will need to take care of quoting yourself.
As an example, let's use the code from the the evoque_mq
benchmark that is included in the distribution:
First, make quoting="str" the domain-wide default, and
we can also set, on the domain's globals, the quoting function we want
to use:
Specify the actual quoting in the template itself,
template_mq.html:
Rendering is as usual:
Switching site template, dynamically, per user
The use of overlays, or template inheritance, where the base template is used to define the site layout could be a simple and effective way to manage a theme for a site. In addition, multiple site themes could be managed simply via multiple site base templates. Evoque supports dynamic overlays, or template inheritance -- and dynamic in two very handy ways:
-
First, by the possibility of using logically named templates that
allows assigning a name to any template, and then retrieving it by that name.
The dynamic part comes from the fact that we can at runtime change which
template is pointed to by a given name (by first doing a
template.unload()and then loading another template under the same name). - Second, by the fact that a template may specify which template it overlays by means of a parameter that will be evaluated for each rendering at runtime.
For this case, the second technique makes it really trivial to parametrize the base template per user. Here's the essence of an overlay page template that does this:
a dynamic page overlay: site_dyn_page_var.html
Where, on each rendering, which base template to use is determined
by the runtime value of my_site_theme.
Assume we have loaded the template with:
We can then specify a different value for my_site_theme
on each rendering:
Note that if you want to make sure you always have a fallback value
for my_site_theme
(for all renderings, all templates, and all users)
then just set it on the domain's globals:
example details
For completeness, let's fill in the full details for this simple example (that forms part of the Evoque test cases, and is included in the distribution).
a couple base templates as site themes
theme one: site_template_table.html
theme two: site_template_divs.html
a couple renderings of the same page overlay
using theme one:
gives the output:
using theme two:
gives the output:
Output the raw source of an evoque template
Just evoque() with raw=True and, if we would like to escape html source,
then also specify quoting="str". It does not matter how the evoque
template is defined, or what source format it is in. Here's an example,
let's first define a template template.html:
And we can evoque it as raw source from an application:
Or, from another template in the same collection:
Notes
Either of
get_template(raw=True).evoque() or
get_template().evoque(raw=True) is valid,
but there is a significant difference here:
stating get_template(raw=True) will
(when template is not yet loaded) load it and not compile it;
stating evoque(raw=True) will, after loading and compiling
the template if necessary, render the template's raw source.
Similarly, it is possible to do either of
get_template(quoting="str").evoque() or
get_template().evoque(quoting="str"): the first, used when
the template is loaded, sets the template's default
quoted string class to use, while the second specifies
which one to use for this evoque'ation.
Using the $evoque{} form, quoting="str" and
raw=True also have the implications that if the template is
not yet loaded then it will be loaded with unicode as its
default quoted string class, and as a raw template, i.e. it will
not be compiled and any subsequent attempts of evoque(raw=False)
will result in an error.
Any template may be rendered raw.
Sometimes you know beforehand that a template will be almost always rendered raw.
In those cases it is more practical to specify a $prefer{ raw=True }
directive once in the template itself, as opposed to specify same on each
evoque'ation of the template. Same suggestion applies for quoting.
Performance benchmarks
All performance benchmarks are included in the distribution -- please try out for yourself, the file bench/README.txt has some additional deatils for how to run each benchmark.
platform
The actual times are coming off a MacBook Pro with 2.4 GHz Intel Core 2 Duo, 2 GB of RAM, and Mac OS X 10.5, running python 2.5.1.
caution
Please remember that performance benchmarks are only relevant when considered within an entire context, and they may vary enormously between one environment and another. In addition, two different systems never do precisely the same thing, however simple and apparently identical the timed task may be.
general observations
Qpy is a templating system almost identical to straight python and offering convenient and fast (at least the C based version) html string building with automatic escaping. From the Qpy readme: Qpy provides a convenient mechanism for generating safely-quoted html text from python code. It does this by implementing a quoted-string data type and a modification of the python compiler.
Amongst currently popular full-featured templating systems for python: Mako is probably the fastest pure-python text-based system; Genshi is probably the feature-richest XML-based system.
The performance of the Evoque and Mako text-based systems seems to be more or less similar. Evoque seems to be faster for very simple templates, while Mako does better on loop-intensive templates -- probably because of additional work that Evoque has to do for the runtime evaluation of loop variables.
Suggestions to make these benchmarks more relevant are welcome, especially from anyone with particular knowledge of the specific templating system in question.
The numbers from these benchmarks are a tribute to python's conceptual integrity -- that rewards a straightforward implementation of a simple design with... amazing performance.
subs benchmark
For curiosity, a very simple performance look at doing only string substitutions -- thus for this one we can even compare against string.Template from the python standard library. We test a small template, mostly static text with just 6 variable substitutions.
For each run, time is averaged over 2000 renderings, and the best time of 4 runs is retained. All times are in ms. What you should want is automatic quoting and less time.
| quoting | template.String | qpy 1.6 | evoque 0.1.2 | mako 0.2.2 | genshi_text 0.5.1 |
|---|---|---|---|---|---|
| automatic | 0.05 | 0.05 | 0.15 | ||
| manual | 0.07 | 0.16 | 0.26 | ||
| none | 0.06 | 0.05 | 0.12 | ||
| restricted | 0.06 | 0.07 | |||
| naturally restricted |
template and data
The (automatically quoted) evoque template.
platform
The actual times are coming off a MacBook Pro with 2.4 GHz Intel Core 2 Duo, 2 GB of RAM, and Mac OS X 10.5, running python 2.5.1.
caution
Please remember that performance benchmarks are only relevant when considered within an entire context, and they may vary enormously between one environment and another. In addition, two different systems never do precisely the same thing, however simple and apparently identical the timed task may be.
basic benchmark
A basic benchmark, consisting of a small template that does a variety of standard templating tasks such as looping, calling nested and external sub-templates, escaping of data, etc. For each run, time is averaged over 2000 renderings, and the best time of 4 runs is retained. All times are in ms. What you should want is automatic quoting and less time.
| quoting | qpy 1.6 | evoque 0.1.2 | mako 0.2.2 | genshi 0.5.1 |
|---|---|---|---|---|
| automatic | 0.17 | 0.55 | 0.60 | 4.10 |
| manual | 0.53 | 0.60 | ||
| none | 0.43 | 0.43 | ||
| restricted | 0.66 | |||
| manual quoting implies no qpy | xml mode |
Inspiration for this benchmark is from the basic benchmark proposed by Genshi.
template and data
The (automatically quoted) evoque templates.
platform
The actual times are coming off a MacBook Pro with 2.4 GHz Intel Core 2 Duo, 2 GB of RAM, and Mac OS X 10.5, running python 2.5.1.
caution
Please remember that performance benchmarks are only relevant when considered within an entire context, and they may vary enormously between one environment and another. In addition, two different systems never do precisely the same thing, however simple and apparently identical the timed task may be.
bigtable benchmark
A simple brute force generation of a 10 columns x 1000 rows table, inspired from the bigtable benchmark proposed by Genshi. For each run, time is averaged over 10 renderings, and the best time of 4 runs is retained. All times are in ms. What you should want is automatic quoting and less time.
| quoting | qpy 1.6 | evoque 0.1.2 | mako 0.2.2 | genshi 0.5.1 |
|---|---|---|---|---|
| automatic | 46.35 | 119.59 | 42.91 | 605.94 |
| manual | 102.52 | 77.72 | ||
| none | 74.36 | 22.78 | ||
| none, tweaked | 9.81 | 9.53 | ||
| restricted | 147.23 | |||
| manual quoting implies no qpy | xml mode |
template and data
The (automatically quoted) evoque template.
platform
The actual times are coming off a MacBook Pro with 2.4 GHz Intel Core 2 Duo, 2 GB of RAM, and Mac OS X 10.5, running python 2.5.1.
caution
Please remember that performance benchmarks are only relevant when considered within an entire context, and they may vary enormously between one environment and another. In addition, two different systems never do precisely the same thing, however simple and apparently identical the timed task may be.
Frequently asked questions
Why yet another templating system?
Hopefully for precisely the same reason as every other one?
Seriously, you may wish to take a look at Evoque's
list of distinguishing features.
Why the evoque name?
It is a triple trip tributing all at once the great little eval
python built-in, the (optional) Qpy third-party package that
guarantees all data input into html templates is not only
always quoted but also quoted once and only once,
and the act of evoking a document from a template and
a bunch of data. Maybe they are more allusions than tributes,
possibly making all this seem a little cubist.
Don't call me pablo,
just that, everything is related.
The three are but one.
Within an h8 quoted template, how do I output escaped html?
By just declaring it unescaped, i.e. cast it to unicode.
E.g. if you want to output "<tag/>"
as part of the content in an html document, you can do
${unicode("<tag/>")}. Of course instead of
the literal "<tag/>" string you can unicode()
anything you like, even the evoque'ations of any other template.
Do I have to install Qpy?
No, but if you don't you will miss out on the killer feature of
automatic cross-site scripting protection.
In addition, you add to your risk of
Being Called a
Bozo When Producing XML.
But, if you still really want to, see the
no qpy howto.
Why can't I just do $variable instead of having to always specify ${variable}?
Evoque plays defensively. Anything immediately after the $ and before the
opening { is reserved for the name of directives, with ${ taken to imply
the start of an expression. This way, other directives could be added in
the future free of any backawards compatibility issues.
Is there something like a def directive?
Yes. For evoque a def is just a sub-template, defined
with a begin/end block.
Further nesting of sub-templates, i.e. sub of a sub etc, is
also supported. A sub-template defined anywhere may be addressed
from anywhere else.
How come there is no include directive?
There is, it is called evoque.
You can evoque any text file you like, or any other template or sub-template.
By default evoque'd files are evaluated as if they were templates -- to
just include raw source evoque with raw=True
(see the raw source howto).
How come there is no rawtext directive?
There is, it is called evoque, with parameter raw=True,
and if you want a different quoting you can also specify it e.g.
quoting="str".
You can evoque any template or sub-template,
e.g. any begin/end block,
or any textual file.
See the raw source howto.
What about template inheritance?
Sure, see the overlay directive.
Is it possible to import anything from within a template?
Allowing arbitrary python imports from directly within a template
will introduce a lot of difficult issues with restricted execution
mode. To support restricted mode,
a python application must retain some control on what may be added
to the evaluation context and it seems that the simplest way to
achieve this is to only allow the main application itself to affect
such additions, using the provided methods,
e.g. domain.set_namespace_on_globals().
See managing the namespace.
Why isn't there an XML-compliant syntax?
Because Evoque is a generic text templating system, not an XML-based one.
However, there is conceptually no problem with providing an XML-compliant
syntax, e.g using processing instructions, for directives -- it will only
need a modified template parser.
Expressions, being already XML-comliant, will require no change.
I do not understand the attribute descriptions I see on this site,
for example
filters:[callable] or file:either(None, str)?
Those descriptions mimic the syntax that the
qp.lib.spec module uses to actually
specify attribute allowed values and to perform the
corresponding runtime validation checks.
Here the clarity of the syntax is only used for documentation purposes.
The examples here mean that filters is allowed to be a
list of callables (taking some liberty here, as callable
is not really a python type), and file may be
either None or an instance of str.
If you would like to know more, here's a
little write-up on the wonderful spec module.
Or see the
spec.py source directly.
How do I specify the encoding of the output?
You don't. It is always unicode.
Once you get the output, you can encode it as you please.
Can Evoque do all that other template systems are able to do?
Most probably, yes. It would help to know the specific task, in which case
I'll be more than happy to help identify how it could be done with Evoque.
Integration with external systems
Descriptions, examples for using Evoque Templating with other systems.
django
Using Evoque Templating with
Django
is very simple. All that is needed is for a view
to return a Django HTTPResponse object of an
evoque'd template. The commented code recipe below shows
one way how to set this up and how to use it. You would need
to copy the evoque_django.py module into your
django project, and customize as necessary.
Why would you want to use Evoque with Django?
Besides being a full-featured template system, and a legitimate contender for the fastest pure-python text templating engine (see benchmark), it also offers some important features not offered by other engines, such as automatic input quoting and guaranteed XSS protection, restricted template execution mode to be able to expose your templates to untrusted editors, processing is always and only done in unicode, etc. For more, see features.
The code
evoque_django.py
views.py
Just an indication of what you would need to add to your
views.py:
In addition to the title kwarg we can of course
specify any others, as needed by the view. They will be transferred to the
template's locals evaluation dictionary.
It is important to remember that an Evoque template collection opens up
access to all files below the specified folder root. Thus you may choose to
organize the templates hierarchically e.g. we could rename and place some_template.html to some_function/template.html
(always relative to the collection's root folder).
We can then specify to use it in this way:
If for some reason we would wish to use a template that is not part of the default collection, we would just need to specify the collection e.g.
gz
Gizmo(QP) is an extension of the QP Web Framework. Work on sites made with Gizmo(QP) has been one of the primary motivator for the design choices that went into Evoque. Thus Gizmo(QP) (version >= 0.8) naturally supports Evoque.
Current examples of integration of Evoque with Gizmo(QP) are the online demos for Gizmo(QP).
pylons
Pylons encourages a good separation between the model, the controllers, and the views. It also is very flexible about using different template engines for different views. This note discusses a way for how to set and use Evoque Templating as the default template engine in a Pylons (minimum version 0.9.7) application.
Let's assume we create a site from scratch, let's call it evoque_site. When asked, enter the template_engine as follows:
adjust application files
We will then need to add the following codes to the files/functions indicated:
evoque_site.config.environment.load_environment(global_conf, app_conf)
development.ini
The above config load_environment() code may be
customized on a per-deployment basis, by specifying any of the
following parameters under the [app:main]
section of the the deployment's .ini file:
If evoque.default_dir is not set,
then Pylon's first templates folder is used.
The filters parameter (on Domain it only serves as site-wide default)
is not settable in the ini.
For a representative usage example of filters see the
Sub-templates in markdown howto.
logging.getLogger("evoque")
Just like any other logger used in a Pylons application, the
evoque logger may be adjusted via a deployment's
conf ini file.
evoque_site/lib/base.py
evoque_site/setup.py
add a controller and template
Let's first add and edit a simple controller and then add a template just to test:
evoque_site/controllers/hello.py
evoque_site/templates/template.html
Define a route for the hello.index() view, and load it
in your web browser.
You should see a simple page that dumps the template evaluation context.
qp
Integrating Evoque Templating with the
QP Web Framework
is pretty straightforward. As an explanation, here's the echo
demo site, included in the QP distribution, modified to use Evoque.
The changes are probably self-explanatory, but for the record
here's a commentary on each one:
- First of all
slash.qpyis renamed toslash.pyas the source code is now 100% python. - We import (first 2 import lines) what we need to be able to
set up an Evoque
domain. - The functions
qp.pub.common.headerandqp.pub.common.footerare no longer needed, so are not imported. Instead, we do importqp.pub.common.page, and the template used for each page rendering will take care of its own header and footer. - We override
Publisher.__init__()to be able to set up an Evoquedomainfor the site, via the separately specifiedset_domain()method.-
We designate the
evoque/sub-folder as root for our default template collection, but this can be anywhere on the file system. We may of course also specify other collections. -
We extend the evaluation global namespace as we need to,
here we only will need the
pformatutility. - We set a template under a name that we will use as default.
-
We designate the
- We override
Publisher.page()so that it knows how to get and evoque a template. Note that thepage()API remains identical -- except for the added optionaltemplatekeyword by which an export may select which template to use. If notemplatekeyword is specified onpage()then the default template name is used. - We migrate the presentation code from
SiteDirectory.index()to theitems.htmltemplate. We also tell theindex()method to use theitems.htmltemplate. - Just for the heck of it, we also make
items.htmlanoverlay, overriding only thecontentsub-template, on top ofbase.html-- that takes over the functionality of theheader()andfooter()QP functions while also offering additional flexibility, e.g. may have as many base templates as desired, or a base template can be more complex than just to handle a header and a footer.
The code - slash.py
The templates
base.html
items.html
werkzeug
Werkzeug is a collection of WSGI utility modules. The werkzeug web site presents (on welcome page click on nutshell to see the code) a very simple demo application, that is described as:
A tiny WSGI application (about 50 lines of code) that just uses werkzeug’s routing system and uses template names as endpoints. These templates are then loaded with Mako, rendered and returned as responses.
It is really trivial to make an equivalent application that uses
Evoque instead. The changes to the two concerned files
werkzeug_nutshell.py and
say_hello.html, shown further below,
are superficial:
- Initialize an Evoque
domain, thus implicitly also defining the default template collection. - Evoque'ing the template is identical, except for a different method name.
- In the sample
say_hello.htmltemplate, remove the unnecessaryhfilter (Evoque automatically quotes all incoming unquoted data). - Make the sample
say_hello.htmltemplate a valid HTML document ;-)
werkzeug_nutshell.py
say_hello.html
Under consideration
For generation of XML documents, add filters for validating the generated output for both document (page) and fragment levels. Should be easy to turn on and off, e.g. for development or for deployment.
Cache the compiled template on disk, for the benefit of faster startup times, or usage in a cgi environment.
Introspection into the collections and their respective caches of loaded templates.
Consider supporting a more convenient way to specify filters on expressions,
e.g. ${ expr | url, trim, etc } would do the same
thing as the current ${etc(trim(url(expr)))}.
Improve ability to "extract" pieces of
text from files as the source of a template,
without needing that those files be touched in any way.
Criteria could be pairs of regular expressions, line numbers,
format-specific parsing directives, combination of criteria, etc.
Could be implemented as an
extension to the evoque() built-in callable,
or as a new callable such as extract().
Support also for python 3.0, maybe from the same Evoque python 2.5 code base.
Evoque copyright and license
Licensed under the Academic Free License version 3.0
Download and support
Single-page documentation
For printing convenience, this entire site may also be evoque'd as a single page:
Typo reports, suggestions, observations are appreciated!
Download Evoque version 0.3 (2008-09-06)
You may wish to take a look at the change log for this and previous releases.
It is also highly recommended to download and install the Qpy unicode templating utility that provides the qpy.h8 quoting class for automatic input escaping. You may prefer to use a command line client such as wget to download these, as indicated below.
Installation
To install, you may either use easy_install or
standard python distutils
(i.e. download, unpack, install) as detailed below:
Acknowledgements
Thank you to MEMS and Nanotechnology Exchange for the great software they have made available to the python community -- the Qpy package that is used by Evoque is one of these packages. In particular, a thank you to David Binger for his intense and infectious appreciation of simplicity, and therefore complexity.
Thank you to Skip Montanaro and to Martin v. Löwis for their help with the decodeh module, now part of Evoque.
The physical source lines of code (SLOC) count is generated using SLOCCount by David A. Wheeler.
Steven Degraeve for the heading area background banner on the pages of this site.
The multitude of templating systems for python over the past years have contributed to the clarification of the ideas and priorities for Evoque. It is impossible to identify which and what -- a big thank you to the python community in general for the openness and sharing of innumerable ideas and disinterested hours of work.
Thank you to Alex Martelli for his help and collaboration on the development of XYAPTU: Lightweight XML/HTML Document Template Engine for Python. This single-module templating system, written back in 2002, is the beginning of the ideas that have evolved over the years since then and that are now manifest in Evoque.