Diffbook easy? diffbook

-t, --taskbar use web2py gui and run in taskbar (system tray)

--nogui text-only, no GUI

-A ARGS, --args=ARGS should be followed by a list of arguments to be passed

to script, to be used with -S, -A must be the last

option

--no-banner Do not print header banner

--interfaces=INTERFACES

listen on multiple addresses: "ip1:port1:key1:cert1:ca

_cert1;ip2:port2:key2:cert2:ca_cert2;..."

(:key:cert:ca_cert optional; no spaces; IPv6 addresses

must be in square [] brackets)

--run_system_tests runs web2py tests

``:code

------

------

-t, --taskbar use web2py gui and run in taskbar (system tray)

--nogui text-only, no GUI

-A ARGS, --args=ARGS should be followed by a list of arguments to be passed

to script, to be used with -S, -A must be the last

option

--no-banner Do not print header banner

--interfaces=INTERFACES

listen on multiple addresses: "ip1:port1:key1:cert1:ca

_cert1;ip2:port2:key2:cert2:ca_cert2;..."

(:key:cert:ca_cert optional; no spaces; IPv6 addresses

must be in square [] brackets)

--run_system_tests runs web2py tests

``:code

------

------

### Command line options

### Command line options

``

python web2py.py -vT welcome

``:code

### Workflow

``

python web2py.py -vT welcome

``:code

### Workflow

The stable API consists of these functions:

- disable()

- resume()

- terminate()

- kill()

- queue_task(),

- task_status()

- stop_task()

The web2py scheduler works very much like the task queue described in the previous sub-section with some differences:

- It provides a standard mechanism for creating, scheduling, and monitoring tasks.

- There is not a single background process but a set of workers processes.

- The job of worker nodes can be monitored because their state, as well as the state of the tasks, is stored in the database.

- It works without web2py but that is not documented here.

The scheduler does not use cron, although one can use cron @reboot to start the worker nodes.

More information about deploying the scheduler under Linux and Windows is in the Deployment recipes chapter.

In the scheduler, a task is simply a function defined in a model (or in a module and imported by a model). For example:

Also, tasks can access another environmental variable that is not present in nor

Remember to call ``db.commit()`` at the end of every task if it involves inserts/updates to the database. web2py commits by default at the end of a successful action but the scheduler tasks are not actions.

------

To enable the scheduler you must instantiate the Scheduler class in a model.

The recommended way to enable the scheduler to your app is to create a model file named ``scheduler.py`` and define your function there. After the functions, you can put the following code into the model:

``

from gluon.scheduler import Scheduler

scheduler = Scheduler(db)

``:code

If your tasks are defined in a module (as opposed to a model) you may have to restart the workers.

The task is scheduled with

``

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

The first argument of the ``Scheduler`` class must be the database to be used by the scheduler to communicate with the workers. This can be the ``db`` of the app or another dedicated ``db``, perhaps one shared by multiple apps. If you use SQLite it's recommended to use a separate db from the one used by your app in order to keep the app responsive.

Once the tasks are defined and the ``Scheduler`` is instantiated, all that is needed to do is to start the workers. You can do that in several ways:

``

python web2py.py -K myapp

``

starts a worker for the app ``myapp``. If you want start multiple workers for the same app, you can do so just passing ``myapp,myapp``. You can pass also the ``group_names`` (overriding the one set in your model) with

``

python web2py.py -K myapp:group1:group2,myotherapp:group1

``

If you have a model called ``scheduler.py`` you can start/stop the workers from web2py's default window (the one you use to set the ip address and the port).

One last nice addition: if you use the embedded webserver, you can start the webserver and the scheduler with just one line of code (this assumes you don't want the web2py window popping up, else you can use the "Schedulers" menu instead)

``

python web2py.py -a yourpass -K myapp -X

``

You can pass the usual parameters (-i, -p, here -a prevents the window from showing up), pass whatever app in the -K parameter and append a -X. The scheduler will run alongside the webserver!

Windows users looking to create a service should see the Deployment Recipes chapter.

Scheduler's complete signature is:

``

Scheduler(

db,

tasks=None,

migrate=True,

worker_name=None,

group_names=None,

heartbeat=HEARTBEAT,

max_empty_runs=0,

discard_results=False,

utc_time=False

)

``:code

Let's see them in order:

- ``db`` is the database DAL instance where you want the scheduler tables be placed.

- ``tasks`` is a dictionary that maps task names into functions. If you do not pass this parameter, function will be searched in the app environment.

Let's see them in order:

------

NB: This is useful if you have different workers instances (e.g. on different machines) and you want to assign tasks to a specific worker.

NB2: It's possible to assign a worker more groups, and they can be also all the same, as ``['mygroup','mygroup']``. Tasks will be distributed taking into consideration that a worker with group_names ``['mygroup','mygroup']`` is able to process the double of the tasks a worker with group_names ``['mygroup']`` is.

------

- ``heartbeat`` is by default set to 3 seconds. This parameter is the one controlling how often a scheduler will check its status on the ``scheduler_worker`` table and see if there are any **ASSIGNED** tasks to itself to process.

- ``max_empty_runs`` is 0 by default, that means that the worker will continue to process tasks as soon as they are **ASSIGNED**. If you set this to a value of, let's say, 10, a worker will die automatically if it's **ACTIVE** and no tasks are **ASSIGNED** to it for 10 loops. A loop is when a worker searches for tasks, every 3 seconds (or the set ``heartbeat``)

- ``discard_results`` is False by default. If set to True, no scheduler_run records will be created.

------

NB: scheduler_run records will be created as before for **FAILED**, **TIMEOUT** and **STOPPED** tasks's statuses.

------

- ``utc_time`` is False by default. If you need to coordinate with workers living in different timezones, or don't have problems with solar/DST times, supplying datetimes from different countries, etc, you can set this to True. The scheduler will honor the UTC time and work leaving the local time aside. Caveat: you need to schedule tasks with UTC times (for start_time, stop_time, and so on.)

Now we have the infrastructure in place: defined the tasks, told the scheduler about them, started the worker(s). What remains is to actually schedule the tasks

Tasks can be scheduled programmatically or via appadmin. In fact, a task is scheduled simply by adding an entry in the table "scheduler_task", which you can access via appadmin:

``

http://127.0.0.1:8000/myapp/appadmin/insert/db/scheduler_task

``

The meaning of the fields in this table is obvious. The "args" and "vars"" fields are the values to be passed to the task in JSON format. In the case of the "task_add" above, an example of "args" and "vars" could be:

``

args = [3, 4]

vars = {}

``:code

or

``

args = []

vars = {'a':3, 'b':4}

``:code

The ``scheduler_task`` table is the one where tasks are organized.

To add tasks via the API, use

``

scheduler.queue_task('mytask',...)

``

which is documented [[below #queue_task_sig]] .

All tasks follow a lifecycle

[[scheduler tasks @///image/ce8edcc3.png center]]

By default, when you send a task to the scheduler,

it is in the **QUEUED** status.

If you need it to be executed later, use the ``start_time`` parameter (default = now).

If for some reason you need to be sure that the task does not

get executed after a certain point in time (maybe a request to a web service

that shuts down at 1AM, a mail that needs to be sent not after the working hours, etc...) you can set a ``stop_time`` (default = None) for it.

If your task is NOT picked up by a worker before ``stop_time``, it will be set as **EXPIRED**.

Tasks with no ``stop_time`` set or picked up **BEFORE** stop_time are **ASSIGNED** to a worker. When a workers picks up a task, its status is set to **RUNNING**.

**RUNNING** tasks may end up:

- **TIMEOUT** when more than ``n`` seconds passed with ``timeout`` parameter (default = 60 seconds).

- **FAILED** when an exception is detected,

- **COMPLETED** when they successfully complete.

Values for ``start_time`` and ``stop_time`` should be datetime objects. To schedule "mytask" to run at 30 seconds from the current time, for example, you would do the following:

scheduler.queue_task('mytask',

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

------

Default behavior: The time period is not calculated between the END of the first round and the START of the next, but from the START time of the first round to the START time of the next cycle).

This can cause accumulating 'drift' in the start time of a job.

After v 2.8.2, a new parameter ``prevent_drift`` was added, defaulting to False. If set to True when queing a task, the start_time parameter will take precedence over the period, preventing drift.

------

You can also set how many times the function can raise an exception (i.e. requesting data from a slow web service) and be queued again instead of stopping in **FAILED** status using the parameter ``retry_failed`` (default = 0, -1 = unlimited).

[[task repeats @///image/7d8b85e4.png center]]

Summary: you have

- ``period`` and ``repeats`` to get an automatically rescheduled function

- ``timeout`` to be sure that a function doesn't exceed a certain amount of time

- ``retry_failed`` to control how many times the task can "fail"

- ``start_time`` and ``stop_time`` to schedule a function in a restricted timeframe

The method:

``

scheduler.queue_task(

function,

pargs=[],

pvars={},

start_time=now, #datetime

stop_time = None, #datetime

timeout = 60, #seconds

prevent_drift=False,

period=60, #seconds

immediate=False,

repeats = 1

)

``:code

allows you to queue tasks to be executed by workers.

It returns a row (see [[here #queue_task_return]]), and it takes the following parameters:

- ``function`` (required): It can be a task name or a reference to an actual function.

scheduler = Scheduler(db, tasks=dict(demo1=task_add))

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

A call to ``scheduler.queue_task`` returns the task ``id`` and ``uuid`` of the task you queued (can be the one you passed or the auto-generated one), and possible ``errors``: [[queue_task_return]]

``

<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>

``

If there are errors (usually syntax error or input validation errors),

you get the result of the validation, and id and uuid will be None

``

<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>

``

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

The scheduler also creates one more table called "scheduler_worker", which stores the workers' heartbeat and their status.

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

When you start a worker, you may later want to:

- kill it "no matter what it's doing"

- kill it only if it is not processing tasks

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

To be more precise, for started workers you can change the ``status`` value of any worker to influence

its behavior.

As for tasks, workers can be in one of the following statuses: ACTIVE, DISABLED, TERMINATE or KILLED.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

[[workers statuses @///image/bd891eed.png center]]

scheduler.queue_task(

pargs=[],

pvars={'a':3,'b':4},

repeats = 10, # run 10 times

period = 3600, # every 1h

timeout = 120, # should take less than 120 seconds

)

``:code

Notice that fields "times_run", "last_run_time" and "assigned_worker_name" are not provided at schedule time but are filled automatically by the workers.

You can also retrieve the output of completed tasks:

``

completed_runs = db(db.scheduler_run.run_status='COMPLETED').select()

``:code

------

The scheduler is considered experimental because it needs more extensive testing and because the table structure may change as more features are added.

------

A special "word" encountered in the print statements of your functions clear all

the previous output. That word is ``!clear!``.

This, coupled with the ``sync_output`` parameter, allows to report percentages.

Here is an example:

``

def reporting_percentages():

time.sleep(5)

print '50%'

time.sleep(5)

print '!clear!100%'

return 1

``

The function ``reporting_percentages`` sleeps for 5 seconds, outputs ``50%``.

Then, it sleeps other 5 seconds and outputs ``100%``. Note that the output in the scheduler_run table is synced every 2 seconds and that the second print statement that contains ``!clear!100%`` gets the ``50%`` output cleared and replaced by ``100%`` only.

``

scheduler.queue_task(reporting_percentages,

sync_output=2)

``:code

### Third party modules

``import``:inxx

The stable API consists of these functions:

- disable()

- resume()

- terminate()

- kill()

- queue_task(),

- task_status()

- stop_task()

The web2py scheduler works very much like the task queue described in the previous sub-section with some differences:

- It provides a standard mechanism for creating, scheduling, and monitoring tasks.

- There is not a single background process but a set of workers processes.

- The job of worker nodes can be monitored because their state, as well as the state of the tasks, is stored in the database.

- It works without web2py but that is not documented here.

The scheduler does not use cron, although one can use cron @reboot to start the worker nodes.

More information about deploying the scheduler under Linux and Windows is in the Deployment recipes chapter.

In the scheduler, a task is simply a function defined in a model (or in a module and imported by a model). For example:

Also, tasks can access another environmental variable that is not present in nor

Remember to call ``db.commit()`` at the end of every task if it involves inserts/updates to the database. web2py commits by default at the end of a successful action but the scheduler tasks are not actions.

------

To enable the scheduler you must instantiate the Scheduler class in a model.

The recommended way to enable the scheduler to your app is to create a model file named ``scheduler.py`` and define your function there. After the functions, you can put the following code into the model:

``

from gluon.scheduler import Scheduler

scheduler = Scheduler(db)

``:code

If your tasks are defined in a module (as opposed to a model) you may have to restart the workers.

The task is scheduled with

``

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

The first argument of the ``Scheduler`` class must be the database to be used by the scheduler to communicate with the workers. This can be the ``db`` of the app or another dedicated ``db``, perhaps one shared by multiple apps. If you use SQLite it's recommended to use a separate db from the one used by your app in order to keep the app responsive.

Once the tasks are defined and the ``Scheduler`` is instantiated, all that is needed to do is to start the workers. You can do that in several ways:

``

python web2py.py -K myapp

``

starts a worker for the app ``myapp``. If you want start multiple workers for the same app, you can do so just passing ``myapp,myapp``. You can pass also the ``group_names`` (overriding the one set in your model) with

``

python web2py.py -K myapp:group1:group2,myotherapp:group1

``

If you have a model called ``scheduler.py`` you can start/stop the workers from web2py's default window (the one you use to set the ip address and the port).

One last nice addition: if you use the embedded webserver, you can start the webserver and the scheduler with just one line of code (this assumes you don't want the web2py window popping up, else you can use the "Schedulers" menu instead)

``

python web2py.py -a yourpass -K myapp -X

``

You can pass the usual parameters (-i, -p, here -a prevents the window from showing up), pass whatever app in the -K parameter and append a -X. The scheduler will run alongside the webserver!

Windows users looking to create a service should see the Deployment Recipes chapter.

Scheduler's complete signature is:

``

Scheduler(

db,

tasks=None,

migrate=True,

worker_name=None,

group_names=None,

heartbeat=HEARTBEAT,

max_empty_runs=0,

discard_results=False,

utc_time=False

)

``:code

Let's see them in order:

- ``db`` is the database DAL instance where you want the scheduler tables be placed.

- ``tasks`` is a dictionary that maps task names into functions. If you do not pass this parameter, function will be searched in the app environment.

Let's see them in order:

------

NB: This is useful if you have different workers instances (e.g. on different machines) and you want to assign tasks to a specific worker.

NB2: It's possible to assign a worker more groups, and they can be also all the same, as ``['mygroup','mygroup']``. Tasks will be distributed taking into consideration that a worker with group_names ``['mygroup','mygroup']`` is able to process the double of the tasks a worker with group_names ``['mygroup']`` is.

------

- ``heartbeat`` is by default set to 3 seconds. This parameter is the one controlling how often a scheduler will check its status on the ``scheduler_worker`` table and see if there are any **ASSIGNED** tasks to itself to process.

- ``max_empty_runs`` is 0 by default, that means that the worker will continue to process tasks as soon as they are **ASSIGNED**. If you set this to a value of, let's say, 10, a worker will die automatically if it's **ACTIVE** and no tasks are **ASSIGNED** to it for 10 loops. A loop is when a worker searches for tasks, every 3 seconds (or the set ``heartbeat``)

- ``discard_results`` is False by default. If set to True, no scheduler_run records will be created.

------

NB: scheduler_run records will be created as before for **FAILED**, **TIMEOUT** and **STOPPED** tasks's statuses.

------

- ``utc_time`` is False by default. If you need to coordinate with workers living in different timezones, or don't have problems with solar/DST times, supplying datetimes from different countries, etc, you can set this to True. The scheduler will honor the UTC time and work leaving the local time aside. Caveat: you need to schedule tasks with UTC times (for start_time, stop_time, and so on.)

Now we have the infrastructure in place: defined the tasks, told the scheduler about them, started the worker(s). What remains is to actually schedule the tasks

Tasks can be scheduled programmatically or via appadmin. In fact, a task is scheduled simply by adding an entry in the table "scheduler_task", which you can access via appadmin:

``

http://127.0.0.1:8000/myapp/appadmin/insert/db/scheduler_task

``

The meaning of the fields in this table is obvious. The "args" and "vars"" fields are the values to be passed to the task in JSON format. In the case of the "task_add" above, an example of "args" and "vars" could be:

``

args = [3, 4]

vars = {}

``:code

or

``

args = []

vars = {'a':3, 'b':4}

``:code

The ``scheduler_task`` table is the one where tasks are organized.

To add tasks via the API, use

``

scheduler.queue_task('mytask',...)

``

which is documented [[below #queue_task_sig]] .

All tasks follow a lifecycle

[[scheduler tasks @///image/ce8edcc3.png center]]

By default, when you send a task to the scheduler,

it is in the **QUEUED** status.

If you need it to be executed later, use the ``start_time`` parameter (default = now).

If for some reason you need to be sure that the task does not

get executed after a certain point in time (maybe a request to a web service

that shuts down at 1AM, a mail that needs to be sent not after the working hours, etc...) you can set a ``stop_time`` (default = None) for it.

If your task is NOT picked up by a worker before ``stop_time``, it will be set as **EXPIRED**.

Tasks with no ``stop_time`` set or picked up **BEFORE** stop_time are **ASSIGNED** to a worker. When a workers picks up a task, its status is set to **RUNNING**.

**RUNNING** tasks may end up:

- **TIMEOUT** when more than ``n`` seconds passed with ``timeout`` parameter (default = 60 seconds).

- **FAILED** when an exception is detected,

- **COMPLETED** when they successfully complete.

Values for ``start_time`` and ``stop_time`` should be datetime objects. To schedule "mytask" to run at 30 seconds from the current time, for example, you would do the following:

scheduler.queue_task('mytask',

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

------

Default behavior: The time period is not calculated between the END of the first round and the START of the next, but from the START time of the first round to the START time of the next cycle).

This can cause accumulating 'drift' in the start time of a job.

After v 2.8.2, a new parameter ``prevent_drift`` was added, defaulting to False. If set to True when queing a task, the start_time parameter will take precedence over the period, preventing drift.

------

You can also set how many times the function can raise an exception (i.e. requesting data from a slow web service) and be queued again instead of stopping in **FAILED** status using the parameter ``retry_failed`` (default = 0, -1 = unlimited).

[[task repeats @///image/7d8b85e4.png center]]

Summary: you have

- ``period`` and ``repeats`` to get an automatically rescheduled function

- ``timeout`` to be sure that a function doesn't exceed a certain amount of time

- ``retry_failed`` to control how many times the task can "fail"

- ``start_time`` and ``stop_time`` to schedule a function in a restricted timeframe

The method:

``

scheduler.queue_task(

function,

pargs=[],

pvars={},

start_time=now, #datetime

stop_time = None, #datetime

timeout = 60, #seconds

prevent_drift=False,

period=60, #seconds

immediate=False,

repeats = 1

)

``:code

allows you to queue tasks to be executed by workers.

It returns a row (see [[here #queue_task_return]]), and it takes the following parameters:

- ``function`` (required): It can be a task name or a reference to an actual function.

scheduler = Scheduler(db, tasks=dict(demo1=task_add))

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

A call to ``scheduler.queue_task`` returns the task ``id`` and ``uuid`` of the task you queued (can be the one you passed or the auto-generated one), and possible ``errors``: [[queue_task_return]]

``

<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>

``

If there are errors (usually syntax error or input validation errors),

you get the result of the validation, and id and uuid will be None

``

<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>

``

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

The scheduler also creates one more table called "scheduler_worker", which stores the workers' heartbeat and their status.

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

When you start a worker, you may later want to:

- kill it "no matter what it's doing"

- kill it only if it is not processing tasks

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

To be more precise, for started workers you can change the ``status`` value of any worker to influence

its behavior.

As for tasks, workers can be in one of the following statuses: ACTIVE, DISABLED, TERMINATE or KILLED.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

[[workers statuses @///image/bd891eed.png center]]

scheduler.queue_task(

pargs=[],

pvars={'a':3,'b':4},

repeats = 10, # run 10 times

period = 3600, # every 1h

timeout = 120, # should take less than 120 seconds

)

``:code

Notice that fields "times_run", "last_run_time" and "assigned_worker_name" are not provided at schedule time but are filled automatically by the workers.

You can also retrieve the output of completed tasks:

``

completed_runs = db(db.scheduler_run.run_status='COMPLETED').select()

``:code

------

The scheduler is considered experimental because it needs more extensive testing and because the table structure may change as more features are added.

------

A special "word" encountered in the print statements of your functions clear all

the previous output. That word is ``!clear!``.

This, coupled with the ``sync_output`` parameter, allows to report percentages.

Here is an example:

``

def reporting_percentages():

time.sleep(5)

print '50%'

time.sleep(5)

print '!clear!100%'

return 1

``

The function ``reporting_percentages`` sleeps for 5 seconds, outputs ``50%``.

Then, it sleeps other 5 seconds and outputs ``100%``. Note that the output in the scheduler_run table is synced every 2 seconds and that the second print statement that contains ``!clear!100%`` gets the ``50%`` output cleared and replaced by ``100%`` only.

``

scheduler.queue_task(reporting_percentages,

sync_output=2)

``:code

### Third party modules

``import``:inxx

-------

Beware! Given ``from gluon import current``, it is correct to use ``current.request`` and any of the other thread local objects but one should never assign them to global variables in the module, such as in

``

request = current.request # WRONG! DANGER!

``

``

class MyClass:

request = current.request # WRONG! DANGER!

``

This is because the thread local object must be extracted at runtime. Global variables instead are defined only once when the model is imported for the first time.

-------

-------

Beware! Given ``from gluon import current``, it is correct to use ``current.request`` and any of the other thread local objects but one should never assign them to global variables in the module, such as in

``

request = current.request # WRONG! DANGER!

``

``

class MyClass:

request = current.request # WRONG! DANGER!

``

This is because the thread local object must be extracted at runtime. Global variables instead are defined only once when the model is imported for the first time.

-------

There are some caveats to keep in mind:

- Models in the same folder/subfolder are executed in alphabetical order.

- Any variable defined in a model will be visible to other models following alphabetically, to the controllers, and to the views.

``models_to_run``:inxx

- Models in subfolders are executed conditionally. For example, if the user has requested "/a/c/f" where "a" is the application, "c" is the controller, and "f" is the function (action), then the following models are executed:

``

applications/a/models/*.py

applications/a/models/c/*.py

applications/a/models/c/f/*.py

``

This behaviour is enforced by default. Altering the ``response.models_to_run`` regex list, you can force the behaviour you want. Look at [[response #response_models_to_run]] for additional details

- The requested controller is executed and the requested function is called. This means all top-level code in the controller is also executed at every request for that controller.

- The view is only called if the action returns a dictionary.

- If a view is not found, web2py tries to use a generic view. By default, generic views are disabled, although the 'welcome' app includes a line in /models/db.py to enable them on localhost only. They can be enabled per extension type and per action (using ``response.generic_patterns``). In general, generic views are a development tool and typically should not be used in production. If you want some actions to use a generic view, list those actions in ``response.generic_patterns`` (discussed in more detail in the chapter on Services).

The possible behaviors of an action are the following:

**Return a string**

``

def index(): return 'data'

``

For backward compatibility ``SQLDB=DAL`` and ``SQLField=Field``. We encourage yo

Other objects and modules are defined in the libraries, but they are not automatically imported since they are not used as often.

The core API entities in the web2py execution environment are ``request``, ``response``, ``session``, ``cache``, ``URL``, ``HTTP``, ``redirect`` and ``T`` and are discussed below.

A few objects and functions, including **Auth**, **Crud** and **Service**, are defined in "gluon/tools.py" and they need to be imported as necessary:

``

from gluon.tools import Auth, Crud, Service

``:code

They are imported in ``db.py`` in the scaffolding application.

#### Accessing the API from Python modules

Your models or controller may import python modules, and these may need to use some of the web2py API. The way to do it is by importing them:

``

from gluon import *

``

In fact, any Python module, even if not imported by a web2py application, can import the web2py API as long as web2py is in the ``sys.path``.

Create a module "/myapp/modules/test.py" that contains:

``

from gluon import *

def ip(): return current.request.client

``

Now from a controller in "myapp" you can do

``

import test

def index():

return "Your ip is " + test.ip()

``

Notice a few things:

- ``import test`` looks for the module first in the current app's modules folder, then in the folders listed in ``sys.path``. Therefore, app-level modules always take precedence over Python modules. This allows different apps to ship with different versions of their modules, without conflicts.

- Different users can call the same action ``index`` concurrently, which calls the function in the module, and yet there is no conflict because ``current.request`` is a different object in different threads. Just be careful not to access ``current.request`` outside of functions or classes (i.e., at the top level) in the module.

- ``import test`` is a shortcut for ``from applications.appname.modules import test``. Using the longer syntax, it is possible to import modules from other applications.

Modules that import ``current`` can access:

- ``current.session``

- ``current.cache``

- ``current.T``

and any other variable your application chooses to store in current. For example a model could do

``

auth = Auth(db)

from gluon import current

current.auth = auth

``

and now all modules imported can access ``current.auth``.

``current`` and ``import`` create a powerful mechanism to build extensible and reusable modules for your applications.

-------

Beware! Given ``from gluon import current``, it is correct to use ``current.request`` and any of the other thread local objects but one should never assign them to global variables in the module, such as in

``

request = current.request # WRONG! DANGER!

``

``

class MyClass:

request = current.request # WRONG! DANGER!

``

This is because the thread local object must be extracted at runtime. Global variables instead are defined only once when the model is imported for the first time.

-------

Another caveat has to do with cache. You cannot use the ``cache`` object to decorate functions in modules, that is because it would not behave as expected. In order to cache a function ``f`` in a module you must use ``lazy_cache``:

``

from gluon.cache import lazy_cache

@lazy_cache('key', time_expire=60, cache_model='ram')

def f(a,b,c,): ....

``:code

Mind that the key is user defined but must be uniquely associated to the function. If omitted web2py will automatically determine a key.

### ``request``

``request``:inxx ``Storage``:inxx ``request.cookies``:inxx ``user_agent``:inxx

T("hello world ## first occurrence")

T("hello world ## second occurrence")

``:code

The text following the ``##``, including the double ``##``, are comments.

#### Pluralization engine

Since version 2.0, web2py includes a powerful pluralization system (PS). This means that when text marked for translation depends on a numeric variable, it may be translated differently based on the numeric value. For example in English we may render:

``

x book(s)

``

with

``

a book (x==1)

5 books (x==5)

``

There are some caveats to keep in mind:

- Models in the same folder/subfolder are executed in alphabetical order.

- Any variable defined in a model will be visible to other models following alphabetically, to the controllers, and to the views.

``models_to_run``:inxx

- Models in subfolders are executed conditionally. For example, if the user has requested "/a/c/f" where "a" is the application, "c" is the controller, and "f" is the function (action), then the following models are executed:

``

applications/a/models/*.py

applications/a/models/c/*.py

applications/a/models/c/f/*.py

``

This behaviour is enforced by default. Altering the ``response.models_to_run`` regex list, you can force the behaviour you want. Look at [[response #response_models_to_run]] for additional details

- The requested controller is executed and the requested function is called. This means all top-level code in the controller is also executed at every request for that controller.

- The view is only called if the action returns a dictionary.

- If a view is not found, web2py tries to use a generic view. By default, generic views are disabled, although the 'welcome' app includes a line in /models/db.py to enable them on localhost only. They can be enabled per extension type and per action (using ``response.generic_patterns``). In general, generic views are a development tool and typically should not be used in production. If you want some actions to use a generic view, list those actions in ``response.generic_patterns`` (discussed in more detail in the chapter on Services).

The possible behaviors of an action are the following:

**Return a string**

``

def index(): return 'data'

``

For backward compatibility ``SQLDB=DAL`` and ``SQLField=Field``. We encourage yo

Other objects and modules are defined in the libraries, but they are not automatically imported since they are not used as often.

The core API entities in the web2py execution environment are ``request``, ``response``, ``session``, ``cache``, ``URL``, ``HTTP``, ``redirect`` and ``T`` and are discussed below.

A few objects and functions, including **Auth**, **Crud** and **Service**, are defined in "gluon/tools.py" and they need to be imported as necessary:

``

from gluon.tools import Auth, Crud, Service

``:code

They are imported in ``db.py`` in the scaffolding application.

#### Accessing the API from Python modules

Your models or controller may import python modules, and these may need to use some of the web2py API. The way to do it is by importing them:

``

from gluon import *

``

In fact, any Python module, even if not imported by a web2py application, can import the web2py API as long as web2py is in the ``sys.path``.

Create a module "/myapp/modules/test.py" that contains:

``

from gluon import *

def ip(): return current.request.client

``

Now from a controller in "myapp" you can do

``

import test

def index():

return "Your ip is " + test.ip()

``

Notice a few things:

- ``import test`` looks for the module first in the current app's modules folder, then in the folders listed in ``sys.path``. Therefore, app-level modules always take precedence over Python modules. This allows different apps to ship with different versions of their modules, without conflicts.

- Different users can call the same action ``index`` concurrently, which calls the function in the module, and yet there is no conflict because ``current.request`` is a different object in different threads. Just be careful not to access ``current.request`` outside of functions or classes (i.e., at the top level) in the module.

- ``import test`` is a shortcut for ``from applications.appname.modules import test``. Using the longer syntax, it is possible to import modules from other applications.

Modules that import ``current`` can access:

- ``current.session``

- ``current.cache``

- ``current.T``

and any other variable your application chooses to store in current. For example a model could do

``

auth = Auth(db)

from gluon import current

current.auth = auth

``

and now all modules imported can access ``current.auth``.

``current`` and ``import`` create a powerful mechanism to build extensible and reusable modules for your applications.

-------

Beware! Given ``from gluon import current``, it is correct to use ``current.request`` and any of the other thread local objects but one should never assign them to global variables in the module, such as in

``

request = current.request # WRONG! DANGER!

``

``

class MyClass:

request = current.request # WRONG! DANGER!

``

This is because the thread local object must be extracted at runtime. Global variables instead are defined only once when the model is imported for the first time.

-------

Another caveat has to do with cache. You cannot use the ``cache`` object to decorate functions in modules, that is because it would not behave as expected. In order to cache a function ``f`` in a module you must use ``lazy_cache``:

``

from gluon.cache import lazy_cache

@lazy_cache('key', time_expire=60, cache_model='ram')

def f(a,b,c,): ....

``:code

Mind that the key is user defined but must be uniquely associated to the function. If omitted web2py will automatically determine a key.

### ``request``

``request``:inxx ``Storage``:inxx ``request.cookies``:inxx ``user_agent``:inxx

T("hello world ## first occurrence")

T("hello world ## second occurrence")

``:code

The text following the ``##``, including the double ``##``, are comments.

#### Pluralization engine

Since version 2.0, web2py includes a powerful pluralization system (PS). This means that when text marked for translation depends on a numeric variable, it may be translated differently based on the numeric value. For example in English we may render:

``

x book(s)

``

with

``

a book (x==1)

5 books (x==5)

``

-C, --cron trigger a cron run manually; usually invoked from a

system crontab

--softcron triggers the use of softcron

-Y, --run-cron start the background cron process

-J, --cronjob identify cron-initiated command

-L CONFIG, --config=CONFIG

config file

-F PROFILER_FILENAME, --profiler=PROFILER_FILENAME

profiler filename

-t, --taskbar use web2py gui and run in taskbar (system tray)

--nogui text-only, no GUI

-A ARGS, --args=ARGS should be followed by a list of arguments to be passed

to script, to be used with -S, -A must be the last

option

--no-banner Do not print header banner

--interfaces=INTERFACES

listen on multiple addresses: "ip1:port1:key1:cert1:ca

_cert1;ip2:port2:key2:cert2:ca_cert2;..."

(:key:cert:ca_cert optional; no spaces; IPv6 addresses

must be in square [] brackets)

--run_system_tests runs web2py tests

``:code

Lower-case options are used to configure the web server. The ``-L`` option tells web2py to read configuration options from a file, ``-W`` installs web2py as a windows service, while ``-S``, ``-P`` and ``-M`` options start an interactive Python shell. The ``-T`` option finds and runs controller doctests in a web2py execution environment. For example, the following example runs doctests from all controllers in the "welcome" application:

``

python web2py.py -vT welcome

``:code

if you run web2py as a Windows Service, ``-W``, it is not convenient to pass the configuration using command line arguments. For this reason, in the web2py folder there is a sample "options_std.py" configuration file for the internal web server:

``

import socket

import os

ip = '0.0.0.0'

port = 80

interfaces = [('0.0.0.0', 80)]

#,('0.0.0.0',443,'ssl_private_key.pem','ssl_certificate.pem')]

password = '<recycle>' # ## <recycle> means use the previous password

pid_filename = 'httpserver.pid'

log_filename = 'httpserver.log'

profiler_filename = None

ssl_certificate = None # 'ssl_certificate.pem' # ## path to certificate file

def cache_controller_on_disk():

t = time.ctime()

return dict(time=t, link=A('click to reload',

_href=request.url))

``:code

The dictionary returned by ``cache_controller_on_disk`` is cached on disk for 5 seconds. Remember that web2py cannot cache a dictionary that contains un-pickleable objects.

It is also possible to cache the view. The trick is to render the view in the controller function, so that the controller returns a string. This is done by returning ``response.render(d)`` where ``d`` is the dictionary we intended to pass to the view. The following example caches the output of the controller function in RAM (including the rendered view):

``cache view``:inxx

``

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)

def cache_controller_and_view():

import time

t = time.ctime()

d = dict(time=t, link=A('click to reload', _href=request.url))

return response.render(d)

``:code

``response.render(d)`` returns the rendered view as a string, which is now cached for 5 seconds. This is the best and fastest way of caching.

------

------

Note, ``time_expire`` is used to compare the current time with the time the requested object was last saved in the cache. It does not affect future requests. This enables ``time_expire`` to be set dynamically when an object is requested rather than being fixed when the object is saved. For example:

``

message = cache.ram('message', lambda: 'Hello', time_expire=5)

``:code

Now, suppose the following call is made 10 seconds after the above call:

``

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

It is also possible to define other caching mechanisms such as memcache. Memcache is available via ``gluon.contrib.memcache`` and is discussed in more detail in Chapter 14.

------

------

#### ``cache.action``

Web2py by default assumes that the returned content is not going to be cached, as this reduces the shortcomings of an improper caching of the page client-side.

-C, --cron trigger a cron run manually; usually invoked from a

system crontab

--softcron triggers the use of softcron

-Y, --run-cron start the background cron process

-J, --cronjob identify cron-initiated command

-L CONFIG, --config=CONFIG

config file

-F PROFILER_FILENAME, --profiler=PROFILER_FILENAME

profiler filename

-t, --taskbar use web2py gui and run in taskbar (system tray)

--nogui text-only, no GUI

-A ARGS, --args=ARGS should be followed by a list of arguments to be passed

to script, to be used with -S, -A must be the last

option

--no-banner Do not print header banner

--interfaces=INTERFACES

listen on multiple addresses: "ip1:port1:key1:cert1:ca

_cert1;ip2:port2:key2:cert2:ca_cert2;..."

(:key:cert:ca_cert optional; no spaces; IPv6 addresses

must be in square [] brackets)

--run_system_tests runs web2py tests

``:code

Lower-case options are used to configure the web server. The ``-L`` option tells web2py to read configuration options from a file, ``-W`` installs web2py as a windows service, while ``-S``, ``-P`` and ``-M`` options start an interactive Python shell. The ``-T`` option finds and runs controller doctests in a web2py execution environment. For example, the following example runs doctests from all controllers in the "welcome" application:

``

python web2py.py -vT welcome

``:code

if you run web2py as a Windows Service, ``-W``, it is not convenient to pass the configuration using command line arguments. For this reason, in the web2py folder there is a sample "options_std.py" configuration file for the internal web server:

``

import socket

import os

ip = '0.0.0.0'

port = 80

interfaces = [('0.0.0.0', 80)]

#,('0.0.0.0',443,'ssl_private_key.pem','ssl_certificate.pem')]

password = '<recycle>' # ## <recycle> means use the previous password

pid_filename = 'httpserver.pid'

log_filename = 'httpserver.log'

profiler_filename = None

ssl_certificate = None # 'ssl_certificate.pem' # ## path to certificate file

def cache_controller_on_disk():

t = time.ctime()

return dict(time=t, link=A('click to reload',

_href=request.url))

``:code

The dictionary returned by ``cache_controller_on_disk`` is cached on disk for 5 seconds. Remember that web2py cannot cache a dictionary that contains un-pickleable objects.

It is also possible to cache the view. The trick is to render the view in the controller function, so that the controller returns a string. This is done by returning ``response.render(d)`` where ``d`` is the dictionary we intended to pass to the view. The following example caches the output of the controller function in RAM (including the rendered view):

``cache view``:inxx

``

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)

def cache_controller_and_view():

import time

t = time.ctime()

d = dict(time=t, link=A('click to reload', _href=request.url))

return response.render(d)

``:code

``response.render(d)`` returns the rendered view as a string, which is now cached for 5 seconds. This is the best and fastest way of caching.

------

------

Note, ``time_expire`` is used to compare the current time with the time the requested object was last saved in the cache. It does not affect future requests. This enables ``time_expire`` to be set dynamically when an object is requested rather than being fixed when the object is saved. For example:

``

message = cache.ram('message', lambda: 'Hello', time_expire=5)

``:code

Now, suppose the following call is made 10 seconds after the above call:

``

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

It is also possible to define other caching mechanisms such as memcache. Memcache is available via ``gluon.contrib.memcache`` and is discussed in more detail in Chapter 14.

------

------

#### ``cache.action``

Web2py by default assumes that the returned content is not going to be cached, as this reduces the shortcomings of an improper caching of the page client-side.

### ``URL``

``URL``:inxx

The ``URL`` function is one of the most important functions in web2py. It generates internal URL paths for the actions and the static files.

### ``URL``

``URL``:inxx

The ``URL`` function is one of the most important functions in web2py. It generates internal URL paths for the actions and the static files.

**markmin** markup:

``

gluon/contrib/markmin

``:code

**fpdf** created my Mariano Reingart for generating PDF documents:

``

gluon/contrib/fpdf

``

This is not documented in this book but it is hosted and documented here:

``

http://code.google.com/p/pyfpdf/

``

**pysimplesoap** is a lightweight SOAP server implementation created by Mariano Reingart:

``

gluon/contrib/pysimplesoap/

``:code

**simplejsonrpc** is a lightweight JSON-RPC client also created by Mariano Reingart: ``jsonrpc``:inxx

``

gluon/contrib/simplejsonrpc.py

Inside ``%%{...}`` you can also use the following modifiers:

- ``!`` to capitalize the text (equivalent to ``string.capitalize``)

- ``!!`` to capitalize every word (equivalent to ``string.title``)

- ``!!!`` to capitalize every character (equivalent to ``string.upper``)

Notice you can use \ to escape ``!`` and ``?``.

#### Translations, pluralization, and MARKMIN

You can also use the powerful MARKMIN syntax inside translation strings by replacing

``

T("hello world")

``:code

with

``

T.M("hello world")

``

**markmin** markup:

``

gluon/contrib/markmin

``:code

**fpdf** created my Mariano Reingart for generating PDF documents:

``

gluon/contrib/fpdf

``

This is not documented in this book but it is hosted and documented here:

``

http://code.google.com/p/pyfpdf/

``

**pysimplesoap** is a lightweight SOAP server implementation created by Mariano Reingart:

``

gluon/contrib/pysimplesoap/

``:code

**simplejsonrpc** is a lightweight JSON-RPC client also created by Mariano Reingart: ``jsonrpc``:inxx

``

gluon/contrib/simplejsonrpc.py

Inside ``%%{...}`` you can also use the following modifiers:

- ``!`` to capitalize the text (equivalent to ``string.capitalize``)

- ``!!`` to capitalize every word (equivalent to ``string.title``)

- ``!!!`` to capitalize every character (equivalent to ``string.upper``)

Notice you can use \ to escape ``!`` and ``?``.

#### Translations, pluralization, and MARKMIN

You can also use the powerful MARKMIN syntax inside translation strings by replacing

``

T("hello world")

``:code

with

``

T.M("hello world")

``

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``response.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.files``: a list of ``.css``, ``.js``, ``.coffee``, and ``.less`` files required by the page. They will automatically be linked in the head of the standard "layout.html" via the included "web2py_ajax.html". To include a new CSS, JS, COFFEE, or LESS file, just append it to this list. It will handle duplicates. The order is important.

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g. ``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.static_version``: a version number for the static asset management.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.title``: optional parameter that may be included in the views. It should contain the title of the page and should be rendered inside the html title tag in the header.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._vars``: this variable is accessible only in a view, not in the action. It contains the values returned by the action to the view.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.optimize_css``: can be set to "concat,minify,inline" to concatenate, minify and inline the CSS files included by web2py.

- ``response.optimize_js``: can be set to "concat,minify,inline" to concatenate, minify and inline the JavaScript files included by web2py.

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``response.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.files``: a list of ``.css``, ``.js``, ``.coffee``, and ``.less`` files required by the page. They will automatically be linked in the head of the standard "layout.html" via the included "web2py_ajax.html". To include a new CSS, JS, COFFEE, or LESS file, just append it to this list. It will handle duplicates. The order is important.

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g. ``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.static_version``: a version number for the static asset management.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.title``: optional parameter that may be included in the views. It should contain the title of the page and should be rendered inside the html title tag in the header.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._vars``: this variable is accessible only in a view, not in the action. It contains the values returned by the action to the view.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.optimize_css``: can be set to "concat,minify,inline" to concatenate, minify and inline the CSS files included by web2py.

- ``response.optimize_js``: can be set to "concat,minify,inline" to concatenate, minify and inline the JavaScript files included by web2py.

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

All tasks follow a lifecycle

[[scheduler tasks @///image/ce8edcc3.png center]]

By default, when you send a task to the scheduler,

it is in the **QUEUED** status.

If you need it to be executed later, use the ``start_time`` parameter (default = now).

If for some reason you need to be sure that the task does not

get executed after a certain point in time (maybe a request to a web service

that shuts down at 1AM, a mail that needs to be sent not after the working hours, etc...) you can set a ``stop_time`` (default = None) for it.

If your task is NOT picked up by a worker before ``stop_time``, it will be set as **EXPIRED**.

Tasks with no ``stop_time`` set or picked up **BEFORE** stop_time are **ASSIGNED** to a worker. When a workers picks up a task, its status is set to **RUNNING**.

**RUNNING** tasks may end up:

- **TIMEOUT** when more than ``n`` seconds passed with ``timeout`` parameter (default = 60 seconds).

- **FAILED** when an exception is detected,

- **COMPLETED** when they successfully complete.

Values for ``start_time`` and ``stop_time`` should be datetime objects. To schedule "mytask" to run at 30 seconds from the current time, for example, you would do the following:

``

from datetime import timedelta as timed

scheduler.queue_task('mytask',

start_time=request.now + timed(seconds=30))

``:code

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

------

------

You can also set how many times the function can raise an exception (i.e. requesting data from a slow web service) and be queued again instead of stopping in **FAILED** status using the parameter ``retry_failed`` (default = 0, -1 = unlimited).

[[task repeats @///image/7d8b85e4.png center]]

Summary: you have

- ``period`` and ``repeats`` to get an automatically rescheduled function

- ``timeout`` to be sure that a function doesn't exceed a certain amount of time

- ``retry_failed`` to control how many times the task can "fail"

- ``start_time`` and ``stop_time`` to schedule a function in a restricted timeframe

The method:

``

``:code

- ``function`` (required): It can be a task name or a reference to an actual function.

- ``pargs``: are the arguments to be passed to the task, stored as a Python list.

- ``pvars`` : are the named arguments to be passed to the task, stored as a Python dictionary.

For example:

``

scheduler.queue_task('demo1', [1,2])

``

does the exact same thing as

``

scheduler.queue_task('demo1', pvars={'a':1, 'b':2})

``:code

as

``

st.validate_and_insert(function_name='demo1', args=json.dumps([1,2]))

``:code

and as:

``

st.validate_and_insert(function_name='demo1', vars=json.dumps({'a':1,'b':2}))

``:code

Here is a more complex complete example:

``

def task_add(a,b):

return a+b

scheduler = Scheduler(db, tasks=dict(demo1=task_add))

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

All tasks follow a lifecycle

[[scheduler tasks @///image/ce8edcc3.png center]]

By default, when you send a task to the scheduler,

it is in the **QUEUED** status.

If you need it to be executed later, use the ``start_time`` parameter (default = now).

If for some reason you need to be sure that the task does not

get executed after a certain point in time (maybe a request to a web service

that shuts down at 1AM, a mail that needs to be sent not after the working hours, etc...) you can set a ``stop_time`` (default = None) for it.

If your task is NOT picked up by a worker before ``stop_time``, it will be set as **EXPIRED**.

Tasks with no ``stop_time`` set or picked up **BEFORE** stop_time are **ASSIGNED** to a worker. When a workers picks up a task, its status is set to **RUNNING**.

**RUNNING** tasks may end up:

- **TIMEOUT** when more than ``n`` seconds passed with ``timeout`` parameter (default = 60 seconds).

- **FAILED** when an exception is detected,

- **COMPLETED** when they successfully complete.

Values for ``start_time`` and ``stop_time`` should be datetime objects. To schedule "mytask" to run at 30 seconds from the current time, for example, you would do the following:

``

from datetime import timedelta as timed

scheduler.queue_task('mytask',

start_time=request.now + timed(seconds=30))

``:code

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

------

------

You can also set how many times the function can raise an exception (i.e. requesting data from a slow web service) and be queued again instead of stopping in **FAILED** status using the parameter ``retry_failed`` (default = 0, -1 = unlimited).

[[task repeats @///image/7d8b85e4.png center]]

Summary: you have

- ``period`` and ``repeats`` to get an automatically rescheduled function

- ``timeout`` to be sure that a function doesn't exceed a certain amount of time

- ``retry_failed`` to control how many times the task can "fail"

- ``start_time`` and ``stop_time`` to schedule a function in a restricted timeframe

The method:

``

``:code

- ``function`` (required): It can be a task name or a reference to an actual function.

- ``pargs``: are the arguments to be passed to the task, stored as a Python list.

- ``pvars`` : are the named arguments to be passed to the task, stored as a Python dictionary.

For example:

``

scheduler.queue_task('demo1', [1,2])

``

does the exact same thing as

``

scheduler.queue_task('demo1', pvars={'a':1, 'b':2})

``:code

as

``

st.validate_and_insert(function_name='demo1', args=json.dumps([1,2]))

``:code

and as:

``

st.validate_and_insert(function_name='demo1', vars=json.dumps({'a':1,'b':2}))

``:code

Here is a more complex complete example:

``

def task_add(a,b):

return a+b

scheduler = Scheduler(db, tasks=dict(demo1=task_add))

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

There are three example files in the "examples" directory:

``

options_std.py

``:code

They are all meant to be copied to the root dir (where web2py.py or web2py.exe is) and edited according to your own preferences.

The former is an optional configuration file that can be passed to web2py.py with the ``-L`` option. The second is an example of a URL mapping file. It is loaded automatically when renamed "routes.py". The third is an alternative syntax for URL mapping, and can also be renamed (or copied to) "routes.py".

The files

``

app.example.yaml

queue.example.yaml

``:code

are example configuration files used for deployment on the Google App Engine. You can read more about them in the Deployment Recipes chapter and on the Google Documentation pages.

There are also additional libraries, some developed by a third party:

**feedparser**``feedparser``:cite by Mark Pilgrim for reading RSS and Atom feeds:

``

gluon/contrib/__init__.py

gluon/contrib/feedparser.py

``:code

routers = dict(

'domain1.com' : 'app1',

'domain2.com' : 'app2',

}

),

)

``:code

does what you'd expect.

``

routers = dict(

BASE = dict(

domains = {

'domain.com:80' : 'app/insecure',

'domain.com:443' : 'app/secure',

}

),

)

``:code

maps ``http://domain.com`` accesses to the controller named ``insecure``, while ``HTTPS`` accesses go to the ``secure`` controller. Alternatively, you can map different ports to different apps, in the obvious way.

Note: The ''parameter-based'' system first appeared in web2py version 1.92.1.

#### Pattern-based system

Although the ''parameter-based'' system just described should be sufficient for most use cases, the alternative ''pattern-based'' system provides some additional flexibility for more complex cases. To use the pattern-based system, instead of defining routers as dictionaries of routing parameters, you define two lists (or tuples) of 2-tuples, ``routes_in`` and ``routes_out``. Each tuple contains two elements: the pattern to be replaced and the string that replaces it. For example:

``

routes_in = (

('/testme', '/examples/default/index'),

)

routes_out = (

('/examples/default/index', '/testme'),

)

``:code

With these routes, the URL:

``

http://127.0.0.1:8000/testme

``:code

Note: These items first appeared in web2py version 1.83.

``routes_onerror``:inxx

You can also use ``routes.py`` to re-route requests to special actions in case there is an error on the server. You can specify this mapping globally, for each app, for each error code, or for each app and error code. Here is an example:

``

routes_onerror = [

('init/400', '/init/default/login'),

('init/*', '/init/static/fail.html'),

('*/404', '/init/static/cantfind.html'),

('*/*', '/init/error/index')

]

``:code

For each tuple, the first string is matched against "[app name]/[error code]". If a match is found, the failed request is re-routed to the URL in the second string of the matching tuple. If the error handling URL is a not a static file, the following GET variables will be passed to the error action:

- ``code``: the HTTP status code (e.g., 404, 500)

- ``ticket``: in the form of "[app name]/[ticket number]" (or "None" if no ticket)

- ``requested_uri``: equivalent to ``request.env.request_uri``

- ``request_url``: equivalent to ``request.url``

These variables will be accessible to the error handling action via ``request.vars`` and can be used in generating the error response. In particular, it is a good idea for the error action to return the original HTTP error code instead of the default 200 (OK) status code. This can be done by setting ``response.status = request.vars.code``. It is also possible to have the error action send (or queue) an email to an administrator, including a link to the ticket in ``admin``.

``

error_message = '<html><body><h1>%s</h1></body></html>'

error_message_ticket = '''<html><body><h1>Internal error</h1>

Ticket issued: <a href="/admin/default/ticket/%(ticket)s"

target="_blank">%(ticket)s</a></body></html>'''

``:code

There are three example files in the "examples" directory:

``

options_std.py

``:code

They are all meant to be copied to the root dir (where web2py.py or web2py.exe is) and edited according to your own preferences.

The former is an optional configuration file that can be passed to web2py.py with the ``-L`` option. The second is an example of a URL mapping file. It is loaded automatically when renamed "routes.py". The third is an alternative syntax for URL mapping, and can also be renamed (or copied to) "routes.py".

The files

``

app.example.yaml

queue.example.yaml

``:code

are example configuration files used for deployment on the Google App Engine. You can read more about them in the Deployment Recipes chapter and on the Google Documentation pages.

There are also additional libraries, some developed by a third party:

**feedparser**``feedparser``:cite by Mark Pilgrim for reading RSS and Atom feeds:

``

gluon/contrib/__init__.py

gluon/contrib/feedparser.py

``:code

routers = dict(

'domain1.com' : 'app1',

'domain2.com' : 'app2',

}

),

)

``:code

does what you'd expect.

``

routers = dict(

BASE = dict(

domains = {

'domain.com:80' : 'app/insecure',

'domain.com:443' : 'app/secure',

}

),

)

``:code

maps ``http://domain.com`` accesses to the controller named ``insecure``, while ``HTTPS`` accesses go to the ``secure`` controller. Alternatively, you can map different ports to different apps, in the obvious way.

Note: The ''parameter-based'' system first appeared in web2py version 1.92.1.

#### Pattern-based system

Although the ''parameter-based'' system just described should be sufficient for most use cases, the alternative ''pattern-based'' system provides some additional flexibility for more complex cases. To use the pattern-based system, instead of defining routers as dictionaries of routing parameters, you define two lists (or tuples) of 2-tuples, ``routes_in`` and ``routes_out``. Each tuple contains two elements: the pattern to be replaced and the string that replaces it. For example:

``

routes_in = (

('/testme', '/examples/default/index'),

)

routes_out = (

('/examples/default/index', '/testme'),

)

``:code

With these routes, the URL:

``

http://127.0.0.1:8000/testme

``:code

Note: These items first appeared in web2py version 1.83.

``routes_onerror``:inxx

You can also use ``routes.py`` to re-route requests to special actions in case there is an error on the server. You can specify this mapping globally, for each app, for each error code, or for each app and error code. Here is an example:

``

routes_onerror = [

('init/400', '/init/default/login'),

('init/*', '/init/static/fail.html'),

('*/404', '/init/static/cantfind.html'),

('*/*', '/init/error/index')

]

``:code

For each tuple, the first string is matched against "[app name]/[error code]". If a match is found, the failed request is re-routed to the URL in the second string of the matching tuple. If the error handling URL is a not a static file, the following GET variables will be passed to the error action:

- ``code``: the HTTP status code (e.g., 404, 500)

- ``ticket``: in the form of "[app name]/[ticket number]" (or "None" if no ticket)

- ``requested_uri``: equivalent to ``request.env.request_uri``

- ``request_url``: equivalent to ``request.url``

These variables will be accessible to the error handling action via ``request.vars`` and can be used in generating the error response. In particular, it is a good idea for the error action to return the original HTTP error code instead of the default 200 (OK) status code. This can be done by setting ``response.status = request.vars.code``. It is also possible to have the error action send (or queue) an email to an administrator, including a link to the ticket in ``admin``.

``

error_message = '<html><body><h1>%s</h1></body></html>'

error_message_ticket = '''<html><body><h1>Internal error</h1>

Ticket issued: <a href="/admin/default/ticket/%(ticket)s"

target="_blank">%(ticket)s</a></body></html>'''

``:code

def f(a,b,c,): ....

``:code

def f(a,b,c,): ....

``:code

A few objects and functions, including **Auth**, **Crud** and **Service**, are defined in "gluon/tools.py" and they need to be imported as necessary:

``

from gluon.tools import Auth, Crud, Service

``:code

A few objects and functions, including **Auth**, **Crud** and **Service**, are defined in "gluon/tools.py" and they need to be imported as necessary:

``

from gluon.tools import Auth, Crud, Service

``:code

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

- ``response.models_to_run`` [[response_models_to_run]] contains a list of regexes that chooses what models to run.

-- By default, this is set automatically to load /a/models/*.py, /a/models/c/*.py, and /a/models/c/f/*.py files when ``/a/c/f`` is requested. You can set, e.g., ``response.models_to_run = ['myfolder/']`` to force the execution only of the models inside your application's ``models/myfolder`` subfolder.

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

- ``response.models_to_run`` [[response_models_to_run]] contains a list of regexes that chooses what models to run.

-- By default, this is set automatically to load /a/models/*.py, /a/models/c/*.py, and /a/models/c/f/*.py files when ``/a/c/f`` is requested. You can set, e.g., ``response.models_to_run = ['myfolder/']`` to force the execution only of the models inside your application's ``models/myfolder`` subfolder.

------

Remember to call ``db.commit()`` at the end of every task if it involves inserts/updates to the database. web2py commits by default at the end of a successful action but the scheduler tasks are not actions.

------

To enable the scheduler you must instantiate the Scheduler class in a model.

The recommended way to enable the scheduler to your app is to create a model file named ``scheduler.py`` and define your function there. After the functions, you can put the following code into the model:

``

from gluon.scheduler import Scheduler

scheduler = Scheduler(db)

``:code

If your tasks are defined in a module (as opposed to a model) you may have to restart the workers.

The task is scheduled with

``

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

you get the result of the validation, and id and uuid will be None

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

------

Remember to call ``db.commit()`` at the end of every task if it involves inserts/updates to the database. web2py commits by default at the end of a successful action but the scheduler tasks are not actions.

------

To enable the scheduler you must instantiate the Scheduler class in a model.

The recommended way to enable the scheduler to your app is to create a model file named ``scheduler.py`` and define your function there. After the functions, you can put the following code into the model:

``

from gluon.scheduler import Scheduler

scheduler = Scheduler(db)

``:code

If your tasks are defined in a module (as opposed to a model) you may have to restart the workers.

The task is scheduled with

``

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

you get the result of the validation, and id and uuid will be None

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

If you specify ``@reboot`` in the first field in the crontab file, the given task will be executed only once, at web2py startup. You can use this feature if you want to pre-cache, check, or initialize data for an application on web2py startup. Note that cron tasks are executed in parallel with the application --- if the application is not ready to serve requests until the cron task is finished, you should implement checks to reflect this. Example:

``

``:code

If you specify ``@reboot`` in the first field in the crontab file, the given task will be executed only once, at web2py startup. You can use this feature if you want to pre-cache, check, or initialize data for an application on web2py startup. Note that cron tasks are executed in parallel with the application --- if the application is not ready to serve requests until the cron task is finished, you should implement checks to reflect this. Example:

``

``:code

``

options_std.py

routes.example.py

router.example.py

``:code

The former is an optional configuration file that can be passed to web2py.py with the ``-L`` option. The second is an example of a URL mapping file. It is loaded automatically when renamed "routes.py". The third is an alternative syntax for URL mapping, and can also be renamed (or copied to) "routes.py".

The files

``

app.example.yaml

queue.example.yaml

``:code

are example configuration files used for deployment on the Google App Engine. You can read more about them in the Deployment Recipes chapter and on the Google Documentation pages.

There are also additional libraries, some developed by a third party:

**feedparser**``feedparser``:cite by Mark Pilgrim for reading RSS and Atom feeds:

``

gluon/contrib/__init__.py

gluon/contrib/feedparser.py

``:code

**markdown2**``markdown2``:cite by Trent Mick for wiki markup:

``

routers = dict(

'domain1.com' : 'app1',

'domain2.com' : 'app2',

}

),

)

``:code

does what you'd expect.

``

routers = dict(

BASE = dict(

domains = {

'domain.com:80' : 'app/insecure',

'domain.com:443' : 'app/secure',

}

),

)

``:code

maps ``http://domain.com`` accesses to the controller named ``insecure``, while ``HTTPS`` accesses go to the ``secure`` controller. Alternatively, you can map different ports to different apps, in the obvious way.

Note: The ''parameter-based'' system first appeared in web2py version 1.92.1.

#### Pattern-based system

Although the ''parameter-based'' system just described should be sufficient for most use cases, the alternative ''pattern-based'' system provides some additional flexibility for more complex cases. To use the pattern-based system, instead of defining routers as dictionaries of routing parameters, you define two lists (or tuples) of 2-tuples, ``routes_in`` and ``routes_out``. Each tuple contains two elements: the pattern to be replaced and the string that replaces it. For example:

``

routes_in = (

('/testme', '/examples/default/index'),

)

routes_out = (

('/examples/default/index', '/testme'),

)

``:code

With these routes, the URL:

``

http://127.0.0.1:8000/testme

``:code

Note: These items first appeared in web2py version 1.83.

``routes_onerror``:inxx

You can also use ``routes.py`` to re-route requests to special actions in case there is an error on the server. You can specify this mapping globally, for each app, for each error code, or for each app and error code. Here is an example:

``

routes_onerror = [

('init/400', '/init/default/login'),

('init/*', '/init/static/fail.html'),

('*/404', '/init/static/cantfind.html'),

('*/*', '/init/error/index')

]

``:code

For each tuple, the first string is matched against "[app name]/[error code]". If a match is found, the failed request is re-routed to the URL in the second string of the matching tuple. If the error handling URL is a not a static file, the following GET variables will be passed to the error action:

- ``code``: the HTTP status code (e.g., 404, 500)

- ``ticket``: in the form of "[app name]/[ticket number]" (or "None" if no ticket)

- ``requested_uri``: equivalent to ``request.env.request_uri``

- ``request_url``: equivalent to ``request.url``

These variables will be accessible to the error handling action via ``request.vars`` and can be used in generating the error response. In particular, it is a good idea for the error action to return the original HTTP error code instead of the default 200 (OK) status code. This can be done by setting ``response.status = request.vars.code``. It is also possible to have the error action send (or queue) an email to an administrator, including a link to the ticket in ``admin``.

``

error_message = '<html><body><h1>%s</h1></body></html>'

error_message_ticket = '''<html><body><h1>Internal error</h1>

Ticket issued: <a href="/admin/default/ticket/%(ticket)s"

target="_blank">%(ticket)s</a></body></html>'''

``:code

The first variable contains the error message when an invalid application or function is requested. The second variable contains the error message when a ticket is issued.

``routes_onerror`` work with both routing mechanisms.

``error_handler``:inxx

In "routes.py" you can also specify an action in charge of error handling:

``

error_handler = dict(application='error',

controller='default',

function='index')

``:code

import logging

logger = logging.getLogger("web2py.app.myapp")

logger.setLevel(logging.DEBUG)

``:code

and you can use it to log messages of various importance

``

logger.debug("Just checking that %s" % details)

logger.info("You ought to know that %s" % details)

logger.warn("Mind that %s" % details)

logger.error("Oops, something bad happened %s" % details)

``:code

``logging`` is a standard python module described here:

``

http://docs.python.org/library/logging.html

``

The string "web2py.app.myapp" defines an app-level logger.

For this to work properly, you need a configuration file for the logger.

``

options_std.py

routes.example.py

router.example.py

``:code

The former is an optional configuration file that can be passed to web2py.py with the ``-L`` option. The second is an example of a URL mapping file. It is loaded automatically when renamed "routes.py". The third is an alternative syntax for URL mapping, and can also be renamed (or copied to) "routes.py".

The files

``

app.example.yaml

queue.example.yaml

``:code

are example configuration files used for deployment on the Google App Engine. You can read more about them in the Deployment Recipes chapter and on the Google Documentation pages.

There are also additional libraries, some developed by a third party:

**feedparser**``feedparser``:cite by Mark Pilgrim for reading RSS and Atom feeds:

``

gluon/contrib/__init__.py

gluon/contrib/feedparser.py

``:code

**markdown2**``markdown2``:cite by Trent Mick for wiki markup:

``

routers = dict(

'domain1.com' : 'app1',

'domain2.com' : 'app2',

}

),

)

``:code

does what you'd expect.

``

routers = dict(

BASE = dict(

domains = {

'domain.com:80' : 'app/insecure',

'domain.com:443' : 'app/secure',

}

),

)

``:code

maps ``http://domain.com`` accesses to the controller named ``insecure``, while ``HTTPS`` accesses go to the ``secure`` controller. Alternatively, you can map different ports to different apps, in the obvious way.

Note: The ''parameter-based'' system first appeared in web2py version 1.92.1.

#### Pattern-based system

Although the ''parameter-based'' system just described should be sufficient for most use cases, the alternative ''pattern-based'' system provides some additional flexibility for more complex cases. To use the pattern-based system, instead of defining routers as dictionaries of routing parameters, you define two lists (or tuples) of 2-tuples, ``routes_in`` and ``routes_out``. Each tuple contains two elements: the pattern to be replaced and the string that replaces it. For example:

``

routes_in = (

('/testme', '/examples/default/index'),

)

routes_out = (

('/examples/default/index', '/testme'),

)

``:code

With these routes, the URL:

``

http://127.0.0.1:8000/testme

``:code

Note: These items first appeared in web2py version 1.83.

``routes_onerror``:inxx

You can also use ``routes.py`` to re-route requests to special actions in case there is an error on the server. You can specify this mapping globally, for each app, for each error code, or for each app and error code. Here is an example:

``

routes_onerror = [

('init/400', '/init/default/login'),

('init/*', '/init/static/fail.html'),

('*/404', '/init/static/cantfind.html'),

('*/*', '/init/error/index')

]

``:code

For each tuple, the first string is matched against "[app name]/[error code]". If a match is found, the failed request is re-routed to the URL in the second string of the matching tuple. If the error handling URL is a not a static file, the following GET variables will be passed to the error action:

- ``code``: the HTTP status code (e.g., 404, 500)

- ``ticket``: in the form of "[app name]/[ticket number]" (or "None" if no ticket)

- ``requested_uri``: equivalent to ``request.env.request_uri``

- ``request_url``: equivalent to ``request.url``

These variables will be accessible to the error handling action via ``request.vars`` and can be used in generating the error response. In particular, it is a good idea for the error action to return the original HTTP error code instead of the default 200 (OK) status code. This can be done by setting ``response.status = request.vars.code``. It is also possible to have the error action send (or queue) an email to an administrator, including a link to the ticket in ``admin``.

``

error_message = '<html><body><h1>%s</h1></body></html>'

error_message_ticket = '''<html><body><h1>Internal error</h1>

Ticket issued: <a href="/admin/default/ticket/%(ticket)s"

target="_blank">%(ticket)s</a></body></html>'''

``:code

The first variable contains the error message when an invalid application or function is requested. The second variable contains the error message when a ticket is issued.

``routes_onerror`` work with both routing mechanisms.

``error_handler``:inxx

In "routes.py" you can also specify an action in charge of error handling:

``

error_handler = dict(application='error',

controller='default',

function='index')

``:code

import logging

logger = logging.getLogger("web2py.app.myapp")

logger.setLevel(logging.DEBUG)

``:code

and you can use it to log messages of various importance

``

logger.debug("Just checking that %s" % details)

logger.info("You ought to know that %s" % details)

logger.warn("Mind that %s" % details)

logger.error("Oops, something bad happened %s" % details)

``:code

``logging`` is a standard python module described here:

``

http://docs.python.org/library/logging.html

``

The string "web2py.app.myapp" defines an app-level logger.

For this to work properly, you need a configuration file for the logger.

``request`` has the following items/attributes, some of which are also an instance of the ``Storage`` class:

- ``request.cookies``: a ``Cookie.SimpleCookie()`` object containing the cookies passed with the HTTP request. It acts like a dictionary of cookies. Each cookie is a Morsel object ``morsel``:cite.

- ``request.env``: a ``Storage`` object containing the environment variables passed to the controller, including HTTP header variables from the HTTP request and standard WSGI parameters. The environment variables are all converted to lower case, and dots are converted to underscores for easier memorization.

- ``request.application``: the name of the requested application.

- ``request.controller``: the name of the requested controller.

- ``request.function``: the name of the requested function.

- ``request.extension``: the extension of the requested action. It defaults to "html". If the controller function returns a dictionary and does not specify a view, this is used to determine the extension of the view file that will render the dictionary (parsed from the ``request.env.path_info``).

- ``request.folder``: the application directory. For example if the application is "welcome", ``request.folder`` is set to the absolute path "/path/to/welcome". In your programs, you should always use this variable and the ``os.path.join`` function to build paths to the files you need to access. Although web2py always uses absolute paths, it is a good rule never to explicitly change the current working folder (whatever that is) since this is not a thread-safe practice.

- ``request.now``: a ``datetime.datetime`` object storing the datetime of the current request.

- ``request.utcnow``: a ``datetime.datetime`` object storing the UTC datetime of the current request.

- ``request.args``: A list of the URL path components following the controller function name; equivalent to ``request.env.path_info.split('/')[3:]``

- ``request.client``: The ip address of the client as determined by, if present, ``request.env.http_x_forwarded_for`` or by ``request.env.remote_addr`` otherwise. While this is useful it should not be trusted because the ``http_x_forwarded_for`` can be spoofed.

- ``request.is_local``: ``True`` if the client is localhost, ``False`` otherwise. Should work behind a proxy if the proxy supports ``http_x_forwarded_for``.

- ``request.is_https``: ``True`` if the request is using the HTTPS protocol, ``False`` otherwise.

- ``request.body``: a read-only file stream that contains the body of the HTTP request. This is automatically parsed to get the ``request.post_vars`` and then rewinded. It can be read with ``request.body.read()``.

- ``request.ajax`` is True if the function is being called via an Ajax request.

- ``request.cid`` is the ``id`` of the component that generated the Ajax request (if any). You can read more about components in Chapter 12.

- ``request.requires_https()`` prevents further code execution if the request is not over HTTPS and redirects the visitor to the current page over HTTPS.

- ``request.restful`` this is a new and very useful decorator that can be used to change the default behavior of web2py actions by separating GET/POST/PUSH/DELETE requests. It will be discussed in some detail in Chapter 10.

- ``request.user_agent()`` parses the user_agent field from the client and returns the information in the form of a dictionary. It is useful to detect mobile devices. It uses "gluon/contrib/user_agent_parser.py" created by Ross Peoples. To see what it does, try to embed the following code in a view:

``

{{=BEAUTIFY(request.user_agent())}}

``:code

``request`` has the following items/attributes, some of which are also an instance of the ``Storage`` class:

- ``request.cookies``: a ``Cookie.SimpleCookie()`` object containing the cookies passed with the HTTP request. It acts like a dictionary of cookies. Each cookie is a Morsel object ``morsel``:cite.

- ``request.env``: a ``Storage`` object containing the environment variables passed to the controller, including HTTP header variables from the HTTP request and standard WSGI parameters. The environment variables are all converted to lower case, and dots are converted to underscores for easier memorization.

- ``request.application``: the name of the requested application.

- ``request.controller``: the name of the requested controller.

- ``request.function``: the name of the requested function.

- ``request.extension``: the extension of the requested action. It defaults to "html". If the controller function returns a dictionary and does not specify a view, this is used to determine the extension of the view file that will render the dictionary (parsed from the ``request.env.path_info``).

- ``request.folder``: the application directory. For example if the application is "welcome", ``request.folder`` is set to the absolute path "/path/to/welcome". In your programs, you should always use this variable and the ``os.path.join`` function to build paths to the files you need to access. Although web2py always uses absolute paths, it is a good rule never to explicitly change the current working folder (whatever that is) since this is not a thread-safe practice.

- ``request.now``: a ``datetime.datetime`` object storing the datetime of the current request.

- ``request.utcnow``: a ``datetime.datetime`` object storing the UTC datetime of the current request.

- ``request.args``: A list of the URL path components following the controller function name; equivalent to ``request.env.path_info.split('/')[3:]``

- ``request.client``: The ip address of the client as determined by, if present, ``request.env.http_x_forwarded_for`` or by ``request.env.remote_addr`` otherwise. While this is useful it should not be trusted because the ``http_x_forwarded_for`` can be spoofed.

- ``request.is_local``: ``True`` if the client is localhost, ``False`` otherwise. Should work behind a proxy if the proxy supports ``http_x_forwarded_for``.

- ``request.is_https``: ``True`` if the request is using the HTTPS protocol, ``False`` otherwise.

- ``request.body``: a read-only file stream that contains the body of the HTTP request. This is automatically parsed to get the ``request.post_vars`` and then rewinded. It can be read with ``request.body.read()``.

- ``request.ajax`` is True if the function is being called via an Ajax request.

- ``request.cid`` is the ``id`` of the component that generated the Ajax request (if any). You can read more about components in Chapter 12.

- ``request.requires_https()`` prevents further code execution if the request is not over HTTPS and redirects the visitor to the current page over HTTPS.

- ``request.restful`` this is a new and very useful decorator that can be used to change the default behavior of web2py actions by separating GET/POST/PUSH/DELETE requests. It will be discussed in some detail in Chapter 10.

- ``request.user_agent()`` parses the user_agent field from the client and returns the information in the form of a dictionary. It is useful to detect mobile devices. It uses "gluon/contrib/user_agent_parser.py" created by Ross Peoples. To see what it does, try to embed the following code in a view:

``

{{=BEAUTIFY(request.user_agent())}}

``:code

``request.get_vars``:inxx ``request.post_vars``:inxx ``request.vars``:inxx

``request.get_vars``:inxx ``request.post_vars``:inxx ``request.vars``:inxx

The web2py scheduler works very much like the task queue described in the previous sub-section with some differences:

- It provides a standard mechanism for creating, scheduling, and monitoring tasks.

- There is not a single background process but a set of workers processes.

- The job of worker nodes can be monitored because their state, as well as the state of the tasks, is stored in the database.

- It works without web2py but that is not documented here.

The scheduler does not use cron, although one can use cron @reboot to start the worker nodes.

More information about deploying the scheduler under Linux and Windows is in the Deployment recipes chapter.

In the scheduler, a task is simply a function defined in a model (or in a module and imported by a model). For example:

``

def task_add(a,b):

return a+b

``:code

Tasks will always be called in the same environment seen by controllers and therefore they see all the global variables defined in models, including database connections (``db``). Tasks differ from a controller action because they are not associated with an HTTP request and therefore there is no ``request.env``.

The task is scheduled with

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

##### Parameters

The first argument of the ``Scheduler`` class must be the database to be used by the scheduler to communicate with the workers. This can be the ``db`` of the app or another dedicated ``db``, perhaps one shared by multiple apps. If you use SQLite it's recommended to use a separate db from the one used by your app in order to keep the app responsive.

Once the tasks are defined and the ``Scheduler`` is instantiated, all that is needed to do is to start the workers. You can do that in several ways:

``

python web2py.py -K myapp

``

starts a worker for the app ``myapp``. If you want start multiple workers for the same app, you can do so just passing ``myapp,myapp``. You can pass also the ``group_names`` (overriding the one set in your model) with

``

python web2py.py -K myapp:group1:group2,myotherapp:group1

``

If you have a model called ``scheduler.py`` you can start/stop the workers from web2py's default window (the one you use to set the ip address and the port).

One last nice addition: if you use the embedded webserver, you can start the webserver and the scheduler with just one line of code (this assumes you don't want the web2py window popping up, else you can use the "Schedulers" menu instead)

``

python web2py.py -a yourpass -K myapp -X

``

You can pass the usual parameters (-i, -p, here -a prevents the window from showing up), pass whatever app in the -K parameter and append a -X. The scheduler will run alongside the webserver!

Scheduler's complete signature is:

The web2py scheduler works very much like the task queue described in the previous sub-section with some differences:

- It provides a standard mechanism for creating, scheduling, and monitoring tasks.

- There is not a single background process but a set of workers processes.

- The job of worker nodes can be monitored because their state, as well as the state of the tasks, is stored in the database.

- It works without web2py but that is not documented here.

The scheduler does not use cron, although one can use cron @reboot to start the worker nodes.

More information about deploying the scheduler under Linux and Windows is in the Deployment recipes chapter.

In the scheduler, a task is simply a function defined in a model (or in a module and imported by a model). For example:

``

def task_add(a,b):

return a+b

``:code

Tasks will always be called in the same environment seen by controllers and therefore they see all the global variables defined in models, including database connections (``db``). Tasks differ from a controller action because they are not associated with an HTTP request and therefore there is no ``request.env``.

The task is scheduled with

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

##### Parameters

The first argument of the ``Scheduler`` class must be the database to be used by the scheduler to communicate with the workers. This can be the ``db`` of the app or another dedicated ``db``, perhaps one shared by multiple apps. If you use SQLite it's recommended to use a separate db from the one used by your app in order to keep the app responsive.

Once the tasks are defined and the ``Scheduler`` is instantiated, all that is needed to do is to start the workers. You can do that in several ways:

``

python web2py.py -K myapp

``

starts a worker for the app ``myapp``. If you want start multiple workers for the same app, you can do so just passing ``myapp,myapp``. You can pass also the ``group_names`` (overriding the one set in your model) with

``

python web2py.py -K myapp:group1:group2,myotherapp:group1

``

If you have a model called ``scheduler.py`` you can start/stop the workers from web2py's default window (the one you use to set the ip address and the port).

One last nice addition: if you use the embedded webserver, you can start the webserver and the scheduler with just one line of code (this assumes you don't want the web2py window popping up, else you can use the "Schedulers" menu instead)

``

python web2py.py -a yourpass -K myapp -X

``

You can pass the usual parameters (-i, -p, here -a prevents the window from showing up), pass whatever app in the -K parameter and append a -X. The scheduler will run alongside the webserver!

Scheduler's complete signature is:

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

----------

**variable** | **value**

``request.application`` | ``examples``

``request.controller`` | ``default``

----------

**variable** | **value**

``request.application`` | ``examples``

``request.controller`` | ``default``

There are some caveats to keep in mind:

- Models in the same folder/subfolder are executed in alphabetical order.

- Any variable defined in a model will be visible to other models following alphabetically, to the controllers, and to the views.

- Models in subfolders are executed conditionally. For example, if the user has requested "/a/c/f" where "a" is the application, "c" is the controller, and "f" is the function (action), then the following models are executed:

``

applications/a/models/*.py

applications/a/models/c/*.py

applications/a/models/c/f/*.py

``

- The requested controller is executed and the requested function is called. This means all top-level code in the controller is also executed at every request for that controller.

- The view is only called if the action returns a dictionary.

- If a view is not found, web2py tries to use a generic view. By default, generic views are disabled, although the 'welcome' app includes a line in /models/db.py to enable them on localhost only. They can be enabled per extension type and per action (using ``response.generic_patterns``). In general, generic views are a development tool and typically should not be used in production. If you want some actions to use a generic view, list those actions in ``response.generic_patterns`` (discussed in more detail in the chapter on Services).

The possible behaviors of an action are the following:

**Return a string**

``

def index(): return 'data'

``

**Return a dictionary for a view**:

``

def index(): return dict(key='value')

``

**Return all local variables**:

``

def index(): return locals()

Also notice the ``request.env.wsgi_*`` variables. They are specific to the wsgi

``response.headers``:inxx

``response.meta``:inxx

``response.menu``:inxx

``response.postprocessing``:inxx

``response.render``:inxx

``response.static_version``:inxx

``response.status``:inxx

``response.stream``:inxx

``response.subtitle``:inxx

``response.title``:inxx

``response.toolbar``:inxx

``response.view``:inxx

``response.delimiters``:inxx

``response.js``:inxx

``response.write``:inxx

``response.include_files``:inxx

``response.include_meta``:inxx

``response.optimize_css``:inxx

``response.optimize_js``:inxx

``response._caller``:inxx

``response`` is another instance of the ``Storage`` class. It contains the following:

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.files``: a list of ``.css``, ``.js``, ``.coffee``, and ``.less`` files required by the page. They will automatically be linked in the head of the standard "layout.html" via the included "web2py_ajax.html". To include a new CSS, JS, COFFEE, or LESS file, just append it to this list. It will handle duplicates. The order is important.

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g. ``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.static_version``: a version number for the static asset management.

Also notice the ``request.env.wsgi_*`` variables. They are specific to the wsgi

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.optimize_css``: can be set to "concat,minify,inline" to concatenate, minify and inline the CSS files included by web2py.

- ``response.optimize_js``: can be set to "concat,minify,inline" to concatenate, minify and inline the JavaScript files included by web2py.

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

Since ``response`` is a ``gluon.storage.Storage`` object, it can be used to store other attributes that you may want to pass to the view. While there is no technical restriction, our recommendation is to store only variables that are to be rendered by all pages in the overall layout ("layout.html").

Anyway, we strongly suggest to stick to the variables listed here:

``

response.title

response.subtitle

response.flash

response.menu

response.meta.author

response.meta.description

response.meta.keywords

response.meta.*

``:code

because this will make it easier for you to replace the standard "layout.html" file that comes with web2py with another layout file, one that uses the same set of variables.

Old versions of web2py used ``response.author`` instead of ``response.meta.author`` and similar for the other meta attributes.

### ``session``

redirect('http://www.web2py.com')

``:code

is simply a shortcut for:

``

raise HTTP(303,

'You are being redirected <a href="%s">here</a>' % location,

Location='http://www.web2py.com')

``:code

The named arguments of the ``HTTP`` initializer method are translated into HTTP header directives, in this case, the redirection target location. ``redirect`` takes an optional second argument, which is the HTTP status code for the redirection (303 by default). Change this number to 307 for a temporary redirect or to 301 for a permanent redirect.

The most common way to use redirect is to redirect to other pages in the same app and (optionally) pass parameters:

``

redirect(URL('index', args=(1,2,3), vars=dict(a='b')))

``:code

In chapter 12 we discuss web2py components. They make Ajax requests to web2py actions. If the called action performs a redirect, you may want the Ajax request to follow the redirect or you may want the entire page performing the Ajax request redirecting. In this latter case you can set:

``

``:code

There are some caveats to keep in mind:

- Models in the same folder/subfolder are executed in alphabetical order.

- Any variable defined in a model will be visible to other models following alphabetically, to the controllers, and to the views.

- Models in subfolders are executed conditionally. For example, if the user has requested "/a/c/f" where "a" is the application, "c" is the controller, and "f" is the function (action), then the following models are executed:

``

applications/a/models/*.py

applications/a/models/c/*.py

applications/a/models/c/f/*.py

``

- The requested controller is executed and the requested function is called. This means all top-level code in the controller is also executed at every request for that controller.

- The view is only called if the action returns a dictionary.

- If a view is not found, web2py tries to use a generic view. By default, generic views are disabled, although the 'welcome' app includes a line in /models/db.py to enable them on localhost only. They can be enabled per extension type and per action (using ``response.generic_patterns``). In general, generic views are a development tool and typically should not be used in production. If you want some actions to use a generic view, list those actions in ``response.generic_patterns`` (discussed in more detail in the chapter on Services).

The possible behaviors of an action are the following:

**Return a string**

``

def index(): return 'data'

``

**Return a dictionary for a view**:

``

def index(): return dict(key='value')

``

**Return all local variables**:

``

def index(): return locals()

Also notice the ``request.env.wsgi_*`` variables. They are specific to the wsgi

``response.headers``:inxx

``response.meta``:inxx

``response.menu``:inxx

``response.postprocessing``:inxx

``response.render``:inxx

``response.static_version``:inxx

``response.status``:inxx

``response.stream``:inxx

``response.subtitle``:inxx

``response.title``:inxx

``response.toolbar``:inxx

``response.view``:inxx

``response.delimiters``:inxx

``response.js``:inxx

``response.write``:inxx

``response.include_files``:inxx

``response.include_meta``:inxx

``response.optimize_css``:inxx

``response.optimize_js``:inxx

``response._caller``:inxx

``response`` is another instance of the ``Storage`` class. It contains the following:

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.files``: a list of ``.css``, ``.js``, ``.coffee``, and ``.less`` files required by the page. They will automatically be linked in the head of the standard "layout.html" via the included "web2py_ajax.html". To include a new CSS, JS, COFFEE, or LESS file, just append it to this list. It will handle duplicates. The order is important.

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g. ``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.static_version``: a version number for the static asset management.

Also notice the ``request.env.wsgi_*`` variables. They are specific to the wsgi

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.optimize_css``: can be set to "concat,minify,inline" to concatenate, minify and inline the CSS files included by web2py.

- ``response.optimize_js``: can be set to "concat,minify,inline" to concatenate, minify and inline the JavaScript files included by web2py.

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

Since ``response`` is a ``gluon.storage.Storage`` object, it can be used to store other attributes that you may want to pass to the view. While there is no technical restriction, our recommendation is to store only variables that are to be rendered by all pages in the overall layout ("layout.html").

Anyway, we strongly suggest to stick to the variables listed here:

``

response.title

response.subtitle

response.flash

response.menu

response.meta.author

response.meta.description

response.meta.keywords

response.meta.*

``:code

because this will make it easier for you to replace the standard "layout.html" file that comes with web2py with another layout file, one that uses the same set of variables.

Old versions of web2py used ``response.author`` instead of ``response.meta.author`` and similar for the other meta attributes.

### ``session``

redirect('http://www.web2py.com')

``:code

is simply a shortcut for:

``

raise HTTP(303,

'You are being redirected <a href="%s">here</a>' % location,

Location='http://www.web2py.com')

``:code

The named arguments of the ``HTTP`` initializer method are translated into HTTP header directives, in this case, the redirection target location. ``redirect`` takes an optional second argument, which is the HTTP status code for the redirection (303 by default). Change this number to 307 for a temporary redirect or to 301 for a permanent redirect.

The most common way to use redirect is to redirect to other pages in the same app and (optionally) pass parameters:

``

redirect(URL('index', args=(1,2,3), vars=dict(a='b')))

``:code

In chapter 12 we discuss web2py components. They make Ajax requests to web2py actions. If the called action performs a redirect, you may want the Ajax request to follow the redirect or you may want the entire page performing the Ajax request redirecting. In this latter case you can set:

``

``:code

-------

-------

-------

-------

For example, in Apache, change this:

``

AliasMatch ^/([^/]+)/static/(.*) \

/home/www-data/web2py/applications/$1/static/$2

``

into this:

``

/home/www-data/web2py/applications/$1/static/$2

``

For example, in Apache, change this:

``

AliasMatch ^/([^/]+)/static/(.*) \

/home/www-data/web2py/applications/$1/static/$2

``

into this:

``

/home/www-data/web2py/applications/$1/static/$2

``

``cache view``:inxx

``

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)

def cache_controller_and_view():

import time

t = time.ctime()

d = dict(time=t, link=A('click to reload', _href=request.url))

return response.render(d)

``:code

``response.render(d)`` returns the rendered view as a string, which is now cached for 5 seconds. This is the best and fastest way of caching.

------

------

Note, ``time_expire`` is used to compare the current time with the time the requested object was last saved in the cache. It does not affect future requests. This enables ``time_expire`` to be set dynamically when an object is requested rather than being fixed when the object is saved. For example:

``

message = cache.ram('message', lambda: 'Hello', time_expire=5)

``:code

Now, suppose the following call is made 10 seconds after the above call:

``

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

It is also possible to define other caching mechanisms such as memcache. Memcache is available via ``gluon.contrib.memcache`` and is discussed in more detail in Chapter 14.

------

Be careful when caching to remeber that caching is usually at the app-level not at the user level. If you need, for example, to cache user specific content, choose a key that includes the user id.

------

Web2py by default assumes that the returned content is not going to be cached, as this reduces the shortcomings of an improper caching of the page client-side.

For example, when you show a form to the user, or a list of records, the web page should not be cached, as other users may have inserted new records on the table you are showing.

Instead, if you are showing to the user a wiki page whose content will never change (or it changes once a week), it is useful to store that page, but it is even more useful

to tell the client that that page is not going to change.

This is accomplished sending out some specific headers along with the page: when the client's browser receives the content, it is stored in the browser's cache and it will not be requested again

to your site. Of course this is a **major** speedup for public-facing sites.

- for setting smart cache headers

- to cache the results accordingly

------

NB: it will do one or another or **both**.

------

The main problem with caching a view with ``@cache(request.env.path_info, time_expire=300, cache_model=cache.ram)`` is that request.env.path_info as a key leads to several problems, e.g.

+ URL vars are not considered

-- you cached the result of ''/app/default/index?**search=foo**'' : for the next 300 seconds ''/app/default/index?**search=bar**'' will return the exact same thing of ''/app/default/index?**search=foo**''

+ User is not considered

-- you user accesses often a page and you choose to cache it.

However, you cached the result of ''/app/default/index'' using request.env.path_info as the key, so another user

will see a page that was not meant for him

-- you cached a page for "Bill", but "Bill" accessed the page from the desktop. Now he tries to access it from its phone: if you prepared a template

for mobile users that is different from the standard one, "Joe" will not see it

+ Language is not considered

-- when you cache the page, if you use T() for some elements, the page will be stored with a fixed translation

+ Method is not considered

-- When you cache a page, you should only cache it if it's a result of a GET operation

+ Status code is not considered

-- When you cached the page for the first time, something went wrong and you returned a nice 404 page.

You don't want to cache errors ^_^

It will by default use smart cache headers to let the browser cache the result: if you pass a cache model to it, it will also figure out the best key automatically,

so different versions of the same page can be stored and retrieved accordingly (e.g. one for English users and one for Spanish ones)

It takes several parameters, with smart defaults:

- time_expire : the usual, defaults to 300 seconds

-- if you pass, e.g., ``cache.ram``, the result will be stored in the cache as well

- prefix : if you want to prefix the auto-generated key (useful for clearing it later with, e.g. ``cache.ram.clear(prefix*)``)

- session : if you want to consider the session, defaults to False

- vars : if you want to consider URL vars, defaults to True

- lang : if you want to consider the language, defaults to True

- user_agent : if you want to consider the user agent, defaults to False

- public : if you want the same page for all the users that will ever access it, defaults to True

- valid_statuses : defaults to None. cache.client will cache only pages requested with a GET method, whose status codes begin with 1,2 or 3.

You can pass a list of status codes (when you want pages to be cached with those statuses, e.g. status_codes=[200] will cache only pages

that resulted in a 200 status code)

- quick : defaults to None, but you can pass a list of initials to set a particular feature:

-- **S**ession, **V**ars, **L**ang, **U**ser_agent, **P**ublic

``cache view``:inxx

``

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)

def cache_controller_and_view():

import time

t = time.ctime()

d = dict(time=t, link=A('click to reload', _href=request.url))

return response.render(d)

``:code

``response.render(d)`` returns the rendered view as a string, which is now cached for 5 seconds. This is the best and fastest way of caching.

------

------

Note, ``time_expire`` is used to compare the current time with the time the requested object was last saved in the cache. It does not affect future requests. This enables ``time_expire`` to be set dynamically when an object is requested rather than being fixed when the object is saved. For example:

``

message = cache.ram('message', lambda: 'Hello', time_expire=5)

``:code

Now, suppose the following call is made 10 seconds after the above call:

``

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

It is also possible to define other caching mechanisms such as memcache. Memcache is available via ``gluon.contrib.memcache`` and is discussed in more detail in Chapter 14.

------

Be careful when caching to remeber that caching is usually at the app-level not at the user level. If you need, for example, to cache user specific content, choose a key that includes the user id.

------

Web2py by default assumes that the returned content is not going to be cached, as this reduces the shortcomings of an improper caching of the page client-side.

For example, when you show a form to the user, or a list of records, the web page should not be cached, as other users may have inserted new records on the table you are showing.

Instead, if you are showing to the user a wiki page whose content will never change (or it changes once a week), it is useful to store that page, but it is even more useful

to tell the client that that page is not going to change.

This is accomplished sending out some specific headers along with the page: when the client's browser receives the content, it is stored in the browser's cache and it will not be requested again

to your site. Of course this is a **major** speedup for public-facing sites.

- for setting smart cache headers

- to cache the results accordingly

------

NB: it will do one or another or **both**.

------

The main problem with caching a view with ``@cache(request.env.path_info, time_expire=300, cache_model=cache.ram)`` is that request.env.path_info as a key leads to several problems, e.g.

+ URL vars are not considered

-- you cached the result of ''/app/default/index?**search=foo**'' : for the next 300 seconds ''/app/default/index?**search=bar**'' will return the exact same thing of ''/app/default/index?**search=foo**''

+ User is not considered

-- you user accesses often a page and you choose to cache it.

However, you cached the result of ''/app/default/index'' using request.env.path_info as the key, so another user

will see a page that was not meant for him

-- you cached a page for "Bill", but "Bill" accessed the page from the desktop. Now he tries to access it from its phone: if you prepared a template

for mobile users that is different from the standard one, "Joe" will not see it

+ Language is not considered

-- when you cache the page, if you use T() for some elements, the page will be stored with a fixed translation

+ Method is not considered

-- When you cache a page, you should only cache it if it's a result of a GET operation

+ Status code is not considered

-- When you cached the page for the first time, something went wrong and you returned a nice 404 page.

You don't want to cache errors ^_^

It will by default use smart cache headers to let the browser cache the result: if you pass a cache model to it, it will also figure out the best key automatically,

so different versions of the same page can be stored and retrieved accordingly (e.g. one for English users and one for Spanish ones)

It takes several parameters, with smart defaults:

- time_expire : the usual, defaults to 300 seconds

-- if you pass, e.g., ``cache.ram``, the result will be stored in the cache as well

- prefix : if you want to prefix the auto-generated key (useful for clearing it later with, e.g. ``cache.ram.clear(prefix*)``)

- session : if you want to consider the session, defaults to False

- vars : if you want to consider URL vars, defaults to True

- lang : if you want to consider the language, defaults to True

- user_agent : if you want to consider the user agent, defaults to False

- public : if you want the same page for all the users that will ever access it, defaults to True

- valid_statuses : defaults to None. cache.client will cache only pages requested with a GET method, whose status codes begin with 1,2 or 3.

You can pass a list of status codes (when you want pages to be cached with those statuses, e.g. status_codes=[200] will cache only pages

that resulted in a 200 status code)

- quick : defaults to None, but you can pass a list of initials to set a particular feature:

-- **S**ession, **V**ars, **L**ang, **U**ser_agent, **P**ublic

``

session.connect(request, response, separate=True)

``:code

``

session.connect(request, response, separate=True)

``:code

### ``response``

``response``:inxx

``response.body``:inxx

``response.cookies``:inxx

``response.download``:inxx

``response.files``:inxx

``response.flash``:inxx

``response.headers``:inxx

``response.meta``:inxx

``response.menu``:inxx

``response.postprocessing``:inxx

``response.render``:inxx

``response.status``:inxx

``response.stream``:inxx

``response.subtitle``:inxx

``response.title``:inxx

``response.toolbar``:inxx

``response.view``:inxx

``response.delimiters``:inxx

``response.js``:inxx

``response.write``:inxx

``response.include_files``:inxx

``response.include_meta``:inxx

``response.optimize_css``:inxx

``response.optimize_js``:inxx

``response._caller``:inxx

``response`` is another instance of the ``Storage`` class. It contains the following:

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.stream(file, chunk_size, request=request, attachment=False, filename=None, headers=None)``: when a controller returns it, web2py streams the file content back to the client in blocks of size ``chunk_size``. The ``request`` parameter is required to use the chunk start in the HTTP header. ``file`` should be a file path (for backward compatibility, it can also be an open file object, but this is not recommended). As noted above, ``response.download`` should be used to retrieve files stored via an upload field. ``response.stream`` can be used in other cases, such as returning a temporary file or StringIO object created by the controller. If ``attachment`` is True, the Content-Disposition header will be set to "attachment", and if ``filename`` is also provided, it will be added to the Content-Disposition header as well (but only when ``attachment`` is True). If not already included in ``response.headers``, the following response headers will be set automatically: Content-Type, Content-Length, Cache-Control, Pragma, and Last-Modified (the latter three are set to allow browser caching of the file). To override any of these automatic header settings, simply set them in ``response.headers`` before calling ``response.stream``.

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.title``: optional parameter that may be included in the views. It should contain the title of the page and should be rendered inside the html title tag in the header.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._vars``: this variable is accessible only in a view, not in the action. It contains the values returned by the action to the view.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

Since ``response`` is a ``gluon.storage.Storage`` object, it can be used to store other attributes that you may want to pass to the view. While there is no technical restriction, our recommendation is to store only variables that are to be rendered by all pages in the overall layout ("layout.html").

Anyway, we strongly suggest to stick to the variables listed here:

``

response.title

To **store sessions in cookies** instead you can do:

``

session.connect(request,response,cookie_key='yoursecret',compression_level=None)

``:code

Here ``cookie_key`` is a symmetric encryption key.

``compression_level`` is an optional ``zlib`` encryption level.

While sessions in cookie are often recommended for scalability reason they are limited in size. Large sessions will result in broken cookies.

You can check the state of your application at any time by printing the ``request``, ``session`` and ``response`` system variables. One way to do it is to create a dedicated action:

``

def status():

return dict(request=request, session=session, response=response)

``:code

In the "generic.html" view this is done using ``{{=response.toolbar()}}``.

#### Separate sessions

``

session.connect(request, response, separate=True)

``:code

By setting ``separate=True`` web2py will store sessions not in the "sessions/" folder but in subfolders of the "sessions/" folder. The subfolder will be created automatically. Sessions with the same prefix will be in the same subfolder. Again, note that the above must be called before any logic that might require the session.

### ``cache``

``cache``:inxx ``cache.ram``:inxx ``cache.disk``:inxx

``cache`` a global object also available in the web2py execution environment. It has two attributes:

- ``cache.ram``: the application cache in main memory.

- ``cache.disk``: the application cache on disk.

``cache`` is callable, this allows it to be used as a decorator for caching actions and views.

The following example caches the ``time.ctime()`` function in RAM:

``

def cache_in_ram():

import time

t = cache.ram('time', lambda: time.ctime(), time_expire=5)

return dict(time=t, link=A('click me', _href=request.url))

``:code

The ``args`` attributes are automatically parsed, decoded, and finally stored in

If args contains only one element, there is no need to pass it in a list.

You can also use the ``URL`` function to generate URLs to actions in other controllers and other applications:

``

URL('a', 'c', 'f', args=['x', 'y'], vars=dict(z='t'))

``:code

is mapped into

``

/a/c/f/x/y?z=t

``

It is also possible to specify application, controller and function using named arguments:

``

URL(a='a', c='c', f='f')

``:code

``

URL('c', 'f')

``:code

``

URL('f')

``:code

Instead of passing the name of a controller function it is also possible to pass the function itself

``

URL(f)

``:code

For the reasons mentioned above, you should always use the ``URL`` function to generate URLs of static files for your applications. Static files are stored in the application's ``static`` subfolder (that's where they go when uploaded using the administrative interface). web2py provides a virtual 'static' controller whose job is to retrieve files from the ``static`` subfolder, determine their content-type, and stream the file to the client. The following example generates the URL for the static file "image.png":

``

URL('static', 'image.png')

``:code

is mapped into

``

where:

<modifier>::= ! | !! | !!!

<word> ::= any word or phrase in singular in lower case (!)

<parameter> ::= [index] | (key) | (number)

``

For example:

- ``%%{word}`` is equivalent to ``%%{word[0]}`` (if no modifiers are used).

- ``%%{word[index]}`` is used when symbols is a tuple. symbols[index] gives us a number used to make a decision on which word form to choose.

- ``%%{word(key)}`` is used to get the numeric parameter from symbols[key]

- ``%%{word(number)}`` allows to set a ``number`` directly (e.g.: ``%%{word(%i)}``)

- ``%%{?word?number}`` returns "word" if ``number==1``, returns the ``number`` otherwise

- ``%%{?number} or %%{??number}`` returns ``number`` if ``number!=1``, return nothing otherwise

``T("blabla %s %%{word}", symbols=var)``

``%%{word}`` by default means ``%%{word[0]}``,

where ``[0]`` is an item index in symbols tuple.

``T("blabla %s %s %%{word[1]}", (var1, var2))``

You can use several ``%%{}`` placeholders with one index:

``T("%%{this} %%{is} %s %%{book}", var)``

or

``T("%%{this[0]} %%{is[0]} %s %%{book[0]}", var)``

They generate:

``

var output

------------------

1 this is 1 book

2 these are 2 books

3 these are 2 books

``

Similarly you can pass a dictionary to symbols:

does the exact same thing as

scheduler.queue_task('demo1', pvars={'a':1, 'b':2})

``:code

as

``

st.validate_and_insert(function_name='demo1', args=json.dumps([1,2]))

``:code

and as:

``

st.validate_and_insert(function_name='demo1', vars=json.dumps({'a':1,'b':2}))

``:code

Here is a more complex complete example:

``

def task_add(a,b):

return a+b

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

A call to ``scheduler.queue_task`` returns the task ``id`` and ``uuid`` of the task you queued (can be the one you passed or the auto-generated one), and possible ``errors``:

``

<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>

``

you get the result of the validation, and id and uuid will be None

``

<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>

``

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

### ``response``

``response``:inxx

``response.body``:inxx

``response.cookies``:inxx

``response.download``:inxx

``response.files``:inxx

``response.flash``:inxx

``response.headers``:inxx

``response.meta``:inxx

``response.menu``:inxx

``response.postprocessing``:inxx

``response.render``:inxx

``response.status``:inxx

``response.stream``:inxx

``response.subtitle``:inxx

``response.title``:inxx

``response.toolbar``:inxx

``response.view``:inxx

``response.delimiters``:inxx

``response.js``:inxx

``response.write``:inxx

``response.include_files``:inxx

``response.include_meta``:inxx

``response.optimize_css``:inxx

``response.optimize_js``:inxx

``response._caller``:inxx

``response`` is another instance of the ``Storage`` class. It contains the following:

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.stream(file, chunk_size, request=request, attachment=False, filename=None, headers=None)``: when a controller returns it, web2py streams the file content back to the client in blocks of size ``chunk_size``. The ``request`` parameter is required to use the chunk start in the HTTP header. ``file`` should be a file path (for backward compatibility, it can also be an open file object, but this is not recommended). As noted above, ``response.download`` should be used to retrieve files stored via an upload field. ``response.stream`` can be used in other cases, such as returning a temporary file or StringIO object created by the controller. If ``attachment`` is True, the Content-Disposition header will be set to "attachment", and if ``filename`` is also provided, it will be added to the Content-Disposition header as well (but only when ``attachment`` is True). If not already included in ``response.headers``, the following response headers will be set automatically: Content-Type, Content-Length, Cache-Control, Pragma, and Last-Modified (the latter three are set to allow browser caching of the file). To override any of these automatic header settings, simply set them in ``response.headers`` before calling ``response.stream``.

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.title``: optional parameter that may be included in the views. It should contain the title of the page and should be rendered inside the html title tag in the header.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._vars``: this variable is accessible only in a view, not in the action. It contains the values returned by the action to the view.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

Since ``response`` is a ``gluon.storage.Storage`` object, it can be used to store other attributes that you may want to pass to the view. While there is no technical restriction, our recommendation is to store only variables that are to be rendered by all pages in the overall layout ("layout.html").

Anyway, we strongly suggest to stick to the variables listed here:

``

response.title

To **store sessions in cookies** instead you can do:

``

session.connect(request,response,cookie_key='yoursecret',compression_level=None)

``:code

Here ``cookie_key`` is a symmetric encryption key.

``compression_level`` is an optional ``zlib`` encryption level.

While sessions in cookie are often recommended for scalability reason they are limited in size. Large sessions will result in broken cookies.

You can check the state of your application at any time by printing the ``request``, ``session`` and ``response`` system variables. One way to do it is to create a dedicated action:

``

def status():

return dict(request=request, session=session, response=response)

``:code

In the "generic.html" view this is done using ``{{=response.toolbar()}}``.

#### Separate sessions

``

session.connect(request, response, separate=True)

``:code

By setting ``separate=True`` web2py will store sessions not in the "sessions/" folder but in subfolders of the "sessions/" folder. The subfolder will be created automatically. Sessions with the same prefix will be in the same subfolder. Again, note that the above must be called before any logic that might require the session.

### ``cache``

``cache``:inxx ``cache.ram``:inxx ``cache.disk``:inxx

``cache`` a global object also available in the web2py execution environment. It has two attributes:

- ``cache.ram``: the application cache in main memory.

- ``cache.disk``: the application cache on disk.

``cache`` is callable, this allows it to be used as a decorator for caching actions and views.

The following example caches the ``time.ctime()`` function in RAM:

``

def cache_in_ram():

import time

t = cache.ram('time', lambda: time.ctime(), time_expire=5)

return dict(time=t, link=A('click me', _href=request.url))

``:code

The ``args`` attributes are automatically parsed, decoded, and finally stored in

If args contains only one element, there is no need to pass it in a list.

You can also use the ``URL`` function to generate URLs to actions in other controllers and other applications:

``

URL('a', 'c', 'f', args=['x', 'y'], vars=dict(z='t'))

``:code

is mapped into

``

/a/c/f/x/y?z=t

``

It is also possible to specify application, controller and function using named arguments:

``

URL(a='a', c='c', f='f')

``:code

``

URL('c', 'f')

``:code

``

URL('f')

``:code

Instead of passing the name of a controller function it is also possible to pass the function itself

``

URL(f)

``:code

For the reasons mentioned above, you should always use the ``URL`` function to generate URLs of static files for your applications. Static files are stored in the application's ``static`` subfolder (that's where they go when uploaded using the administrative interface). web2py provides a virtual 'static' controller whose job is to retrieve files from the ``static`` subfolder, determine their content-type, and stream the file to the client. The following example generates the URL for the static file "image.png":

``

URL('static', 'image.png')

``:code

is mapped into

``

where:

<modifier>::= ! | !! | !!!

<word> ::= any word or phrase in singular in lower case (!)

<parameter> ::= [index] | (key) | (number)

``

For example:

- ``%%{word}`` is equivalent to ``%%{word[0]}`` (if no modifiers are used).

- ``%%{word[index]}`` is used when symbols is a tuple. symbols[index] gives us a number used to make a decision on which word form to choose.

- ``%%{word(key)}`` is used to get the numeric parameter from symbols[key]

- ``%%{word(number)}`` allows to set a ``number`` directly (e.g.: ``%%{word(%i)}``)

- ``%%{?word?number}`` returns "word" if ``number==1``, returns the ``number`` otherwise

- ``%%{?number} or %%{??number}`` returns ``number`` if ``number!=1``, return nothing otherwise

``T("blabla %s %%{word}", symbols=var)``

``%%{word}`` by default means ``%%{word[0]}``,

where ``[0]`` is an item index in symbols tuple.

``T("blabla %s %s %%{word[1]}", (var1, var2))``

You can use several ``%%{}`` placeholders with one index:

``T("%%{this} %%{is} %s %%{book}", var)``

or

``T("%%{this[0]} %%{is[0]} %s %%{book[0]}", var)``

They generate:

``

var output

------------------

1 this is 1 book

2 these are 2 books

3 these are 2 books

``

Similarly you can pass a dictionary to symbols:

does the exact same thing as

scheduler.queue_task('demo1', pvars={'a':1, 'b':2})

``:code

as

``

st.validate_and_insert(function_name='demo1', args=json.dumps([1,2]))

``:code

and as:

``

st.validate_and_insert(function_name='demo1', vars=json.dumps({'a':1,'b':2}))

``:code

Here is a more complex complete example:

``

def task_add(a,b):

return a+b

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

A call to ``scheduler.queue_task`` returns the task ``id`` and ``uuid`` of the task you queued (can be the one you passed or the auto-generated one), and possible ``errors``:

``

<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>

``

you get the result of the validation, and id and uuid will be None

``

<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>

``

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.files``: a list of ``.css``, ``.js``, ``coffee``, and ``.less`` files required by the page. They will automatically be linked in the head of the standard "layout.html" via the included "web2py_ajax.html". To include a new CSS, JS, COFFEE, or LESS file, just append it to this list. It will handle duplicates. The order is important.

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g.``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.stream(file, chunk_size, request=request, attachment=False, filename=None, headers=None)``: when a controller returns it, web2py streams the file content back to the client in blocks of size ``chunk_size``. The ``request`` parameter is required to use the chunk start in the HTTP header. ``file`` should be a file path (for backward compatibility, it can also be an open file object, but this is not recommended). As noted above, ``response.download`` should be used to retrieve files stored via an upload field. ``response.stream`` can be used in other cases, such as returning a temporary file or StringIO object created by the controller. If ``attachment`` is True, the Content-Disposition header will be set to "attachment", and if ``filename`` is also provided, it will be added to the Content-Disposition header as well (but only when ``attachment`` is True). If not already included in ``response.headers``, the following response headers will be set automatically: Content-Type, Content-Length, Cache-Control, Pragma, and Last-Modified (the latter three are set to allow browser caching of the file). To override any of these automatic header settings, simply set them in ``response.headers`` before calling ``response.stream``.

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

Since ``response`` is a ``gluon.storage.Storage`` object, it can be used to store other attributes that you may want to pass to the view. While there is no technical restriction, our recommendation is to store only variables that are to be rendered by all pages in the overall layout ("layout.html").

Anyway, we strongly suggest to stick to the variables listed here:

``

response.title

To **store sessions in cookies** instead you can do:

``

session.connect(request,response,cookie_key='yoursecret',compression_level=None)

``:code

Here ``cookie_key`` is a symmetric encryption key.

``compression_level`` is an optional ``zlib`` encryption level.

While sessions in cookie are often recommended for scalability reason they are limited in size. Large sessions will result in broken cookies.

You can check the state of your application at any time by printing the ``request``, ``session`` and ``response`` system variables. One way to do it is to create a dedicated action:

``

def status():

return dict(request=request, session=session, response=response)

``:code

In the "generic.html" view this is done using ``{{=response.toolbar()}}``.

#### Separate sessions

``

session.connect(request, response, separate=True)

``:code

By setting ``separate=True`` web2py will store sessions not in the "sessions/" folder but in subfolders of the "sessions/" folder. The subfolder will be created automatically. Sessions with the same prefix will be in the same subfolder. Again, note that the above must be called before any logic that might require the session.

### ``cache``

``cache``:inxx ``cache.ram``:inxx ``cache.disk``:inxx

``cache`` a global object also available in the web2py execution environment. It has two attributes:

- ``cache.ram``: the application cache in main memory.

- ``cache.disk``: the application cache on disk.

``cache`` is callable, this allows it to be used as a decorator for caching actions and views.

The following example caches the ``time.ctime()`` function in RAM:

``

def cache_in_ram():

import time

t = cache.ram('time', lambda: time.ctime(), time_expire=5)

return dict(time=t, link=A('click me', _href=request.url))

``:code

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

------

Be careful when caching to remeber that caching is usually at the app-level not at the user level. If you need, for example, to cache user specific content, choose a key that includes the user id.

------

#### ``cache.client``

Web2py by default assumes that the returned content is not going to be cached, as this reduces the shortcomings of an improper caching of the page client-side.

For example, when you show a form to the user, or a list of records, the web page should not be cached, as other users may have inserted new records on the table you are showing.

Instead, if you are showing to the user a wiki page whose content will never change (or it changes once a week), it is useful to store that page, but it is even more useful

to tell the client that that page is not going to change.

This is accomplished sending out some specific headers along with the page: when the client's browser receives the content, it is stored in the browser's cache and it will not be requested again

to your site. Of course this is a **major** speedup for public-facing sites.

Web2py 2.4.4 introduced a new ``cache.client`` decorator to allow a smarter handling of this situation.

``cache.client`` can be used:

- for setting smart cache headers

- to cache the results accordingly

redirect(...,type='auto')

``T``:inxx ``internationalization``:inxx

The object ``T`` is the language translator. It constitutes a single global instance of the web2py class ``gluon.language.translator``. All string constants (and only string constants) should be marked by ``T``, for example:

``

a = T("hello world")

``:code

Strings that are marked with ``T`` are identified by web2py as needing language translation and they will be translated when the code (in the model, controller, or view) is executed. If the string to be translated is not a constant but a variable, it will be added to the translation file at runtime (except on GAE) to be translated later.

The ``T`` object can also contain interpolated variables and supports multiple equivalent syntaxes:

``

a = T("hello %s", ('Tim',))

a = T("hello %(name)s", dict(name='Tim'))

a = T("hello %s") % ('Tim',)

a = T("hello %(name)s") % dict(name='Tim')

``:code

The latter syntax is recommended because it makes translation easier.

The first string is translated according to the requested language file and the ``name`` variable is replaced independently of the language.

``

T("blah ") + name + T(" blah")

``:code

The following code is also allowed and often preferable:

``

T("blah %(name)s blah", dict(name='Tim'))

``:code

or the alternative syntax

``

T("blah %(name)s blah") % dict(name='Tim')

``:code

In both cases the translation occurs before the variable name is substituted in the "%(name)s" slot. The following alternative should NOT BE USED:

``

T("blah %(name)s blah" % dict(name='Tim'))

``:code

You can also force a per-string language:

T("Hello World", language="it-it")

``:code

--------------

In the case multiple languages are requested, for example "it-it, fr-ft", web2py tries to locate "it-it.py" and "fr-fr.py" translation files. If none of the requested files is present, it tries to fall back on "it.py" and "fr.py". If these files are not present it defaults to "default.py". If this is not present either, it default to no-translation. The more general rule is that web2py tries "xx-xy-yy.py", "xx-xy.py", "xx.py", "default.py" for each of the "xx-xy-yy" accepted languages trying to find the closest match to the visitor's preferences.

-------------

You can turn off translations completely via

``

T.force(None)

``:code

Normally, string translation is evaluated lazily when the view is rendered; hence, the translator ``force`` method should not be called inside a view.

It is possible to disable lazy evaluation via

``

T.lazy = False

``:code

It is also possible to disable lazy evaluation for individual strings:

``

T("Hello World", lazy=False)

``:code

A common issue is the following. The original application is in English. Suppose that there is a translation file (for example Italian, "it-it.py") and the HTTP client declares that it accepts both English (en) and Italian (it-it) in that order. The following unwanted situation occurs: web2py does not know the default is written in English (en). Therefore, it prefers translating everything into Italian (it-it) because it only found the Italian translation file. If it had not found the "it-it.py" file, it would have used the default language strings (English).

There are two solutions for this problem: create a translation language for English, which would be redundant and unnecessary, or better, tell web2py which languages should use the default language strings (the strings coded into the application). This can be done with:

``

T.set_current_languages('en', 'en-en')

``:code

It stores in ``T.current_languages`` a list of languages that do not require translation and forces a reload of the language files.

Notice that "it" and "it-it" are different languages from the point of view of web2py. To support both of them, one would need two translation files, always lower case. The same is true for all other languages.

The currently accepted language is stored in

``

x book(s)

with

``

a book (x==1)

5 books (x==5)

``

English has one singular form and one plural form. The plural form is constructed by adding a "-s" or "-es" or using an exceptional form. web2py provides a way to define pluralization rules for each languages, as well as exceptions to the default rules. In fact web2py already knows pluralization rules for many languages. It knows, for example, that Slovenian has one singular form and 3 plural forms (for x==1, x==3 or x==4 and x>4). These rules are encoded in "gluon/contrib/plural_rules/*.py" files and new files can be created. Explicit pluralizations for words are created by editing pluralization files using the administrative interface.

By default the PS is not activated. It is triggered by the ``symbol`` argument of the ``T`` function. For example:

``

T("You have %s %%{book}", symbols=10)

``:code

Now the PS is activated for the word "book" and for the number 10.

The result in English will be: "You have 10 books". Notice that "book" has been pluralized into "books".

The PS consists of 3 parts:

- rule to give a decision which word form to use ("rules/plural_rules/*.py")

- dictionary with word plural forms ("app/languages/plural-*.py")

The value of symbols can be a single variable, a list/tuple of variables, or a dictionary.

The placeholder ``%%{}`` consists of 3 parts:

``

``

where:

``

<modifier>::= ! | !! | !!!

<word> ::= any word or phrase in singular in lower case (!)

<parameter> ::= [index] | (key) | (number)

``

For example:

- ``%%{word}`` is equivalent to ``%%{word[0]}`` (if no modifiers are used).

- ``%%{word[index]}`` is used when symbols is a tuple. symbols[index] gives us a number used to make a decision on which word form to choose.

- ``%%{word(key)}`` is used to get the numeric parameter from symbols[key]

- ``%%{word(number)}`` allows to set a ``number`` directly (e.g.: ``%%{word(%i)}``)

- ``%%{?word?number}`` returns "word" if ``number==1``, returns the ``number`` otherwise

- ``%%{?number} or %%{??number}`` returns ``number`` if ``number!=1``, return nothing otherwise

``T("blabla %s %%{word}", symbols=var)``

error_handler = dict(application='error',

If the ``error_handler`` is specified the action is called without user redirection and the handler action will be in charge of dealing with the error. In the event that the error-handling page itself returns an error, web2py will fall back to its old static responses.

#### Static asset management

Since version 2.1.0, web2py has the ability to manage static assets.

When an application is in development, static file can change often, therefore web2py sends static files with no cache headers. This has the side-effect of "forcing" the browser to request static files at every request. This results in low performance when loading the page.

In a "production" site, you may want to serve static files with ``cache`` headers to prevent un-necessary downloads since static files do not change.

``cache`` headers allow the browser to fetch each file only once, thus saving bandwidth and reducing loading time.

Yet there is a problem: What should the cache headers declare? When should the files expire? When the files are first served, the server cannot forecast when they will be changed.

A manual approach consists of creating subfolders for different versions of static files. For example an early version of "layout.css" can be made available at the URL "/myapp/static/css/1.2.3/layout.css". When you change the file, you create a new subfolder and you link it as "/myapp/static/css/1.2.4/layout.css".

This procedure works but it is pedantic since every time you update the css file, you must remember to move it to another folder, change the URL of the file in your layout.html and deploy.

If you want to serve "/myapp/static/layout.css" with the cache headers, you just need to include the file with a modified URL that includes a version number:

``

/myapp/static/_1.2.3/layout.css

``

(notice the URL defines a version number, it does not appear anywhere else).

Notice that the URL starts with "/myapp/static/", followed by a version number composed by an underscore and 3 integers separated by a period (as described in [[SemVer http://semver.org/]]), then followed by the filename. Also notice that you do not have to create a "_1.2.3/" folder.

Every time the static file is requested with a version in the url, it will be served with "far in the future" cache headers, specifically:

``

Cache-Control : max-age=315360000

Expires: Thu, 31 Dec 2037 23:59:59 GMT

``

This means that the browser will fetch those files only once, and they will be saved "forever" in the browser's cache.

Every time the "_1.2.3/filename" is requested, web2py will remove the version part from the path and serve your file with far in the future headers so they will be cached forever. If you changed the version number in the URL, this tricks the browser into thinking it is requesting a different file, and the file is fetched again.

You can use "_1.2.3", "_0.0.0", "_999.888.888", as long as the version starts with underscore followed by three numbers separated by period.

By **cron** we refer to a web2py functionality not to the Unix Cron mechanism. T

web2py cron is the way to go if you need tasks in the background at scheduled times and these tasks take a relatively short time compared to the time interval between two calls. Each task runs in its own process, and multiple tasks can run concurrently, but you have no control over how many tasks run. If accidentally one task overlaps with itself, it can cause a database lock and a spike in memory usage.

web2py scheduler takes a different approach. The number of running processes is fixed, and they can run on different machines. Each process is called a worker. Each worker picks a task when available and executes it as soon as possible after the time when it is scheduled to run, but not necessarily at that exact time. There cannot be more processes running than the number of scheduled tasks and therefore no memory spikes. Scheduler tasks can be defined in models and are stored in the database. The web2py scheduler does not implement a distributed queue since it assumes that the time to distribute tasks is negligible compared with the time to run the tasks. Workers pick up the task from the database.

Homemade tasks queues can be a simpler alternative to the web2py scheduler in some cases.

#### Cron

``cron``:inxx

The web2py cron provides the ability for applications to execute tasks at preset times, in a platform-independent manner.

For each application, cron functionality is defined by a crontab file:

``

app/cron/crontab

``

It follows the syntax defined in ref. ``cron``:cite (with some extensions that are specific to web2py).

------

------

This means that every application can have a separate cron configuration and that cron config can be changed from within web2py without affecting the host OS itself.

Here is an example:

``

0-59/1 * * * * root python /path/to/python/script.py

30 3 * * * root *applications/admin/cron/db_vacuum.py

*/30 * * * * root **applications/admin/cron/something.py

@reboot root *mycontroller/myfunction

@hourly root *applications/admin/cron/expire_sessions.py

``:code

The last two lines in this example use extensions to regular cron syntax to provide additional web2py functionality.

-------

The file "applications/admin/cron/expire_sessions.py" actually exists and ships with the **admin** app. It checks for expired sessions and deletes them. "applications/admin/cron/crontab" runs this task hourly.

-------

Notice that scripts/functions executed in the web2py environment require a manual ``db.commit()`` at the end of the function or the transaction will be reverted.

web2py does not generate tickets or meaningful tracebacks in shell mode, which is how cron is run, so make sure that your web2py code runs without errors before you set it up as a cron task as you will likely not be able to see those errors when run from cron. Moreover, be careful how you use models: while the execution happens in a separate process, database locks have to be taken into account in order to avoid pages waiting for cron tasks that may be blocking the database. Use the ``**`` syntax if you don't need to use the database in your cron task.

You can also call a controller function, in which case there is no need to specify a path. The controller and function will be that of the invoking application. Take special care about the caveats listed above. Example:

``

*/30 * * * * root *mycontroller/myfunction

``:code

If you specify ``@reboot`` in the first field in the crontab file, the given task will be executed only once, at web2py startup. You can use this feature if you want to pre-cache, check, or initialize data for an application on web2py startup. Note that cron tasks are executed in parallel with the application --- if the application is not ready to serve requests until the cron task is finished, you should implement checks to reflect this. Example:

``

@reboot * * * * root *mycontroller/myfunction

``:code

Depending on how you are invoking web2py, there are four modes of operation for web2py cron.

- ''soft cron'': available under all execution modes

- ''hard cron'': available if using the built-in web server (either directly or via Apache mod_proxy)

- ''external cron'': available if you have access to the system's own cron service

- No cron

Scheduler(

migrate=True,

worker_name=None,

group_names=None,

heartbeat=HEARTBEAT,

max_empty_runs=0,

discard_results=False,

utc_time=False

)

``:code

Let's see them in order:

- ``db`` is the database DAL instance where you want the scheduler tables be placed.

- ``tasks`` is a dictionary that maps task names into functions. If you do not pass this parameter, function will be searched in the app environment.

- ``worker_name`` is None by default. As soon as the worker is started, a worker name is generated as hostname-uuid. If you want to specify that, be sure that it's unique.

- ``group_names`` is by default set to **[main]**. All tasks have a ``group_name`` parameter, set to **main** by default. Workers can only pick up tasks of their assigned group.

------

NB: This is useful if you have different workers instances (e.g. on different machines) and you want to assign tasks to a specific worker.

------

- ``heartbeat`` is by default set to 3 seconds. This parameter is the one controlling how often a scheduler will check its status on the ``scheduler_worker`` table and see if there are any **ASSIGNED** tasks to itself to process.

- ``max_empty_runs`` is 0 by default, that means that the worker will continue to process tasks as soon as they are **ASSIGNED**. If you set this to a value of, let's say, 10, a worker will die automatically if it's **ACTIVE** and no tasks are **ASSIGNED** to it for 10 loops. A loop is when a worker searches for tasks, every 3 seconds (or the set ``heartbeat``)

- ``discard_results`` is False by default. If set to True, no scheduler_run records will be created.

------

NB: scheduler_run records will be created as before for **FAILED**, **TIMEOUT** and **STOPPED** tasks's statuses.

------

- ``utc_time`` is False by default. If you need to coordinate with workers living in different timezones, or don't have problems with solar/DST times, supplying datetimes from different countries, etc, you can set this to True. The scheduler will honor the UTC time and work leaving the local time aside. Caveat: you need to schedule tasks with UTC times (for start_time, stop_time, and so on.)

Now we have the infrastructure in place: defined the tasks, told the scheduler about them, started the worker(s). What remains is to actually schedule the tasks

##### Tasks

Tasks can be scheduled programmatically or via appadmin. In fact, a task is scheduled simply by adding an entry in the table "scheduler_task", which you can access via appadmin:

``

http://127.0.0.1:8000/myapp/appadmin/insert/db/scheduler_task

``

st.validate_and_insert(function_name='demo1', vars=json.dumps({'a':1,'b':2}))

Here is a more complex complete example:

``

def task_add(a,b):

return a+b

secheduler = Scheduler(db, tasks=dict(demo1=task_add))

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

A call to ``scheduler.queue_task`` returns the task ``id`` and ``uuid`` of the task you queued (can be the one you passed or the auto-generated one), and possible ``errors``:

``

<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>

``

you get the result of the validation, and id and uuid will be None

``

<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>

``

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

##### Managing processes

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

When you start a worker, you may later want to:

- kill it "no matter what it's doing"

- kill it only if it is not processing tasks

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

To be more precise, for started workers you can change the ``status`` value of any worker to influence

its behavior.

As for tasks, workers can be in one of the following statuses: ACTIVE, DISABLED, TERMINATE or KILLED.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

scheduler.queue_task(

timeout = 120, # should take less than 120 seconds

)

``:code

Notice that fields "times_run", "last_run_time" and "assigned_worker_name" are not provided at schedule time but are filled automatically by the workers.

You can also retrieve the output of completed tasks:

``

completed_runs = db(db.scheduler_run.run_status='COMPLETED').select()

``:code

------

The scheduler is considered experimental because it needs more extensive testing and because the table structure may change as more features are added.

------

##### Reporting percentages

A special "word" encountered in the print statements of your functions clear all

the previous output. That word is ``!clear!``.

Here is an example:

``

def reporting_percentages():

time.sleep(5)

print '50%'

time.sleep(5)

print '!clear!100%'

return 1

``

The function ``reporting_percentages`` sleeps for 5 seconds, outputs ``50%``.

Then, it sleeps other 5 seconds and outputs ``100%``. Note that the output in the scheduler_run table is synced every 2 seconds and that the second print statement that contains ``!clear!100%`` gets the ``50%`` output cleared and replaced by ``100%`` only.

``

scheduler.queue_task(reporting_percentages,

sync_output=2)

``:code

result = other.some_action()

``:code

In line 2, ``request=request`` is optional. It has the effect of passing the current request to the environment of "other". Without this argument, the environment would contain a new and empty (apart from ``request.folder``) request object. It is also possible to pass a response and a session object to ``exec_environment``. Be careful when passing request, response and session objects --- modification by the called action or coding dependencies in the called action could lead to unexpected side effects.

The function call in line 3 does not execute the view; it simply returns the dictionary unless ``response.render`` is called explicitly by "some_action".

One final caution: don't use ``exec_environment`` inappropriately. If you want the results of actions in another application, you probably should implement an XML-RPC API (implementing an XML-RPC API with web2py is almost trivial). Don't use ``exec_environment`` as a redirection mechanism; use the ``redirect`` helper.

### Cooperation

``cooperation``:inxx

There are many ways applications can cooperate:

- Applications can connect to the same database and thus share tables. It is not necessary that all tables in the database are defined by all applications, but they must be defined by those applications that use them. All applications that use the same table, but one, must define the table with ``migrate=False``.

- Applications can embed components from other applications using the LOAD helper (described in Chapter 12).

- Applications can share sessions.

- Applications can call each other's actions remotely via XML-RPC.

- Applications can access each other's files via the filesystem (assuming they share the same filesystem).

- Applications can call each other's actions locally using ``exec_environment`` as discussed above.

- Applications can import each other's modules using the syntax:

``

``:code

- Applications can import any module in the ``PYTHONPATH`` search path, ``sys.path``.

One app can load the session of another app using the command:

``

session.connect(request, response, masterapp='appname', db=db)

``:code

Here "appname" is the name of the master application, the one that sets the initial session_id in the cookie. ``db`` is a database connection to the database that contains the session table (``web2py_session``). All apps that share sessions must use the same database for session storage.

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.files``: a list of ``.css``, ``.js``, ``coffee``, and ``.less`` files required by the page. They will automatically be linked in the head of the standard "layout.html" via the included "web2py_ajax.html". To include a new CSS, JS, COFFEE, or LESS file, just append it to this list. It will handle duplicates. The order is important.

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g.``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.meta``: a Storage object that contains optional ``<meta>`` information like ``response.meta.author``, ``.description``, and/or ``.keywords``. The content of each meta variable is automatically placed in the proper ``META`` tag by the code in "views/web2py_ajax.html", which is included by default in "views/layout.html".

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.stream(file, chunk_size, request=request, attachment=False, filename=None, headers=None)``: when a controller returns it, web2py streams the file content back to the client in blocks of size ``chunk_size``. The ``request`` parameter is required to use the chunk start in the HTTP header. ``file`` should be a file path (for backward compatibility, it can also be an open file object, but this is not recommended). As noted above, ``response.download`` should be used to retrieve files stored via an upload field. ``response.stream`` can be used in other cases, such as returning a temporary file or StringIO object created by the controller. If ``attachment`` is True, the Content-Disposition header will be set to "attachment", and if ``filename`` is also provided, it will be added to the Content-Disposition header as well (but only when ``attachment`` is True). If not already included in ``response.headers``, the following response headers will be set automatically: Content-Type, Content-Length, Cache-Control, Pragma, and Last-Modified (the latter three are set to allow browser caching of the file). To override any of these automatic header settings, simply set them in ``response.headers`` before calling ``response.stream``.

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.view``: the name of the view template that must render the page. This is set by default to:

``

"%s/%s.%s" % (request.controller, request.function, request.extension)

``:code

or, if the above file cannot be located, to

``

"generic.%s" % (request.extension)

``:code

Change the value of this variable to modify the view file associated with a particular action.

- ``response.delimiters`` defaults to ``('{{','}}')``. It allows you to change the delimiter of code embedded in views.

- ``response.xmlrpc(request, methods)``: when a controller returns it, this function exposes the methods via XML-RPC``xmlrpc``:cite . This function is deprecated since a better mechanism is available and described in Chapter 10.

- ``response.write(text)``: a method to write text into the output page body.

- ``response.js`` can contain Javascript code. This code will be executed if and only if the response is received by a web2py component as discussed in Chapter 12.

Since ``response`` is a ``gluon.storage.Storage`` object, it can be used to store other attributes that you may want to pass to the view. While there is no technical restriction, our recommendation is to store only variables that are to be rendered by all pages in the overall layout ("layout.html").

Anyway, we strongly suggest to stick to the variables listed here:

``

response.title

To **store sessions in cookies** instead you can do:

``

session.connect(request,response,cookie_key='yoursecret',compression_level=None)

``:code

Here ``cookie_key`` is a symmetric encryption key.

``compression_level`` is an optional ``zlib`` encryption level.

While sessions in cookie are often recommended for scalability reason they are limited in size. Large sessions will result in broken cookies.

You can check the state of your application at any time by printing the ``request``, ``session`` and ``response`` system variables. One way to do it is to create a dedicated action:

``

def status():

return dict(request=request, session=session, response=response)

``:code

In the "generic.html" view this is done using ``{{=response.toolbar()}}``.

#### Separate sessions

``

session.connect(request, response, separate=True)

``:code

By setting ``separate=True`` web2py will store sessions not in the "sessions/" folder but in subfolders of the "sessions/" folder. The subfolder will be created automatically. Sessions with the same prefix will be in the same subfolder. Again, note that the above must be called before any logic that might require the session.

### ``cache``

``cache``:inxx ``cache.ram``:inxx ``cache.disk``:inxx

``cache`` a global object also available in the web2py execution environment. It has two attributes:

- ``cache.ram``: the application cache in main memory.

- ``cache.disk``: the application cache on disk.

``cache`` is callable, this allows it to be used as a decorator for caching actions and views.

The following example caches the ``time.ctime()`` function in RAM:

``

def cache_in_ram():

import time

t = cache.ram('time', lambda: time.ctime(), time_expire=5)

return dict(time=t, link=A('click me', _href=request.url))

``:code

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

------

Be careful when caching to remeber that caching is usually at the app-level not at the user level. If you need, for example, to cache user specific content, choose a key that includes the user id.

------

#### ``cache.client``

Web2py by default assumes that the returned content is not going to be cached, as this reduces the shortcomings of an improper caching of the page client-side.

For example, when you show a form to the user, or a list of records, the web page should not be cached, as other users may have inserted new records on the table you are showing.

Instead, if you are showing to the user a wiki page whose content will never change (or it changes once a week), it is useful to store that page, but it is even more useful

to tell the client that that page is not going to change.

This is accomplished sending out some specific headers along with the page: when the client's browser receives the content, it is stored in the browser's cache and it will not be requested again

to your site. Of course this is a **major** speedup for public-facing sites.

Web2py 2.4.4 introduced a new ``cache.client`` decorator to allow a smarter handling of this situation.

``cache.client`` can be used:

- for setting smart cache headers

- to cache the results accordingly

redirect(...,type='auto')

``T``:inxx ``internationalization``:inxx

The object ``T`` is the language translator. It constitutes a single global instance of the web2py class ``gluon.language.translator``. All string constants (and only string constants) should be marked by ``T``, for example:

``

a = T("hello world")

``:code

Strings that are marked with ``T`` are identified by web2py as needing language translation and they will be translated when the code (in the model, controller, or view) is executed. If the string to be translated is not a constant but a variable, it will be added to the translation file at runtime (except on GAE) to be translated later.

The ``T`` object can also contain interpolated variables and supports multiple equivalent syntaxes:

``

a = T("hello %s", ('Tim',))

a = T("hello %(name)s", dict(name='Tim'))

a = T("hello %s") % ('Tim',)

a = T("hello %(name)s") % dict(name='Tim')

``:code

The latter syntax is recommended because it makes translation easier.

The first string is translated according to the requested language file and the ``name`` variable is replaced independently of the language.

``

T("blah ") + name + T(" blah")

``:code

The following code is also allowed and often preferable:

``

T("blah %(name)s blah", dict(name='Tim'))

``:code

or the alternative syntax

``

T("blah %(name)s blah") % dict(name='Tim')

``:code

In both cases the translation occurs before the variable name is substituted in the "%(name)s" slot. The following alternative should NOT BE USED:

``

T("blah %(name)s blah" % dict(name='Tim'))

``:code

You can also force a per-string language:

T("Hello World", language="it-it")

``:code

--------------

In the case multiple languages are requested, for example "it-it, fr-ft", web2py tries to locate "it-it.py" and "fr-fr.py" translation files. If none of the requested files is present, it tries to fall back on "it.py" and "fr.py". If these files are not present it defaults to "default.py". If this is not present either, it default to no-translation. The more general rule is that web2py tries "xx-xy-yy.py", "xx-xy.py", "xx.py", "default.py" for each of the "xx-xy-yy" accepted languages trying to find the closest match to the visitor's preferences.

-------------

You can turn off translations completely via

``

T.force(None)

``:code

Normally, string translation is evaluated lazily when the view is rendered; hence, the translator ``force`` method should not be called inside a view.

It is possible to disable lazy evaluation via

``

T.lazy = False

``:code

It is also possible to disable lazy evaluation for individual strings:

``

T("Hello World", lazy=False)

``:code

A common issue is the following. The original application is in English. Suppose that there is a translation file (for example Italian, "it-it.py") and the HTTP client declares that it accepts both English (en) and Italian (it-it) in that order. The following unwanted situation occurs: web2py does not know the default is written in English (en). Therefore, it prefers translating everything into Italian (it-it) because it only found the Italian translation file. If it had not found the "it-it.py" file, it would have used the default language strings (English).

There are two solutions for this problem: create a translation language for English, which would be redundant and unnecessary, or better, tell web2py which languages should use the default language strings (the strings coded into the application). This can be done with:

``

T.set_current_languages('en', 'en-en')

``:code

It stores in ``T.current_languages`` a list of languages that do not require translation and forces a reload of the language files.

Notice that "it" and "it-it" are different languages from the point of view of web2py. To support both of them, one would need two translation files, always lower case. The same is true for all other languages.

The currently accepted language is stored in

``

x book(s)

with

``

a book (x==1)

5 books (x==5)

``

English has one singular form and one plural form. The plural form is constructed by adding a "-s" or "-es" or using an exceptional form. web2py provides a way to define pluralization rules for each languages, as well as exceptions to the default rules. In fact web2py already knows pluralization rules for many languages. It knows, for example, that Slovenian has one singular form and 3 plural forms (for x==1, x==3 or x==4 and x>4). These rules are encoded in "gluon/contrib/plural_rules/*.py" files and new files can be created. Explicit pluralizations for words are created by editing pluralization files using the administrative interface.

By default the PS is not activated. It is triggered by the ``symbol`` argument of the ``T`` function. For example:

``

T("You have %s %%{book}", symbols=10)

``:code

Now the PS is activated for the word "book" and for the number 10.

The result in English will be: "You have 10 books". Notice that "book" has been pluralized into "books".

The PS consists of 3 parts:

- rule to give a decision which word form to use ("rules/plural_rules/*.py")

- dictionary with word plural forms ("app/languages/plural-*.py")

The value of symbols can be a single variable, a list/tuple of variables, or a dictionary.

The placeholder ``%%{}`` consists of 3 parts:

``

``

where:

``

<modifier>::= ! | !! | !!!

<word> ::= any word or phrase in singular in lower case (!)

<parameter> ::= [index] | (key) | (number)

``

For example:

- ``%%{word}`` is equivalent to ``%%{word[0]}`` (if no modifiers are used).

- ``%%{word[index]}`` is used when symbols is a tuple. symbols[index] gives us a number used to make a decision on which word form to choose.

- ``%%{word(key)}`` is used to get the numeric parameter from symbols[key]

- ``%%{word(number)}`` allows to set a ``number`` directly (e.g.: ``%%{word(%i)}``)

- ``%%{?word?number}`` returns "word" if ``number==1``, returns the ``number`` otherwise

- ``%%{?number} or %%{??number}`` returns ``number`` if ``number!=1``, return nothing otherwise

``T("blabla %s %%{word}", symbols=var)``

error_handler = dict(application='error',

If the ``error_handler`` is specified the action is called without user redirection and the handler action will be in charge of dealing with the error. In the event that the error-handling page itself returns an error, web2py will fall back to its old static responses.

#### Static asset management

Since version 2.1.0, web2py has the ability to manage static assets.

When an application is in development, static file can change often, therefore web2py sends static files with no cache headers. This has the side-effect of "forcing" the browser to request static files at every request. This results in low performance when loading the page.

In a "production" site, you may want to serve static files with ``cache`` headers to prevent un-necessary downloads since static files do not change.

``cache`` headers allow the browser to fetch each file only once, thus saving bandwidth and reducing loading time.

Yet there is a problem: What should the cache headers declare? When should the files expire? When the files are first served, the server cannot forecast when they will be changed.

A manual approach consists of creating subfolders for different versions of static files. For example an early version of "layout.css" can be made available at the URL "/myapp/static/css/1.2.3/layout.css". When you change the file, you create a new subfolder and you link it as "/myapp/static/css/1.2.4/layout.css".

This procedure works but it is pedantic since every time you update the css file, you must remember to move it to another folder, change the URL of the file in your layout.html and deploy.

If you want to serve "/myapp/static/layout.css" with the cache headers, you just need to include the file with a modified URL that includes a version number:

``

/myapp/static/_1.2.3/layout.css

``

(notice the URL defines a version number, it does not appear anywhere else).

Notice that the URL starts with "/myapp/static/", followed by a version number composed by an underscore and 3 integers separated by a period (as described in [[SemVer http://semver.org/]]), then followed by the filename. Also notice that you do not have to create a "_1.2.3/" folder.

Every time the static file is requested with a version in the url, it will be served with "far in the future" cache headers, specifically:

``

Cache-Control : max-age=315360000

Expires: Thu, 31 Dec 2037 23:59:59 GMT

``

This means that the browser will fetch those files only once, and they will be saved "forever" in the browser's cache.

Every time the "_1.2.3/filename" is requested, web2py will remove the version part from the path and serve your file with far in the future headers so they will be cached forever. If you changed the version number in the URL, this tricks the browser into thinking it is requesting a different file, and the file is fetched again.

You can use "_1.2.3", "_0.0.0", "_999.888.888", as long as the version starts with underscore followed by three numbers separated by period.

By **cron** we refer to a web2py functionality not to the Unix Cron mechanism. T

web2py cron is the way to go if you need tasks in the background at scheduled times and these tasks take a relatively short time compared to the time interval between two calls. Each task runs in its own process, and multiple tasks can run concurrently, but you have no control over how many tasks run. If accidentally one task overlaps with itself, it can cause a database lock and a spike in memory usage.

web2py scheduler takes a different approach. The number of running processes is fixed, and they can run on different machines. Each process is called a worker. Each worker picks a task when available and executes it as soon as possible after the time when it is scheduled to run, but not necessarily at that exact time. There cannot be more processes running than the number of scheduled tasks and therefore no memory spikes. Scheduler tasks can be defined in models and are stored in the database. The web2py scheduler does not implement a distributed queue since it assumes that the time to distribute tasks is negligible compared with the time to run the tasks. Workers pick up the task from the database.

Homemade tasks queues can be a simpler alternative to the web2py scheduler in some cases.

#### Cron

``cron``:inxx

The web2py cron provides the ability for applications to execute tasks at preset times, in a platform-independent manner.

For each application, cron functionality is defined by a crontab file:

``

app/cron/crontab

``

It follows the syntax defined in ref. ``cron``:cite (with some extensions that are specific to web2py).

------

------

This means that every application can have a separate cron configuration and that cron config can be changed from within web2py without affecting the host OS itself.

Here is an example:

``

0-59/1 * * * * root python /path/to/python/script.py

30 3 * * * root *applications/admin/cron/db_vacuum.py

*/30 * * * * root **applications/admin/cron/something.py

@reboot root *mycontroller/myfunction

@hourly root *applications/admin/cron/expire_sessions.py

``:code

The last two lines in this example use extensions to regular cron syntax to provide additional web2py functionality.

-------

The file "applications/admin/cron/expire_sessions.py" actually exists and ships with the **admin** app. It checks for expired sessions and deletes them. "applications/admin/cron/crontab" runs this task hourly.

-------

Notice that scripts/functions executed in the web2py environment require a manual ``db.commit()`` at the end of the function or the transaction will be reverted.

web2py does not generate tickets or meaningful tracebacks in shell mode, which is how cron is run, so make sure that your web2py code runs without errors before you set it up as a cron task as you will likely not be able to see those errors when run from cron. Moreover, be careful how you use models: while the execution happens in a separate process, database locks have to be taken into account in order to avoid pages waiting for cron tasks that may be blocking the database. Use the ``**`` syntax if you don't need to use the database in your cron task.

You can also call a controller function, in which case there is no need to specify a path. The controller and function will be that of the invoking application. Take special care about the caveats listed above. Example:

``

*/30 * * * * root *mycontroller/myfunction

``:code

If you specify ``@reboot`` in the first field in the crontab file, the given task will be executed only once, at web2py startup. You can use this feature if you want to pre-cache, check, or initialize data for an application on web2py startup. Note that cron tasks are executed in parallel with the application --- if the application is not ready to serve requests until the cron task is finished, you should implement checks to reflect this. Example:

``

@reboot * * * * root *mycontroller/myfunction

``:code

Depending on how you are invoking web2py, there are four modes of operation for web2py cron.

- ''soft cron'': available under all execution modes

- ''hard cron'': available if using the built-in web server (either directly or via Apache mod_proxy)

- ''external cron'': available if you have access to the system's own cron service

- No cron

Scheduler(

migrate=True,

worker_name=None,

group_names=None,

heartbeat=HEARTBEAT,

max_empty_runs=0,

discard_results=False,

utc_time=False

)

``:code

Let's see them in order:

- ``db`` is the database DAL instance where you want the scheduler tables be placed.

- ``tasks`` is a dictionary that maps task names into functions. If you do not pass this parameter, function will be searched in the app environment.

- ``worker_name`` is None by default. As soon as the worker is started, a worker name is generated as hostname-uuid. If you want to specify that, be sure that it's unique.

- ``group_names`` is by default set to **[main]**. All tasks have a ``group_name`` parameter, set to **main** by default. Workers can only pick up tasks of their assigned group.

------

NB: This is useful if you have different workers instances (e.g. on different machines) and you want to assign tasks to a specific worker.

------

- ``heartbeat`` is by default set to 3 seconds. This parameter is the one controlling how often a scheduler will check its status on the ``scheduler_worker`` table and see if there are any **ASSIGNED** tasks to itself to process.

- ``max_empty_runs`` is 0 by default, that means that the worker will continue to process tasks as soon as they are **ASSIGNED**. If you set this to a value of, let's say, 10, a worker will die automatically if it's **ACTIVE** and no tasks are **ASSIGNED** to it for 10 loops. A loop is when a worker searches for tasks, every 3 seconds (or the set ``heartbeat``)

- ``discard_results`` is False by default. If set to True, no scheduler_run records will be created.

------

NB: scheduler_run records will be created as before for **FAILED**, **TIMEOUT** and **STOPPED** tasks's statuses.

------

- ``utc_time`` is False by default. If you need to coordinate with workers living in different timezones, or don't have problems with solar/DST times, supplying datetimes from different countries, etc, you can set this to True. The scheduler will honor the UTC time and work leaving the local time aside. Caveat: you need to schedule tasks with UTC times (for start_time, stop_time, and so on.)

Now we have the infrastructure in place: defined the tasks, told the scheduler about them, started the worker(s). What remains is to actually schedule the tasks

##### Tasks

Tasks can be scheduled programmatically or via appadmin. In fact, a task is scheduled simply by adding an entry in the table "scheduler_task", which you can access via appadmin:

``

http://127.0.0.1:8000/myapp/appadmin/insert/db/scheduler_task

``

st.validate_and_insert(function_name='demo1', vars=json.dumps({'a':1,'b':2}))

Here is a more complex complete example:

``

def task_add(a,b):

return a+b

secheduler = Scheduler(db, tasks=dict(demo1=task_add))

scheduler.queue_task('demo1', pvars=dict(a=1,b=2),

repeats = 0, period = 180)

``:code

Since version 2.4.1 if you pass an additional parameter ``immediate=True`` it will force the main worker to reassign tasks. Until 2.4.1, the worker checks for new tasks every 5 cycles (so, ``5*heartbeats`` seconds). If you had an app that needed to check frequently for new tasks, to get a ''snappy'' behaviour you were forced to lower the ``heartbeat`` parameter, putting the db under pressure for no reason. With ``immediate=True`` you can force the check for new tasks: it will happen at most as ``heartbeat`` seconds are passed

A call to ``scheduler.queue_task`` returns the task ``id`` and ``uuid`` of the task you queued (can be the one you passed or the auto-generated one), and possible ``errors``:

``

<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>

``

you get the result of the validation, and id and uuid will be None

``

<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>

``

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

##### Managing processes

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

When you start a worker, you may later want to:

- kill it "no matter what it's doing"

- kill it only if it is not processing tasks

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

To be more precise, for started workers you can change the ``status`` value of any worker to influence

its behavior.

As for tasks, workers can be in one of the following statuses: ACTIVE, DISABLED, TERMINATE or KILLED.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

scheduler.queue_task(

timeout = 120, # should take less than 120 seconds

)

``:code

Notice that fields "times_run", "last_run_time" and "assigned_worker_name" are not provided at schedule time but are filled automatically by the workers.

You can also retrieve the output of completed tasks:

``

completed_runs = db(db.scheduler_run.run_status='COMPLETED').select()

``:code

------

The scheduler is considered experimental because it needs more extensive testing and because the table structure may change as more features are added.

------

##### Reporting percentages

A special "word" encountered in the print statements of your functions clear all

the previous output. That word is ``!clear!``.

Here is an example:

``

def reporting_percentages():

time.sleep(5)

print '50%'

time.sleep(5)

print '!clear!100%'

return 1

``

The function ``reporting_percentages`` sleeps for 5 seconds, outputs ``50%``.

Then, it sleeps other 5 seconds and outputs ``100%``. Note that the output in the scheduler_run table is synced every 2 seconds and that the second print statement that contains ``!clear!100%`` gets the ``50%`` output cleared and replaced by ``100%`` only.

``

scheduler.queue_task(reporting_percentages,

sync_output=2)

``:code

result = other.some_action()

``:code

In line 2, ``request=request`` is optional. It has the effect of passing the current request to the environment of "other". Without this argument, the environment would contain a new and empty (apart from ``request.folder``) request object. It is also possible to pass a response and a session object to ``exec_environment``. Be careful when passing request, response and session objects --- modification by the called action or coding dependencies in the called action could lead to unexpected side effects.

The function call in line 3 does not execute the view; it simply returns the dictionary unless ``response.render`` is called explicitly by "some_action".

One final caution: don't use ``exec_environment`` inappropriately. If you want the results of actions in another application, you probably should implement an XML-RPC API (implementing an XML-RPC API with web2py is almost trivial). Don't use ``exec_environment`` as a redirection mechanism; use the ``redirect`` helper.

### Cooperation

``cooperation``:inxx

There are many ways applications can cooperate:

- Applications can connect to the same database and thus share tables. It is not necessary that all tables in the database are defined by all applications, but they must be defined by those applications that use them. All applications that use the same table, but one, must define the table with ``migrate=False``.

- Applications can embed components from other applications using the LOAD helper (described in Chapter 12).

- Applications can share sessions.

- Applications can call each other's actions remotely via XML-RPC.

- Applications can access each other's files via the filesystem (assuming they share the same filesystem).

- Applications can call each other's actions locally using ``exec_environment`` as discussed above.

- Applications can import each other's modules using the syntax:

``

``:code

- Applications can import any module in the ``PYTHONPATH`` search path, ``sys.path``.

One app can load the session of another app using the command:

``

session.connect(request, response, masterapp='appname', db=db)

``:code

Here "appname" is the name of the master application, the one that sets the initial session_id in the cookie. ``db`` is a database connection to the database that contains the session table (``web2py_session``). All apps that share sessions must use the same database for session storage.

``cache view``:inxx

``

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)

def cache_controller_and_view():

import time

t = time.ctime()

d = dict(time=t, link=A('click to reload', _href=request.url))

return response.render(d)

``:code

``response.render(d)`` returns the rendered view as a string, which is now cached for 5 seconds. This is the best and fastest way of caching.

Note, ``time_expire`` is used to compare the current time with the time the requested object was last saved in the cache. It does not affect future requests. This enables ``time_expire`` to be set dynamically when an object is requested rather than being fixed when the object is saved. For example:

``

message = cache.ram('message', lambda: 'Hello', time_expire=5)

``:code

Now, suppose the following call is made 10 seconds after the above call:

``

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

It is also possible to define other caching mechanisms such as memcache. Memcache is available via ``gluon.contrib.memcache`` and is discussed in more details in Chapter 14.

------

Be careful when caching to remeber that caching is usually at the app-level not at the user level. If you need, for example, to cache user specific content, choose a key that includes the user id.

------

``cache view``:inxx

``

@cache(request.env.path_info, time_expire=5, cache_model=cache.ram)

def cache_controller_and_view():

import time

t = time.ctime()

d = dict(time=t, link=A('click to reload', _href=request.url))

return response.render(d)

``:code

``response.render(d)`` returns the rendered view as a string, which is now cached for 5 seconds. This is the best and fastest way of caching.

Note, ``time_expire`` is used to compare the current time with the time the requested object was last saved in the cache. It does not affect future requests. This enables ``time_expire`` to be set dynamically when an object is requested rather than being fixed when the object is saved. For example:

``

message = cache.ram('message', lambda: 'Hello', time_expire=5)

``:code

Now, suppose the following call is made 10 seconds after the above call:

``

message = cache.ram('message', lambda: 'Goodbye', time_expire=20)

``:code

Because ``time_expire`` is set to 20 seconds in the second call and only 10 seconds has elapsed since the message was first saved, the value "Hello" will be retrieved from the cache, and it will not be updated with "Goodbye". The ``time_expire`` value of 5 seconds in the first call has no impact on the second call.

Setting ``time_expire=0`` (or a negative value) forces the cached item to be refreshed (because the elapsed time since the last save will always be > 0), and setting ``time_expire=None`` forces retrieval of the cached value, regardless of the time elapsed since it was saved (if ``time_expire`` is always ``None``, the cached item will effectively never expire).

You can clear one or more cache variables with

``cache clear``:inxx

``

cache.ram.clear(regex='...')

``:code

where ``regex`` is a regular expression matching all the keys you want removed from the cache. You can also clear a single item with:

``

cache.ram(key, None)

``:code

where ``key`` is the key of the cached item.

It is also possible to define other caching mechanisms such as memcache. Memcache is available via ``gluon.contrib.memcache`` and is discussed in more details in Chapter 14.

------

Be careful when caching to remeber that caching is usually at the app-level not at the user level. If you need, for example, to cache user specific content, choose a key that includes the user id.

------

-------

When web2py is upgraded it comes with a file called "NEWINSTALL". If web2py finds this file, it understands an upgrade was performed, hence it removed the file and creates a new "welcome.w2p".

The current web2py version is stored in the field "VERSION" and it follows standard semantic versioning notation where the build id is the build timestamp.

web2py unit-tests are in

``

gluon/tests/

``:code

There are handlers for connecting with various web servers:

``

cgihandler.py # discouraged

gaehandler.py # for Google App Engine

fcgihandler.py # for FastCGI

wsgihandler.py # for WSGI

isapiwsgihandler.py # for IIS

modpythonhandler.py # deprecated

``:code

web2py also contains a folder with useful scripts including

``

scripts/setup-web2py-fedora.sh

scripts/setup-web2py-ubuntu.sh

scripts/setup-web2py-nginx-uwsgi-ubuntu.sh

scripts/setup-web2py-heroku.sh

scripts/update-web2py.sh

scripts/make_min_web2py.py

...

scripts/sessions2trash.py

scripts/sync_languages.py

scripts/tickets2db.py

scripts/tickets2email.py

...

scripts/extract_mysql_models.py

scripts/extract_pgsql_models.py

...

scripts/access.wsgi

scripts/cpdb.py

``:code

Some of these are discussed in Chapter 14, but all of them contain a documentation string inside that explains their purpose and usage.

Finally web2py includes these files required to build the binary distributions.

``

Makefile

setup_exe.py

setup_app.py

``:code

These are setup scripts for **py2exe** and **py2app**, respectively, and they are only required to build the binary distributions of web2py. YOU SHOULD NEVER NEED TO RUN THEM.

web2py applications contain additional files, particularly third-party JavaScript libraries, such as jQuery, calendar, and Codemirror. Their authors are acknowledged in the files themselves.

### Applications

Applications developed in web2py are composed of the following parts:

- **models** describe a representation of the data as database tables and relations between tables.

- **controllers** describe the application logic and workflow.

- **views** describe how data should be presented to the user using HTML and JavaScript.

- **languages** describe how to translate strings in the application into various supported languages.

Now from a controller in "myapp" you can do

import test

def index():

return "Your ip is " + test.ip()

``

Notice a few things:

- ``import test`` looks for the module first in the current app's modules folder, then in the folders listed in ``sys.path``. Therefore, app-level modules always take precedence over Python modules. This allows different apps to ship with different versions of their modules, without conflicts.

- Different users can call the same action ``index`` concurrently, which calls the function in the module, and yet there is no conflict because ``current.request`` is a different object in different threads. Just be careful not to access ``current.request`` outside of functions or classes (i.e., at the top level) in the module.

- ``import test`` is a shortcut for ``from applications.appname.modules import test``. Using the longer syntax, it is possible to import modules from other applications.

For uniformity with normal Python behavior, by default web2py does not reload modules when changes are made. Yet this can be changed. To turn on the auto-reload feature for modules, use the ``track_changes`` function as follows (typically in a model file, before any imports):

``

from gluon.custom_import import track_changes; track_changes(True)

``:code

From now on, every time a module is imported, the importer will check if the Python source file (.py) has changed. If it has changed, the module will be reloaded.

Track changes only tracks changes for modules that are stored in the application.

Modules that import ``current`` can access:

- ``current.request``

- ``current.response``

- ``current.session``

- ``current.cache``

- ``current.T``

and any other variable your application chooses to store in current. For example a model could do

``

auth = Auth(db)

from gluon import current

current.auth = auth

``

``current`` and ``import`` create a powerful mechanism to build extensible and reusable modules for your applications.

-------

``

request = current.request # WRONG! DANGER!

``

nor one should use it assign class attributes

``

class MyClass:

request = current.request # WRONG! DANGER!

``

This is because the thread local object must be extracted at runtime. Global variables instead are defined only once when the model is imported for the first time.

-------

Another caveat has to do with cache. You cannot use the ``cache`` object to decorate functions in modules, that is because it would not behave as expected. In order to cache a function ``f`` in a module you must use ``lazy_cache``:

``

from gluon.cache import lazy_cache

lazy_cache('key', time_expire=60, cache_model='ram')

def f(a,b,c,): ....

``:code

The ``request`` object is an instance of the ubiquitous web2py class that is cal

request.vars

``:code

is the same as:

``

request['vars']

``:code

Unlike a dictionary, if an attribute (or key) does not exist, it does not raise an exception. Instead, it returns ``None``.

-----

It is sometimes useful to create your own Storage objects. You can do so as follows:

``

from gluon.storage import Storage

my_storage = Storage() # empty storage object

my_other_storage = Storage(dict(a=1, b=2)) # convert dictionary to Storage

``:code

-----

``request`` has the following items/attributes, some of which are also an instance of the ``Storage`` class:

- ``request.env``: a ``Storage`` object containing the environment variables passed to the controller, including HTTP header variables from the HTTP request and standard WSGI parameters. The environment variables are all converted to lower case, and dots are converted to underscores for easier memorization.

- ``request.extension``: the extension of the requested action. It defaults to "html". If the controller function returns a dictionary and does not specify a view, this is used to determine the extension of the view file that will render the dictionary (parsed from the ``request.env.path_info``).

- ``request.folder``: the application directory. For example if the application is "welcome", ``request.folder`` is set to the absolute path "/path/to/welcome". In your programs, you should always use this variable and the ``os.path.join`` function to build paths to the files you need to access. Although web2py always uses absolute paths, it is a good rule never to explicitly change the current working folder (whatever that is) since this is not a thread-safe practice.

- ``request.now``: a ``datetime.datetime`` object storing the datetime of the current request.

- ``request.utcnow``: a ``datetime.datetime`` object storing the UTC datetime of the current request.

- ``request.args``: A list of the URL path components following the controller function name; equivalent to ``request.env.path_info.split('/')[3:]``

- ``request.vars``: a ``gluon.storage.Storage`` object containing both the HTTP GET and HTTP POST query variables.

- ``request.get_vars``: a ``gluon.storage.Storage`` object containing only the HTTP GET query variables.

- ``request.post_vars``: a ``gluon.storage.Storage`` object containing only the HTTP POST query variables.

- ``request.client``: The ip address of the client as determined by, if present, ``request.env.http_x_forwarded_for`` or by ``request.env.remote_addr`` otherwise. While this is useful it should not be trusted because the ``http_x_forwarded_for`` can be spoofed.

- ``request.is_local``: ``True`` if the client is localhost, ``False`` otherwise. Should work behind a proxy if the proxy supports ``http_x_forwarded_for``.

- ``request.is_https``: ``True`` if the request is using the HTTPS protocol, ``False`` otherwise.

- ``request.body``: a read-only file stream that contains the body of the HTTP request. This is automatically parsed to get the ``request.post_vars`` and then rewinded. It can be read with ``request.body.read()``.

- ``request.ajax`` is True if the function is being called via an Ajax request.

- ``request.cid`` is the ``id`` of the component that generated the Ajax request (if any). You can read more about components in Chapter 12.

- ``request.requires_https()`` prevents further code execution if the request is not over HTTPS and redirects the visitor to the current page over HTTPS.

- ``request.restful`` this is a new and very useful decorator that can be used to change the default behavior of web2py actions by separating GET/POST/PUSH/DELETE requests. It will be discussed in some detail in Chapter 10.

- ``request.user_agent()`` parses the user_agent field from the client and returns the information in the form of a dictionary. It is useful to detect mobile devices. It uses "gluon/contrib/user_agent_parser.py" created by Ross Peoples. To see what it does, try to embed the following code in a view:

``

{{=BEAUTIFY(request.user_agent())}}

``:code

Also notice the ``request.env.wsgi_*`` variables. They are specific to the wsgi

``response.status``:inxx

``response.stream``:inxx

``response.subtitle``:inxx

``response.title``:inxx

``response.toolbar``:inxx

``response.view``:inxx

``response.delimiters``:inxx

``response.js``:inxx

``response.write``:inxx

``response.include_files``:inxx

``response.include_meta``:inxx

``response.optimize_css``:inxx

``response.optimize_js``:inxx

``response._caller``:inxx

``response`` is another instance of the ``Storage`` class. It contains the following:

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g.``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.stream(file, chunk_size, request=request, attachment=False, filename=None, headers=None)``: when a controller returns it, web2py streams the file content back to the client in blocks of size ``chunk_size``. The ``request`` parameter is required to use the chunk start in the HTTP header. ``file`` should be a file path (for backward compatibility, it can also be an open file object, but this is not recommended). As noted above, ``response.download`` should be used to retrieve files stored via an upload field. ``response.stream`` can be used in other cases, such as returning a temporary file or StringIO object created by the controller. If ``attachment`` is True, the Content-Disposition header will be set to "attachment", and if ``filename`` is also provided, it will be added to the Content-Disposition header as well (but only when ``attachment`` is True). If not already included in ``response.headers``, the following response headers will be set automatically: Content-Type, Content-Length, Cache-Control, Pragma, and Last-Modified (the latter three are set to allow browser caching of the file). To override any of these automatic header settings, simply set them in ``response.headers`` before calling ``response.stream``.

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.title``: optional parameter that may be included in the views. It should contain the title of the page and should be rendered by the HTML title TAG in the header.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._vars``: this variable is accessible only in a view, not in the action. It contains the value returned by the action to the view.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.optimize_css``: if can be set to "concat,minify,inline" to concatenate, minify and inline the CSS files included by web2py.

- ``response.optimize_js``: if can be set to "concat,minify,inline" to concatenate, minify and inline the JavaScript files included by web2py.

- ``response.view``: the name of the view template that must render the page. This is set by default to:

is simply a shortcut for:

raise HTTP(303,

'You are being redirected <a href="%s">here</a>' % location,

Location='http://www.web2py.com')

``:code

The named arguments of the ``HTTP`` initializer method are translated into HTTP header directives, in this case, the redirection target location. ``redirect`` takes an optional second argument, which is the HTTP status code for the redirection (303 by default). Change this number to 307 for a temporary redirect or to 301 for a permanent redirect.

The most common way to use redirect is to redirect to other pages in the same app and (optionally) pass parameters:

``

redirect(URL('index', args=(1,2,3), vars=dict(a='b')))

``:code

In chapter 12 we discuss web2py components. They make Ajax requests to web2py actions. If the called action performs a redirect, you may want the Ajax request to follow the redirect or you may want the entire page performing the Ajax request redirecting. In this latter case you can set:

``

redirect(...,type='auto')

``:code

``T``:inxx ``internationalization``:inxx

The object ``T`` is the language translator. It constitutes a single global instance of the web2py class ``gluon.language.translator``. All string constants (and only string constants) should be marked by ``T``, for example:

``

a = T("hello world")

``:code

Strings that are marked with ``T`` are identified by web2py as needing language translation and they will be translated when the code (in the model, controller, or view) is executed. If the string to be translated is not a constant but a variable, it will be added to the translation file at runtime (except on GAE) to be translated later.

The ``T`` object can also contain interpolated variables and supports multiple equivalent syntaxes:

``

a = T("hello %s", ('Tim',))

a = T("hello %(name)s", dict(name='Tim'))

a = T("hello %s") % ('Tim',)

a = T("hello %(name)s") % dict(name='Tim')

``:code

The latter syntax is recommended because it makes translation easier.

The first string is translated according to the requested language file and the ``name`` variable is replaced independently of the language.

T("Hello World", lazy=False)

``:code

A common issue is the following. The original application is in English. Suppose that there is a translation file (for example Italian, "it-it.py") and the HTTP client declares that it accepts both English (en) and Italian (it-it) in that order. The following unwanted situation occurs: web2py does not know the default is written in English (en). Therefore, it prefers translating everything into Italian (it-it) because it only found the Italian translation file. If it had not found the "it-it.py" file, it would have used the default language strings (English).

There are two solutions for this problem: create a translation language for English, which would be redundant and unnecessary, or better, tell web2py which languages should use the default language strings (the strings coded into the application). This can be done with:

``

T.set_current_languages('en', 'en-en')

``:code

It stores in ``T.current_languages`` a list of languages that do not require translation and forces a reload of the language files.

Notice that "it" and "it-it" are different languages from the point of view of web2py. To support both of them, one would need two translation files, always lower case. The same is true for all other languages.

The currently accepted language is stored in

``

T.accepted_language

``:code

#### Translating variables

``

>>> a="test"

>>> print T(a)

``:code

In this case the word "test" is translated but, if not found and if the filesystem is writable, it will add it to the list of words to be translated in the language file.

Notice that this can result in lots of file IO and you may want to disable it:

``

T.is_writable = False

``:code

prevents T from dynamically updating language files.

#### Comments and multiple translations

It is possible that the same string appears in different contexts in the application and needs different translations based on context. In order to do this, one can add comments to the original string. The comments will not be rendered but will be used by web2py to determine the most appropriate translation. For example:

``

T("hello world ## first occurrence")

T("hello world ## second occurrence")

``:code

The text following the ``##``, including the double ``##``, are comments.

#### Pluralization engine

Since version 2.0, web2py includes a powerful pluralization system (PS). This means that when text marked for translation depends on a numeric variable, it may be translated differently based on the numeric value. For example in English we may render:

``

x book(s)

``

with

``

``

English has one singular form and one plural form. The plural form is constructed by adding a "-s" or "-es" or using an exceptional form. web2py provides a way to define pluralization rules for each languages, as well as exceptions to the default rules. In fact web2py already knows pluralization rules for many languages. It knows, for example, that Slovenian has one singular form and 3 plural forms (for x==1, x==3 or x==4 and x>4). These rules are encoded in "gluon/contrib/plural_rules/*.py" files and new files can be created. Explicit pluralizations for words are created by editing pluralization files using the administrative interface.

By default the PS is not activated. It is triggered by the ``symbol`` argument of the ``T`` function. For example:

``

T("You have %s %%{book}", symbols=10)

``:code

Now the PS is activated for the word "book" and for the number 10.

The result in English will be: "You have 10 books". Notice that "book" has been pluralized into "books".

The PS consists of 3 parts:

- placeholders ``%%{}`` to mark words in ``T``-messages

- rule to give a decision which word form to use ("rules/plural_rules/*.py")

- dictionary with word plural forms ("app/languages/plural-*.py")

The value of symbols can be a single variable, a list/tuple of variables, or a dictionary.

The placeholder ``%%{}`` consists of 3 parts:

``

%%{[<modifier>]<world>[<parameter>]},

``

where:

``

<modifier>::= ! | !! | !!!

<word> ::= any word or phrase in singular in lower case (!)

<parameter> ::= [index] | (key) | (number)

``

For example:

- ``%%{word[index]}`` is used when symbols is a tuple. symbols[index] gives us a number used to make a decision on which word form to choose.

- ``%%{word(key)}`` is used to get the numeric parameter from symbols[key]

- ``%%{word(number)}`` allows to set a ``number`` directly (e.g.: ``%%{word(%i)}``)

- ``%%{?word?number}`` returns "word" if ``number==1``, returns the ``number`` otherwise

- ``%%{?number} or %%{??number}`` returns ``number`` if ``number!=1``, return nothing otherwise

``T("blabla %s %%{word}", symbols=var)``

``%%{word}`` by default means ``%%{word[0]}``,

where ``[0]`` is an item index in symbols tuple.

``T("blabla %s %s %%{word[1]}", (var1, var2))``

PS is used for "word" and var2 respectively.

``T("%%{this} %%{is} %s %%{book}", var)``

or

``T("%%{this[0]} %%{is[0]} %s %%{book[0]}", var)``

``

var output

------------------

1 this is 1 book

2 these are 2 books

3 these are 2 books

``

Similarly you can pass a dictionary to symbols:

``

T("blabla %(var1)s %(wordcnt)s %%{word(wordcnt)}",

dict(var1="tututu", wordcnt=20))

``

which produces

``

blabla tututu 20 words

For example

``T("%%{this} %%{is} %%{?a?%s} %%{book}", var)``

produces:

``

var output

------------------

1 this is a book

2 these are 2 books

3 these are 3 books

...

``

Inside ``%%{...}`` you can also use the following modifiers:

- ``!`` to capitalize the text (equivalent to ``string.capitalize``)

- ``!!`` to capitalize every word (equivalent to ``string.title``)

- ``!!!`` to capitalize every character (equivalent to ``string.upper``)

#### Translations, pluralization, and MARKMIN

You can also use the powerful MARKMIN syntax inside translation strings by replacing

``

T("hello world")

``:code

with

``

T.M("hello world")

``

Now the string accepts MARKMIN markup as described later in the book. You can also use the pluralization system inside MARKMIN.

### Cookies

``cookies``:inxx

Here are four ways to set the default application:

- Make a symbolic link from "applications/init" to your application's folder.

- Use URL rewrite as discussed in the next section.

### URL rewrite

``url rewrite``:inxx

``routes_in``:inxx

``routes_out``:inxx

web2py has the ability to rewrite the URL path of incoming requests prior to calling the controller action (URL mapping), and conversely, web2py can rewrite the URL path generated by the ``URL`` function (reverse URL mapping). One reason to do this is for handling legacy URLs, another is to simplify paths and make them shorter.

web2py includes two distinct URL rewrite systems: an easy-to-use ''parameter-based'' system for most use cases, and a flexible ''pattern-based'' system for more complex cases. To specify the URL rewrite rules, create a new file in the "web2py" folder called ``routes.py`` (the contents of ``routes.py`` will depend on which of the two rewrite systems you choose, as described in the next two sections). The two systems cannot be mixed.

-------

Notice that if you edit routes.py, you must reload it. This can be done in two ways: by restarting the web server or by clicking on the routes reload button in admin. If there is a bug in routes, they will not reload.

-------

#### Parameter-based system

The parameter-based (parametric) router provides easy access to several "canned" URL-rewrite methods. Its capabilities include:

The parametric router also provides somewhat more flexible validation of incoming URLs.

Suppose you've written an application called ``myapp`` and wish to make it the default, so that the application name is no longer part of the URL as seen by the user. Your default controller is still ``default``, and you want to remove its name from user-visible URLs as well. Here's what you put in ``routes.py``:

``

routers = dict(

BASE = dict(default_application='myapp'),

)

``:code

That's it. The parametric router is smart enough to know how to do the right thing with URLs such as:

``

http://domain.com/myapp/default/myapp

``:code

or

``

http://domain.com/myapp/myapp/index

``:code

where normal shortening would be ambiguous. If you have two applications, ``myapp`` and ``myapp2``, you'll get the same effect, and additionally ``myapp2``'s default controller will be stripped from the URL whenever it's safe (which is mostly all the time).

AliasMatch ^/([^/]+)/static/(?:/_[\d]+\.[\d]+\.[\d]+)?(.*) \

/home/www-data/web2py/applications/$1/static/$2

``

Similarly, in Nginx change this:

``

location ~* /(\w+)/static/ {

root /home/www-data/web2py/applications/;

expires max;

}

``

into this:

``

location ~* /(\w+)/static(?:/_[\d]+\.[\d]+\.[\d]+)?/(.*)$ {

alias /home/www-data/web2py/applications/$1/static/$2;

expires max;

}

``

### Running tasks in the background

The proper way to run time-consuming tasks is doing it in the background. There is not a single way of doing it, but here we describe three mechanisms that are built into web2py: **cron**, **homemade task queues**, and **scheduler**.

By **cron** we refer to a web2py functionality not to the Unix Cron mechanism. The web2py cron works on windows too.

web2py cron is the way to go if you need tasks in the background at scheduled times and these tasks take a relatively short time compared to the time interval between two calls. Each task runs in its own process, and multiple tasks can run concurrently, but you have no control over how many tasks run. If accidentally one task overlaps with itself, it can cause a database lock and a spike in memory usage.

web2py scheduler takes a different approach. The number of running processes is fixed, and they can run on different machines. Each process is called a worker. Each worker picks a task when available and executes it as soon as possible after the time when it is scheduled to run, but not necessarily at that exact time. There cannot be more processes running than the number of scheduled tasks and therefore no memory spikes. Scheduler tasks can be defined in models and are stored in the database. The web2py scheduler does not implement a distributed queue since it assumes that the time to distribute tasks is negligible compared with the time to run the tasks. Workers pick up the task from the database.

Homemade tasks queues can be a simpler alternative to the web2py scheduler in some cases.

#### Cron

``cron``:inxx

The web2py cron provides the ability for applications to execute tasks at preset times, in a platform-independent manner.

For each application, cron functionality is defined by a crontab file:

``

app/cron/crontab

Example of line to add to the system crontab, (usually /etc/crontab):

With external ``cron``, make sure to add either ``-J`` (or ``--cronjob``, which is the same) as indicated above so that web2py knows that task is executed by cron. Web2py sets this internally with soft and hard ``cron``.

#### Homemade task queues

While cron is useful to run tasks at regular time intervals, it is not always the best solution to run a background task. For this purpose web2py provides the ability to run any python script as if it were inside a controller:

``

python web2py.py -S app -M -R applications/app/private/myscript.py -A a b c

``:code

where ``-S app`` tells web2py to run "myscript.py" as "app", ``-M`` tells web2py to execute models, and ``-A a b c`` passes optional command line arguments ``sys.args=['a','b','c']`` to "myscript.py".

This type of background process should not be executed via cron (except perhaps for cron @reboot) because you need to be sure that no more than one instance is running at the same time. With cron it is possible that a process starts at cron iteration 1 and is not completed by cron iteration 2, so cron starts it again, and again, and again - thus jamming the mail server.

In chapter 8, we will provide an example of how to use the above method to send emails.

#### Scheduler (experimental)

The web2py scheduler works very much like the task queue described in the previous sub-section with some differences:

- There is not a single background process but a set of workers processes.

- The job of worker nodes can be monitored because their state, as well as the state of the tasks, is stored in the database.

- It works without web2py but that is not documented here.

The scheduler does not use cron, although one can use cron @reboot to start the worker nodes.

More information about deploying the scheduler under Linux and Windows is in the Deployment recipes chapter.

In the scheduler, a task is simply a function defined in a model (or in a module and imported by a model). For example:

``

def task_add(a,b):

return a+b

``:code

Tasks will always be called in the same environment seen by controllers and therefore they see all the global variables defined in models, including database connections (``db``). Tasks differ from a controller action because they are not associated with an HTTP request and therefore there is no ``request.env``.

------

------

The recommended way to enable the scheduler to your app is to create a model file named ``scheduler.py`` and define your function there. After the functions, you can put the following code into the model:

``

from gluon.scheduler import Scheduler

scheduler = Scheduler(db)

``:code

The task is scheduled with

``

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

##### Parameters

The first argument of the ``Scheduler`` class must be the database to be used by the scheduler to communicate with the workers. This can be the ``db`` of the app or another dedicated ``db``, perhaps one shared by multiple apps. If you use SQLite it's recommended to use a separate db from the one used by your app in order to keep the app responsive.

Once the tasks are defined and the ``Scheduler`` is instantiated, all that is needed to do is to start the workers. You can do that in several ways:

``

python web2py.py -K myapp

``

starts a worker for the app ``myapp``. If you want start multiple workers for the same app, you can do so just passing ``myapp,myapp``. You can pass also the ``group_names`` (overriding the one set in your model) with

``

python web2py.py -K myapp:group1:group2,myotherapp:group1

You can pass the usual parameters (-i, -p, here -a prevents the window from show

Scheduler's complete signature is:

``

Scheduler(

db,

tasks=None,

migrate=True,

worker_name=None,

group_names=None,

heartbeat=HEARTBEAT,

max_empty_runs=0,

discard_results=False,

utc_time=False

)

``:code

Let's see them in order:

- ``db`` is the database DAL instance where you want the scheduler tables be placed.

- ``worker_name`` is None by default. As soon as the worker is started, a worker name is generated as hostname-uuid. If you want to specify that, be sure that it's unique.

- ``group_names`` is by default set to **[main]**. All tasks have a ``group_name`` parameter, set to **main** by default. Workers can only pick up tasks of their assigned group.

------

NB: This is useful if you have different workers instances (e.g. on different machines) and you want to assign tasks to a specific worker.

NB2: It's possible to assign a worker more groups, and they can be also all the same, as

------

``['mygroup','mygroup']``. Tasks will be distributed taking into consideration that

a worker with group_names ``['mygroup','mygroup']`` is able to process the double of the tasks

a worker with group_names ``['mygroup']`` is.

- ``heartbeat`` is by default set to 3 seconds. This parameter is the one controlling how often a scheduler will check its status on the ``scheduler_worker`` table and see if there are any **ASSIGNED** tasks to itself to process.

- ``max_empty_runs`` is 0 by default, that means that the worker will continue to process tasks as soon as they are **ASSIGNED**. If you set this to a value of, let's say, 10, a worker will die automatically if it's **ACTIVE** and no tasks are **ASSIGNED** to it for 10 loops. A loop is when a worker searches for tasks, every 3 seconds (or the set ``heartbeat``)

- ``discard_results`` is False by default. If set to True, no scheduler_run records will be created.

------

NB: scheduler_run records will be created as before for **FAILED**, **TIMEOUT** and **STOPPED** tasks's statuses.

------

- ``utc_time`` is False by default. If you need to coordinate with workers living in different timezones, or don't have problems with solar/DST times, supplying datetimes from different countries, etc, you can set this to True. The scheduler will honor the UTC time and work leaving the local time aside. Caveat: you need to schedule tasks with UTC times (for start_time, stop_time, and so on.)

http://127.0.0.1:8000/myapp/appadmin/insert/db/scheduler_task

The meaning of the fields in this table is obvious. The "args" and "vars"" fields are the values to be passed to the task in JSON format. In the case of the "task_add" above, an example of "args" and "vars" could be:

``

args = [3, 4]

vars = {}

``:code

or

``

args = []

vars = {'a':3, 'b':4}

``:code

The ``scheduler_task`` table is the one where tasks are organized.

All tasks follow a lifecycle

[[scheduler tasks @///image/ce8edcc3.png center]]

If you need it to be executed later, use the ``start_time`` parameter (default = now).

that shuts down at 1AM, a mail that needs to be sent not after the working hours, etc...) you can set a ``stop_time`` (default = None) for it.

**RUNNING** tasks may end up:

Values for ``start_time`` and ``stop_time`` should be datetime objects. To schedule "mytask" to run at 30 seconds from the current time, for example, you would do the following:

``

from datetime import timedelta as timed

scheduler.queue_task('mytask',

start_time=request.now + timed(seconds=30))

``:code

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

------

------

[[task repeats @///image/7d8b85e4.png center]]

Summary: you have

- ``period`` and ``repeats`` to get an automatically rescheduled function

- ``timeout`` to be sure that a function doesn't exceed a certain amount of time

- ``retry_failed`` to control how many times the task can "fail"

- ``start_time`` and ``stop_time`` to schedule a function in a restricted timeframe

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

The scheduler also creates one more table called "scheduler_worker", which stores the workers' heartbeat and their status. Possible worker statuses are:

##### Managing processes

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

- kill it "no matter what it's doing"

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

its behavior.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

[[workers statuses @///image/bd891eed.png center]]

There are a few commodity functions since version 2.4.1 (self-explanatory)

``

scheduler.disable()

scheduler.resume()

scheduler.terminate()

scheduler.kill()

``:code

each function take an optional parameter, that can be a string or a list, to manage workers based on their ``group_names``. It defaults to the ``group_names`` defined in the scheduler istantiation.

An example is better than a thousand words: ``scheduler.terminate('high_prio')`` will TERMINATE all the workers that are processing the ``high_prio`` tasks, while ``scheduler.terminate(['high_prio', 'low_prio'])`` will terminate all ``high_prio`` and ``low_prio`` workers.

------

Watch out: if you have a worker processing ``high_prio`` and ``low_prio``, ``scheduler.terminate('high_prio')`` will terminate the worker alltogether, even if you didn't want to terminate ``low_prio`` too.

For example:

``

scheduler.queue_task(

function_name='task_add',

pargs=[],

pvars={'a':3,'b':4},

repeats = 10, # run 10 times

period = 3600, # every 1h

timeout = 120, # should take less than 120 seconds

)

``:code

Notice that fields "times_run", "last_run_time" and "assigned_worker_name" are not provided at schedule time but are filled automatically by the workers.

You can also retrieve the output of completed tasks:

``

completed_runs = db(db.scheduler_run.run_status='COMPLETED').select()

``:code

------

------

##### Reporting percentages

A special "word" encountered in the print statements of your functions clear all

the previous output. That word is ``!clear!``.

This, coupled with the ``sync_output`` parameter, allows to report percentages

-------

When web2py is upgraded it comes with a file called "NEWINSTALL". If web2py finds this file, it understands an upgrade was performed, hence it removed the file and creates a new "welcome.w2p".

The current web2py version is stored in the field "VERSION" and it follows standard semantic versioning notation where the build id is the build timestamp.

web2py unit-tests are in

``

gluon/tests/

``:code

There are handlers for connecting with various web servers:

``

cgihandler.py # discouraged

gaehandler.py # for Google App Engine

fcgihandler.py # for FastCGI

wsgihandler.py # for WSGI

isapiwsgihandler.py # for IIS

modpythonhandler.py # deprecated

``:code

web2py also contains a folder with useful scripts including

``

scripts/setup-web2py-fedora.sh

scripts/setup-web2py-ubuntu.sh

scripts/setup-web2py-nginx-uwsgi-ubuntu.sh

scripts/setup-web2py-heroku.sh

scripts/update-web2py.sh

scripts/make_min_web2py.py

...

scripts/sessions2trash.py

scripts/sync_languages.py

scripts/tickets2db.py

scripts/tickets2email.py

...

scripts/extract_mysql_models.py

scripts/extract_pgsql_models.py

...

scripts/access.wsgi

scripts/cpdb.py

``:code

Some of these are discussed in Chapter 14, but all of them contain a documentation string inside that explains their purpose and usage.

Finally web2py includes these files required to build the binary distributions.

``

Makefile

setup_exe.py

setup_app.py

``:code

These are setup scripts for **py2exe** and **py2app**, respectively, and they are only required to build the binary distributions of web2py. YOU SHOULD NEVER NEED TO RUN THEM.

web2py applications contain additional files, particularly third-party JavaScript libraries, such as jQuery, calendar, and Codemirror. Their authors are acknowledged in the files themselves.

### Applications

Applications developed in web2py are composed of the following parts:

- **models** describe a representation of the data as database tables and relations between tables.

- **controllers** describe the application logic and workflow.

- **views** describe how data should be presented to the user using HTML and JavaScript.

- **languages** describe how to translate strings in the application into various supported languages.

Now from a controller in "myapp" you can do

import test

def index():

return "Your ip is " + test.ip()

``

Notice a few things:

- ``import test`` looks for the module first in the current app's modules folder, then in the folders listed in ``sys.path``. Therefore, app-level modules always take precedence over Python modules. This allows different apps to ship with different versions of their modules, without conflicts.

- Different users can call the same action ``index`` concurrently, which calls the function in the module, and yet there is no conflict because ``current.request`` is a different object in different threads. Just be careful not to access ``current.request`` outside of functions or classes (i.e., at the top level) in the module.

- ``import test`` is a shortcut for ``from applications.appname.modules import test``. Using the longer syntax, it is possible to import modules from other applications.

For uniformity with normal Python behavior, by default web2py does not reload modules when changes are made. Yet this can be changed. To turn on the auto-reload feature for modules, use the ``track_changes`` function as follows (typically in a model file, before any imports):

``

from gluon.custom_import import track_changes; track_changes(True)

``:code

From now on, every time a module is imported, the importer will check if the Python source file (.py) has changed. If it has changed, the module will be reloaded.

Track changes only tracks changes for modules that are stored in the application.

Modules that import ``current`` can access:

- ``current.request``

- ``current.response``

- ``current.session``

- ``current.cache``

- ``current.T``

and any other variable your application chooses to store in current. For example a model could do

``

auth = Auth(db)

from gluon import current

current.auth = auth

``

``current`` and ``import`` create a powerful mechanism to build extensible and reusable modules for your applications.

-------

``

request = current.request # WRONG! DANGER!

``

nor one should use it assign class attributes

``

class MyClass:

request = current.request # WRONG! DANGER!

``

This is because the thread local object must be extracted at runtime. Global variables instead are defined only once when the model is imported for the first time.

-------

Another caveat has to do with cache. You cannot use the ``cache`` object to decorate functions in modules, that is because it would not behave as expected. In order to cache a function ``f`` in a module you must use ``lazy_cache``:

``

from gluon.cache import lazy_cache

lazy_cache('key', time_expire=60, cache_model='ram')

def f(a,b,c,): ....

``:code

The ``request`` object is an instance of the ubiquitous web2py class that is cal

request.vars

``:code

is the same as:

``

request['vars']

``:code

Unlike a dictionary, if an attribute (or key) does not exist, it does not raise an exception. Instead, it returns ``None``.

-----

It is sometimes useful to create your own Storage objects. You can do so as follows:

``

from gluon.storage import Storage

my_storage = Storage() # empty storage object

my_other_storage = Storage(dict(a=1, b=2)) # convert dictionary to Storage

``:code

-----

``request`` has the following items/attributes, some of which are also an instance of the ``Storage`` class:

- ``request.env``: a ``Storage`` object containing the environment variables passed to the controller, including HTTP header variables from the HTTP request and standard WSGI parameters. The environment variables are all converted to lower case, and dots are converted to underscores for easier memorization.

- ``request.extension``: the extension of the requested action. It defaults to "html". If the controller function returns a dictionary and does not specify a view, this is used to determine the extension of the view file that will render the dictionary (parsed from the ``request.env.path_info``).

- ``request.folder``: the application directory. For example if the application is "welcome", ``request.folder`` is set to the absolute path "/path/to/welcome". In your programs, you should always use this variable and the ``os.path.join`` function to build paths to the files you need to access. Although web2py always uses absolute paths, it is a good rule never to explicitly change the current working folder (whatever that is) since this is not a thread-safe practice.

- ``request.now``: a ``datetime.datetime`` object storing the datetime of the current request.

- ``request.utcnow``: a ``datetime.datetime`` object storing the UTC datetime of the current request.

- ``request.args``: A list of the URL path components following the controller function name; equivalent to ``request.env.path_info.split('/')[3:]``

- ``request.vars``: a ``gluon.storage.Storage`` object containing both the HTTP GET and HTTP POST query variables.

- ``request.get_vars``: a ``gluon.storage.Storage`` object containing only the HTTP GET query variables.

- ``request.post_vars``: a ``gluon.storage.Storage`` object containing only the HTTP POST query variables.

- ``request.client``: The ip address of the client as determined by, if present, ``request.env.http_x_forwarded_for`` or by ``request.env.remote_addr`` otherwise. While this is useful it should not be trusted because the ``http_x_forwarded_for`` can be spoofed.

- ``request.is_local``: ``True`` if the client is localhost, ``False`` otherwise. Should work behind a proxy if the proxy supports ``http_x_forwarded_for``.

- ``request.is_https``: ``True`` if the request is using the HTTPS protocol, ``False`` otherwise.

- ``request.body``: a read-only file stream that contains the body of the HTTP request. This is automatically parsed to get the ``request.post_vars`` and then rewinded. It can be read with ``request.body.read()``.

- ``request.ajax`` is True if the function is being called via an Ajax request.

- ``request.cid`` is the ``id`` of the component that generated the Ajax request (if any). You can read more about components in Chapter 12.

- ``request.requires_https()`` prevents further code execution if the request is not over HTTPS and redirects the visitor to the current page over HTTPS.

- ``request.restful`` this is a new and very useful decorator that can be used to change the default behavior of web2py actions by separating GET/POST/PUSH/DELETE requests. It will be discussed in some detail in Chapter 10.

- ``request.user_agent()`` parses the user_agent field from the client and returns the information in the form of a dictionary. It is useful to detect mobile devices. It uses "gluon/contrib/user_agent_parser.py" created by Ross Peoples. To see what it does, try to embed the following code in a view:

``

{{=BEAUTIFY(request.user_agent())}}

``:code

Also notice the ``request.env.wsgi_*`` variables. They are specific to the wsgi

``response.status``:inxx

``response.stream``:inxx

``response.subtitle``:inxx

``response.title``:inxx

``response.toolbar``:inxx

``response.view``:inxx

``response.delimiters``:inxx

``response.js``:inxx

``response.write``:inxx

``response.include_files``:inxx

``response.include_meta``:inxx

``response.optimize_css``:inxx

``response.optimize_js``:inxx

``response._caller``:inxx

``response`` is another instance of the ``Storage`` class. It contains the following:

- ``response.body``: a ``StringIO`` object into which web2py writes the output page body. NEVER CHANGE THIS VARIABLE.

- ``response.cookies``: similar to ``request.cookies``, but while the latter contains the cookies sent from the client to the server, the former contains cookies sent by the server to the client. The session cookie is handled automatically.

- ``response.download(request, db)``: a method used to implement the controller function that allows downloading of uploaded files. ``request.download`` expects the last ``arg`` in ``request.args`` to be the encoded filename (i.e., the filename generated at upload time and stored in the upload field). It extracts the upload field name and table name as well as the original filename from the encoded filename. ``response.download`` takes two optional arguments: ``chunk_size`` sets the size in bytes for chunked streaming (defaults to 64K), and ``attachments`` determines whether the downloaded file should be treated as an attachment or not (default to ``True``). Note, ``response.download`` is specifically for downloading files associated with ``db`` upload fields. Use ``response.stream`` (see below) for other types of file downloads and streaming. Also, note that it is not necessary to use ``response.download`` to access files uploaded to the /static folder -- static files can (and generally should) be accessed directly via URL (e.g., /app/static/files/myfile.pdf).

- ``response.include_files()`` generates html head tags to include all ``response.files`` (used in "views/web2py_ajax.html").

- ``response.flash``: optional parameter that may be included in the views. Normally used to notify the user about something that happened.

- ``response.headers``: a ``dict`` for HTTP response headers. Web2py sets some headers by default, including "Content-Length", "Content-Type", and "X-Powered-By" (set equal to web2py). Web2py also sets the "Cache-Control", "Expires", and "Pragma" headers to prevent client-side caching, except for static file requests, for which client-side caching is enabled. The headers that web2py sets can be overwritten or removed, and new headers can be added (e.g., ``response.headers['Cache-Control'] = 'private'``). You can remove a header by removing its key from the response.headers dict, e.g.``del response.headers['Custom-Header']``, however web2py's default headers will be re-added just before returning the response. To avoid this behavior, just set the header value to None, e.g. to remove the default Content-Type header, ``response.headers['Content-Type'] = None``

- ``response.menu``: optional parameter that may be included in the views, normally used to pass a navigation menu tree to the view. It can be rendered by the MENU helper.

- ``response.include_meta()`` generates a string that includes all ``response.meta`` headers serialized (used in "views/web2py_ajax.html").

- ``response.postprocessing``: this is a list of functions, empty by default. These functions are used to filter the response object at the output of an action, before the output is rendered by the view. It can be used to implement support for other template languages.

- ``response.render(view, vars)``: a method used to call the view explicitly inside the controller. ``view`` is an optional parameter which is the name of the view file, ``vars`` is a dictionary of named values passed to the view.

- ``response.session_file``: file stream containing the session.

- ``response.session_file_name``: name of the file where the session will be saved.

- ``response.session_id``: the id of the current session. It is determined automatically. NEVER CHANGE THIS VARIABLE.

- ``response.session_id_name``: the name of the session cookie for this application. NEVER CHANGE THIS VARIABLE.

- ``response.status``: the HTTP status code integer to be passed to the response. Default is 200 (OK).

- ``response.stream(file, chunk_size, request=request, attachment=False, filename=None, headers=None)``: when a controller returns it, web2py streams the file content back to the client in blocks of size ``chunk_size``. The ``request`` parameter is required to use the chunk start in the HTTP header. ``file`` should be a file path (for backward compatibility, it can also be an open file object, but this is not recommended). As noted above, ``response.download`` should be used to retrieve files stored via an upload field. ``response.stream`` can be used in other cases, such as returning a temporary file or StringIO object created by the controller. If ``attachment`` is True, the Content-Disposition header will be set to "attachment", and if ``filename`` is also provided, it will be added to the Content-Disposition header as well (but only when ``attachment`` is True). If not already included in ``response.headers``, the following response headers will be set automatically: Content-Type, Content-Length, Cache-Control, Pragma, and Last-Modified (the latter three are set to allow browser caching of the file). To override any of these automatic header settings, simply set them in ``response.headers`` before calling ``response.stream``.

- ``response.subtitle``: optional parameter that may be included in the views. It should contain the subtitle of the page.

- ``response.title``: optional parameter that may be included in the views. It should contain the title of the page and should be rendered by the HTML title TAG in the header.

- ``response.toolbar``: a function that allows you to embed a toolbar into page for debugging purposes ``{{=response.toolbar()}}``. The toolbar displays request, response, session variables and database access time for each query.

- ``response._vars``: this variable is accessible only in a view, not in the action. It contains the value returned by the action to the view.

- ``response._caller``: this is a function that wraps all action calls. It defaults to the identity function, but it can be modified in order to catch special types of exception to do extra logging;

``

response._caller = lambda f: f()

``

- ``response.optimize_css``: if can be set to "concat,minify,inline" to concatenate, minify and inline the CSS files included by web2py.

- ``response.optimize_js``: if can be set to "concat,minify,inline" to concatenate, minify and inline the JavaScript files included by web2py.

- ``response.view``: the name of the view template that must render the page. This is set by default to:

is simply a shortcut for:

raise HTTP(303,

'You are being redirected <a href="%s">here</a>' % location,

Location='http://www.web2py.com')

``:code

The named arguments of the ``HTTP`` initializer method are translated into HTTP header directives, in this case, the redirection target location. ``redirect`` takes an optional second argument, which is the HTTP status code for the redirection (303 by default). Change this number to 307 for a temporary redirect or to 301 for a permanent redirect.

The most common way to use redirect is to redirect to other pages in the same app and (optionally) pass parameters:

``

redirect(URL('index', args=(1,2,3), vars=dict(a='b')))

``:code

In chapter 12 we discuss web2py components. They make Ajax requests to web2py actions. If the called action performs a redirect, you may want the Ajax request to follow the redirect or you may want the entire page performing the Ajax request redirecting. In this latter case you can set:

``

redirect(...,type='auto')

``:code

``T``:inxx ``internationalization``:inxx

The object ``T`` is the language translator. It constitutes a single global instance of the web2py class ``gluon.language.translator``. All string constants (and only string constants) should be marked by ``T``, for example:

``

a = T("hello world")

``:code

Strings that are marked with ``T`` are identified by web2py as needing language translation and they will be translated when the code (in the model, controller, or view) is executed. If the string to be translated is not a constant but a variable, it will be added to the translation file at runtime (except on GAE) to be translated later.

The ``T`` object can also contain interpolated variables and supports multiple equivalent syntaxes:

``

a = T("hello %s", ('Tim',))

a = T("hello %(name)s", dict(name='Tim'))

a = T("hello %s") % ('Tim',)

a = T("hello %(name)s") % dict(name='Tim')

``:code

The latter syntax is recommended because it makes translation easier.

The first string is translated according to the requested language file and the ``name`` variable is replaced independently of the language.

T("Hello World", lazy=False)

``:code

A common issue is the following. The original application is in English. Suppose that there is a translation file (for example Italian, "it-it.py") and the HTTP client declares that it accepts both English (en) and Italian (it-it) in that order. The following unwanted situation occurs: web2py does not know the default is written in English (en). Therefore, it prefers translating everything into Italian (it-it) because it only found the Italian translation file. If it had not found the "it-it.py" file, it would have used the default language strings (English).

There are two solutions for this problem: create a translation language for English, which would be redundant and unnecessary, or better, tell web2py which languages should use the default language strings (the strings coded into the application). This can be done with:

``

T.set_current_languages('en', 'en-en')

``:code

It stores in ``T.current_languages`` a list of languages that do not require translation and forces a reload of the language files.

Notice that "it" and "it-it" are different languages from the point of view of web2py. To support both of them, one would need two translation files, always lower case. The same is true for all other languages.

The currently accepted language is stored in

``

T.accepted_language

``:code

#### Translating variables

``

>>> a="test"

>>> print T(a)

``:code

In this case the word "test" is translated but, if not found and if the filesystem is writable, it will add it to the list of words to be translated in the language file.

Notice that this can result in lots of file IO and you may want to disable it:

``

T.is_writable = False

``:code

prevents T from dynamically updating language files.

#### Comments and multiple translations

It is possible that the same string appears in different contexts in the application and needs different translations based on context. In order to do this, one can add comments to the original string. The comments will not be rendered but will be used by web2py to determine the most appropriate translation. For example:

``

T("hello world ## first occurrence")

T("hello world ## second occurrence")

``:code

The text following the ``##``, including the double ``##``, are comments.

#### Pluralization engine

Since version 2.0, web2py includes a powerful pluralization system (PS). This means that when text marked for translation depends on a numeric variable, it may be translated differently based on the numeric value. For example in English we may render:

``

x book(s)

``

with

``

``

English has one singular form and one plural form. The plural form is constructed by adding a "-s" or "-es" or using an exceptional form. web2py provides a way to define pluralization rules for each languages, as well as exceptions to the default rules. In fact web2py already knows pluralization rules for many languages. It knows, for example, that Slovenian has one singular form and 3 plural forms (for x==1, x==3 or x==4 and x>4). These rules are encoded in "gluon/contrib/plural_rules/*.py" files and new files can be created. Explicit pluralizations for words are created by editing pluralization files using the administrative interface.

By default the PS is not activated. It is triggered by the ``symbol`` argument of the ``T`` function. For example:

``

T("You have %s %%{book}", symbols=10)

``:code

Now the PS is activated for the word "book" and for the number 10.

The result in English will be: "You have 10 books". Notice that "book" has been pluralized into "books".

The PS consists of 3 parts:

- placeholders ``%%{}`` to mark words in ``T``-messages

- rule to give a decision which word form to use ("rules/plural_rules/*.py")

- dictionary with word plural forms ("app/languages/plural-*.py")

The value of symbols can be a single variable, a list/tuple of variables, or a dictionary.

The placeholder ``%%{}`` consists of 3 parts:

``

%%{[<modifier>]<world>[<parameter>]},

``

where:

``

<modifier>::= ! | !! | !!!

<word> ::= any word or phrase in singular in lower case (!)

<parameter> ::= [index] | (key) | (number)

``

For example:

- ``%%{word[index]}`` is used when symbols is a tuple. symbols[index] gives us a number used to make a decision on which word form to choose.

- ``%%{word(key)}`` is used to get the numeric parameter from symbols[key]

- ``%%{word(number)}`` allows to set a ``number`` directly (e.g.: ``%%{word(%i)}``)

- ``%%{?word?number}`` returns "word" if ``number==1``, returns the ``number`` otherwise

- ``%%{?number} or %%{??number}`` returns ``number`` if ``number!=1``, return nothing otherwise

``T("blabla %s %%{word}", symbols=var)``

``%%{word}`` by default means ``%%{word[0]}``,

where ``[0]`` is an item index in symbols tuple.

``T("blabla %s %s %%{word[1]}", (var1, var2))``

PS is used for "word" and var2 respectively.

``T("%%{this} %%{is} %s %%{book}", var)``

or

``T("%%{this[0]} %%{is[0]} %s %%{book[0]}", var)``

``

var output

------------------

1 this is 1 book

2 these are 2 books

3 these are 2 books

``

Similarly you can pass a dictionary to symbols:

``

T("blabla %(var1)s %(wordcnt)s %%{word(wordcnt)}",

dict(var1="tututu", wordcnt=20))

``

which produces

``

blabla tututu 20 words

For example

``T("%%{this} %%{is} %%{?a?%s} %%{book}", var)``

produces:

``

var output

------------------

1 this is a book

2 these are 2 books

3 these are 3 books

...

``

Inside ``%%{...}`` you can also use the following modifiers:

- ``!`` to capitalize the text (equivalent to ``string.capitalize``)

- ``!!`` to capitalize every word (equivalent to ``string.title``)

- ``!!!`` to capitalize every character (equivalent to ``string.upper``)

#### Translations, pluralization, and MARKMIN

You can also use the powerful MARKMIN syntax inside translation strings by replacing

``

T("hello world")

``:code

with

``

T.M("hello world")

``

Now the string accepts MARKMIN markup as described later in the book. You can also use the pluralization system inside MARKMIN.

### Cookies

``cookies``:inxx

Here are four ways to set the default application:

- Make a symbolic link from "applications/init" to your application's folder.

- Use URL rewrite as discussed in the next section.

### URL rewrite

``url rewrite``:inxx

``routes_in``:inxx

``routes_out``:inxx

web2py has the ability to rewrite the URL path of incoming requests prior to calling the controller action (URL mapping), and conversely, web2py can rewrite the URL path generated by the ``URL`` function (reverse URL mapping). One reason to do this is for handling legacy URLs, another is to simplify paths and make them shorter.

web2py includes two distinct URL rewrite systems: an easy-to-use ''parameter-based'' system for most use cases, and a flexible ''pattern-based'' system for more complex cases. To specify the URL rewrite rules, create a new file in the "web2py" folder called ``routes.py`` (the contents of ``routes.py`` will depend on which of the two rewrite systems you choose, as described in the next two sections). The two systems cannot be mixed.

-------

Notice that if you edit routes.py, you must reload it. This can be done in two ways: by restarting the web server or by clicking on the routes reload button in admin. If there is a bug in routes, they will not reload.

-------

#### Parameter-based system

The parameter-based (parametric) router provides easy access to several "canned" URL-rewrite methods. Its capabilities include:

The parametric router also provides somewhat more flexible validation of incoming URLs.

Suppose you've written an application called ``myapp`` and wish to make it the default, so that the application name is no longer part of the URL as seen by the user. Your default controller is still ``default``, and you want to remove its name from user-visible URLs as well. Here's what you put in ``routes.py``:

``

routers = dict(

BASE = dict(default_application='myapp'),

)

``:code

That's it. The parametric router is smart enough to know how to do the right thing with URLs such as:

``

http://domain.com/myapp/default/myapp

``:code

or

``

http://domain.com/myapp/myapp/index

``:code

where normal shortening would be ambiguous. If you have two applications, ``myapp`` and ``myapp2``, you'll get the same effect, and additionally ``myapp2``'s default controller will be stripped from the URL whenever it's safe (which is mostly all the time).

AliasMatch ^/([^/]+)/static/(?:/_[\d]+\.[\d]+\.[\d]+)?(.*) \

/home/www-data/web2py/applications/$1/static/$2

``

Similarly, in Nginx change this:

``

location ~* /(\w+)/static/ {

root /home/www-data/web2py/applications/;

expires max;

}

``

into this:

``

location ~* /(\w+)/static(?:/_[\d]+\.[\d]+\.[\d]+)?/(.*)$ {

alias /home/www-data/web2py/applications/$1/static/$2;

expires max;

}

``

### Running tasks in the background

The proper way to run time-consuming tasks is doing it in the background. There is not a single way of doing it, but here we describe three mechanisms that are built into web2py: **cron**, **homemade task queues**, and **scheduler**.

By **cron** we refer to a web2py functionality not to the Unix Cron mechanism. The web2py cron works on windows too.

web2py cron is the way to go if you need tasks in the background at scheduled times and these tasks take a relatively short time compared to the time interval between two calls. Each task runs in its own process, and multiple tasks can run concurrently, but you have no control over how many tasks run. If accidentally one task overlaps with itself, it can cause a database lock and a spike in memory usage.

web2py scheduler takes a different approach. The number of running processes is fixed, and they can run on different machines. Each process is called a worker. Each worker picks a task when available and executes it as soon as possible after the time when it is scheduled to run, but not necessarily at that exact time. There cannot be more processes running than the number of scheduled tasks and therefore no memory spikes. Scheduler tasks can be defined in models and are stored in the database. The web2py scheduler does not implement a distributed queue since it assumes that the time to distribute tasks is negligible compared with the time to run the tasks. Workers pick up the task from the database.

Homemade tasks queues can be a simpler alternative to the web2py scheduler in some cases.

#### Cron

``cron``:inxx

The web2py cron provides the ability for applications to execute tasks at preset times, in a platform-independent manner.

For each application, cron functionality is defined by a crontab file:

``

app/cron/crontab

Example of line to add to the system crontab, (usually /etc/crontab):

With external ``cron``, make sure to add either ``-J`` (or ``--cronjob``, which is the same) as indicated above so that web2py knows that task is executed by cron. Web2py sets this internally with soft and hard ``cron``.

#### Homemade task queues

While cron is useful to run tasks at regular time intervals, it is not always the best solution to run a background task. For this purpose web2py provides the ability to run any python script as if it were inside a controller:

``

python web2py.py -S app -M -R applications/app/private/myscript.py -A a b c

``:code

where ``-S app`` tells web2py to run "myscript.py" as "app", ``-M`` tells web2py to execute models, and ``-A a b c`` passes optional command line arguments ``sys.args=['a','b','c']`` to "myscript.py".

This type of background process should not be executed via cron (except perhaps for cron @reboot) because you need to be sure that no more than one instance is running at the same time. With cron it is possible that a process starts at cron iteration 1 and is not completed by cron iteration 2, so cron starts it again, and again, and again - thus jamming the mail server.

In chapter 8, we will provide an example of how to use the above method to send emails.

#### Scheduler (experimental)

The web2py scheduler works very much like the task queue described in the previous sub-section with some differences:

- There is not a single background process but a set of workers processes.

- The job of worker nodes can be monitored because their state, as well as the state of the tasks, is stored in the database.

- It works without web2py but that is not documented here.

The scheduler does not use cron, although one can use cron @reboot to start the worker nodes.

More information about deploying the scheduler under Linux and Windows is in the Deployment recipes chapter.

In the scheduler, a task is simply a function defined in a model (or in a module and imported by a model). For example:

``

def task_add(a,b):

return a+b

``:code

Tasks will always be called in the same environment seen by controllers and therefore they see all the global variables defined in models, including database connections (``db``). Tasks differ from a controller action because they are not associated with an HTTP request and therefore there is no ``request.env``.

------

------

The recommended way to enable the scheduler to your app is to create a model file named ``scheduler.py`` and define your function there. After the functions, you can put the following code into the model:

``

from gluon.scheduler import Scheduler

scheduler = Scheduler(db)

``:code

The task is scheduled with

``

scheduler.queue_task(task_add,pvars=dict(a=1,b=2))

``:code

##### Parameters

The first argument of the ``Scheduler`` class must be the database to be used by the scheduler to communicate with the workers. This can be the ``db`` of the app or another dedicated ``db``, perhaps one shared by multiple apps. If you use SQLite it's recommended to use a separate db from the one used by your app in order to keep the app responsive.

Once the tasks are defined and the ``Scheduler`` is instantiated, all that is needed to do is to start the workers. You can do that in several ways:

``

python web2py.py -K myapp

``

starts a worker for the app ``myapp``. If you want start multiple workers for the same app, you can do so just passing ``myapp,myapp``. You can pass also the ``group_names`` (overriding the one set in your model) with

``

python web2py.py -K myapp:group1:group2,myotherapp:group1

You can pass the usual parameters (-i, -p, here -a prevents the window from show

Scheduler's complete signature is:

``

Scheduler(

db,

tasks=None,

migrate=True,

worker_name=None,

group_names=None,

heartbeat=HEARTBEAT,

max_empty_runs=0,

discard_results=False,

utc_time=False

)

``:code

Let's see them in order:

- ``db`` is the database DAL instance where you want the scheduler tables be placed.

- ``worker_name`` is None by default. As soon as the worker is started, a worker name is generated as hostname-uuid. If you want to specify that, be sure that it's unique.

- ``group_names`` is by default set to **[main]**. All tasks have a ``group_name`` parameter, set to **main** by default. Workers can only pick up tasks of their assigned group.

------

NB: This is useful if you have different workers instances (e.g. on different machines) and you want to assign tasks to a specific worker.

NB2: It's possible to assign a worker more groups, and they can be also all the same, as

------

``['mygroup','mygroup']``. Tasks will be distributed taking into consideration that

a worker with group_names ``['mygroup','mygroup']`` is able to process the double of the tasks

a worker with group_names ``['mygroup']`` is.

- ``heartbeat`` is by default set to 3 seconds. This parameter is the one controlling how often a scheduler will check its status on the ``scheduler_worker`` table and see if there are any **ASSIGNED** tasks to itself to process.

- ``max_empty_runs`` is 0 by default, that means that the worker will continue to process tasks as soon as they are **ASSIGNED**. If you set this to a value of, let's say, 10, a worker will die automatically if it's **ACTIVE** and no tasks are **ASSIGNED** to it for 10 loops. A loop is when a worker searches for tasks, every 3 seconds (or the set ``heartbeat``)

- ``discard_results`` is False by default. If set to True, no scheduler_run records will be created.

------

NB: scheduler_run records will be created as before for **FAILED**, **TIMEOUT** and **STOPPED** tasks's statuses.

------

- ``utc_time`` is False by default. If you need to coordinate with workers living in different timezones, or don't have problems with solar/DST times, supplying datetimes from different countries, etc, you can set this to True. The scheduler will honor the UTC time and work leaving the local time aside. Caveat: you need to schedule tasks with UTC times (for start_time, stop_time, and so on.)

http://127.0.0.1:8000/myapp/appadmin/insert/db/scheduler_task

The meaning of the fields in this table is obvious. The "args" and "vars"" fields are the values to be passed to the task in JSON format. In the case of the "task_add" above, an example of "args" and "vars" could be:

``

args = [3, 4]

vars = {}

``:code

or

``

args = []

vars = {'a':3, 'b':4}

``:code

The ``scheduler_task`` table is the one where tasks are organized.

All tasks follow a lifecycle

[[scheduler tasks @///image/ce8edcc3.png center]]

If you need it to be executed later, use the ``start_time`` parameter (default = now).

that shuts down at 1AM, a mail that needs to be sent not after the working hours, etc...) you can set a ``stop_time`` (default = None) for it.

**RUNNING** tasks may end up:

Values for ``start_time`` and ``stop_time`` should be datetime objects. To schedule "mytask" to run at 30 seconds from the current time, for example, you would do the following:

``

from datetime import timedelta as timed

scheduler.queue_task('mytask',

start_time=request.now + timed(seconds=30))

``:code

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

------

------

[[task repeats @///image/7d8b85e4.png center]]

Summary: you have

- ``period`` and ``repeats`` to get an automatically rescheduled function

- ``timeout`` to be sure that a function doesn't exceed a certain amount of time

- ``retry_failed`` to control how many times the task can "fail"

- ``start_time`` and ``stop_time`` to schedule a function in a restricted timeframe

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

When a ``RUNNING`` task throws an exception, the run is mark as ``FAILED`` and the task is marked as ``FAILED``. The traceback is stored in the run record.

Similarly, when a run exceeds the timeout, it is stopped and marked as ``TIMEOUT``, and the task is marked as ``TIMEOUT``.

In any case, the stdout is captured and also logged into the run record.

Using appadmin, one can check all ``RUNNING`` tasks, the output of ``COMPLETED`` tasks, the error of ``FAILED`` tasks, etc.

The scheduler also creates one more table called "scheduler_worker", which stores the workers' heartbeat and their status. Possible worker statuses are:

##### Managing processes

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

- kill it "no matter what it's doing"

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

its behavior.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

[[workers statuses @///image/bd891eed.png center]]

There are a few commodity functions since version 2.4.1 (self-explanatory)

``

scheduler.disable()

scheduler.resume()

scheduler.terminate()

scheduler.kill()

``:code

each function take an optional parameter, that can be a string or a list, to manage workers based on their ``group_names``. It defaults to the ``group_names`` defined in the scheduler istantiation.

An example is better than a thousand words: ``scheduler.terminate('high_prio')`` will TERMINATE all the workers that are processing the ``high_prio`` tasks, while ``scheduler.terminate(['high_prio', 'low_prio'])`` will terminate all ``high_prio`` and ``low_prio`` workers.

------

Watch out: if you have a worker processing ``high_prio`` and ``low_prio``, ``scheduler.terminate('high_prio')`` will terminate the worker alltogether, even if you didn't want to terminate ``low_prio`` too.

For example:

``

scheduler.queue_task(

function_name='task_add',

pargs=[],

pvars={'a':3,'b':4},

repeats = 10, # run 10 times

period = 3600, # every 1h

timeout = 120, # should take less than 120 seconds

)

``:code

Notice that fields "times_run", "last_run_time" and "assigned_worker_name" are not provided at schedule time but are filled automatically by the workers.

You can also retrieve the output of completed tasks:

``

completed_runs = db(db.scheduler_run.run_status='COMPLETED').select()

``:code

------

------

##### Reporting percentages

A special "word" encountered in the print statements of your functions clear all

the previous output. That word is ``!clear!``.

This, coupled with the ``sync_output`` parameter, allows to report percentages

Modules that import ``current`` can access:

- ``current.request``

- ``current.response``

- ``current.session``

- ``current.cache``

- ``current.T``

and any other variable your application chooses to store in current. For example a model could do

Modules that import ``current`` can access:

- ``current.request``

- ``current.response``

- ``current.session``

- ``current.cache``

- ``current.T``

and any other variable your application chooses to store in current. For example a model could do

The method returns the result of validate_and_insert, with the ``uuid`` of the task you queued (can be the one you passed or the auto-generated one).

``<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>``

If there are errors (e.g. you used ``period = 'a'``), you'll get the result of the validation, and id and uuid will be None

``<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>``

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

The scheduler also creates one more table called "scheduler_worker", which store

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

When you start a worker, you may want later to:

- kill it "no matter what it's doing"

- kill it only if it's not processing tasks

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

To be more precise, for started workers you will change the ``status`` value of any worker to influence

its behavior.

As tasks, workers can be in some fixed statuses : ACTIVE, DISABLED, TERMINATE or KILLED.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

[[workers statuses @///image/bd891eed.png center]]

Everything that one can do via appadmin one can do programmatically by inserting and updating records in these tables.

The method returns the result of validate_and_insert, with the ``uuid`` of the task you queued (can be the one you passed or the auto-generated one).

``<Row {'errors': {}, 'id': 1, 'uuid': '08e6433a-cf07-4cea-a4cb-01f16ae5f414'}>``

If there are errors (e.g. you used ``period = 'a'``), you'll get the result of the validation, and id and uuid will be None

``<Row {'errors': {'period': 'enter an integer greater than or equal to 0'}, 'id': None, 'uuid': None}>``

##### Results and output

The table "scheduler_run" stores the status of all running tasks. Each record references a task that has been picked up by a worker. One task can have multiple runs. For example, a task scheduled to repeat 10 times an hour will probably have 10 runs (unless one fails or they take longer than 1 hour). Beware that if the task has no return values, it is removed from the scheduler_run table as soon as it is finished.

Possible run statuses are:

``

RUNNING, COMPLETED, FAILED, TIMEOUT

``

If the run is completed, no exceptions are thrown, and there is no task timeout, the run is marked as ``COMPLETED`` and the task is marked as ``QUEUED`` or ``COMPLETED`` depending on whether it is supposed to run again at a later time. The output of the task is serialized in JSON and stored in the run record.

The scheduler also creates one more table called "scheduler_worker", which store

Worker fine management is hard. This module tries not to leave behind any platform (Mac, Win, Linux) .

When you start a worker, you may want later to:

- kill it "no matter what it's doing"

- kill it only if it's not processing tasks

- put it to sleep

Maybe you have yet some tasks queued, and you want to save some resources.

You know you want them processed every hour, so, you'll want to:

- process all queued tasks and die automatically

All of these things are possible managing ``Scheduler`` parameters or the ``scheduler_worker`` table.

To be more precise, for started workers you will change the ``status`` value of any worker to influence

its behavior.

As tasks, workers can be in some fixed statuses : ACTIVE, DISABLED, TERMINATE or KILLED.

**ACTIVE** and **DISABLED** are "persistent", while **TERMINATE** or **KILL**, as statuses

name suggest, are more "commands" than real statuses.

Hitting ctrl+c is equal to set a worker to **KILL**

[[workers statuses @///image/bd891eed.png center]]

Everything that one can do via appadmin one can do programmatically by inserting and updating records in these tables.

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

Additionally, you can control how many times a task should be repeated (i.e. you need to aggregate some data at specified intervals). To do so, set the ``repeats``

parameter (default = 1 time only, 0 = unlimited). You can influence how many seconds should pass between executions with the ``period`` parameter (default = 60 seconds).

The web2py workflow is the following:

- An HTTP requests arrives to the web server (the built-in Rocket server or a different server connected to web2py via WSGI or another adapter). The web server handles each request in its own thread, in parallel.

- The HTTP request header is parsed and passed to the dispatcher (explained later in this chapter).

- The dispatcher decides which of the installed application will handle the request and maps the PATH_INFO in the URL into a function call. Each URL corresponds to one function call.

- Requests for files in the static folder are handled directly, and large files are automatically streamed to the client.

- Requests for anything but a static file are mapped into an action (i.e. a function in a controller file, in the requested application).

- Before calling the action, a few things happen: if the request header contains a session cookie for the app, the session object is retrieved; if not, a session id is created (but the session file is not saved until later); an execution environment for the request is created; models are executed in this environment.

- Finally the controller action is executed in the pre-built environment.

- If the action returns a string, this is returned to the client (or if the action returns a web2py HTML helper object, it is serialized and returned to the client).

- If the action returns an iterable, this is used to loop and stream the data to the client.

- If the action returns a dictionary, web2py tries to locate a view to render the dictionary. The view must have the same name as the action (unless specified otherwise) and the same extension as the requested page (defaults to .html); on failure, web2py may pick up a generic view (if available and if enabled). The view sees every variable defined in the models as well as those in the dictionary returned by the action, but does not see global variables defined in the controller.

- If the user code succeeds, the transaction is committed.

- If the user code fails, the traceback is stored in a ticket, and a ticket ID is issued to the client. Only the system administrator can search and read the tracebacks in tickets.

There are some caveats to keep in mind:

- Models in the same folder/subfolder are executed in alphabetical order.

- Any variable defined in a model will be visible to other models following alphabetically, to the controllers, and to the views.

- Models in subfolders are executed conditionally. For example, if the user has requested "/a/c/f" where "a" is the application, "c" is the controller, and "f" is the function (action), then the following models are executed:

``

applications/a/models/*.py

applications/a/models/c/*.py

applications/a/models/c/f/*.py

``

- The requested controller is executed and the requested function is called. This means all top-level code in the controller is also executed at every request for that controller.

- The view is only called if the action returns a dictionary.

- If a view is not found, web2py tries to use a generic view. By default, generic views are disabled, although the 'welcome' app includes a line in /models/db.py to enable them on localhost only. They can be enabled per extension type and per action (using ``response.generic_patterns``). In general, generic views are a development tool and typically should not be used in production. If you want some actions to use a generic view, list those actions in ``response.generic_patterns`` (discussed in more detail in the chapter on Services).

The possible behaviors of an action are the following:

Notice that web2py validates all URLs to prevent directory traversal attacks.

URLs are only allowed to contain alphanumeric characters, underscores, and slashes; the ``args`` may contain non-consecutive dots. Spaces are replaced by underscores before validation. If the URL syntax is invalid, web2py returns an HTTP 400 error message``http-w``:cite ``http-o``:cite .

If the URL corresponds to a request for a static file, web2py simply reads and returns (streams) the requested file.

If the URL does not request a static file, web2py processes the request in the following order:

- Parses cookies.

- Creates an environment in which to execute the function.

- Initializes ``request``, ``response``, ``cache``.

- Opens the existing ``session`` or creates a new one.

- Executes the models belonging to the requested application.

- Executes the requested controller action function.

- If the function returns a dictionary, executes the associated view.

- On success, commits all open transactions.

- Saves the session.

- Returns an HTTP response.

Notice that the controller and the view are executed in different copies of the same environment; therefore, the view does not see the controller, but it sees the models and it sees the variables returned by the controller action function.

If an exception (other than HTTP) is raised, web2py does the following:

- Stores the traceback in an error file and assigns a ticket number to it.

- Returns an error page reporting the ticket number.

If the exception is an ``HTTP`` exception, this is assumed to be the intended behavior (for example, an ``HTTP`` redirect), and all open database transactions are committed. The behavior after that is specified by the ``HTTP`` exception itself. The ``HTTP`` exception class is not a standard Python exception; it is defined by web2py.

### Libraries

The web2py libraries are exposed to the user applications as global objects. For example (``request``, ``response``, ``session``, ``cache``), classes (helpers, validators, DAL API), and functions (``T`` and ``redirect``).

These objects are defined in the following core files:

``

web2py.py

gluon/__init__.py gluon/highlight.py gluon/restricted.py gluon/streamer.py

gluon/admin.py gluon/html.py gluon/rewrite.py gluon/template.py

gluon/cache.py gluon/http.py gluon/rocket.py gluon/storage.py

gluon/cfs.py gluon/import_all.py gluon/sanitizer.py gluon/tools.py

gluon/compileapp.py gluon/languages.py gluon/serializers.py gluon/utils.py

gluon/contenttype.py gluon/main.py gluon/settings.py gluon/validators.py

gluon/dal.py gluon/myregex.py gluon/shell.py gluon/widget.py

gluon/decoder.py gluon/newcron.py gluon/sql.py gluon/winservice.py

gluon/fileutils.py gluon/portalocker.py gluon/sqlhtml.py gluon/xmlrpc.py

gluon/globals.py gluon/reserved_sql_keywords.py

``:code

------

Notice that many of these modules, specifically ``dal`` (the Database Abstraction Layer), ``template`` (the template language), ``rocket`` (the web server), and

``html`` (the helpers) have no dependencies and can be used outside of web2py.

-----

The tar gzipped scaffolding app that ship with web2py is

``

welcome.w2p

``:code

This is created upon installation and overwritten on upgrade.

-------

The first time you start web2py, two new folders are created: deposit and applications. The deposit folder is used as temporary storage for installing and uninstalling applications.

Also, the first time you start web2py and after an upgrade, the "welcome" app is zipped into a "welcome.w2p" file to be used as a scaffolding app.

-------

The current web2py version is stored in the field "VERSION" and it follows standard semantic versioning notation where the build id is the build timestamp.

web2py unit-tests are in

``

gluon/tests/

``:code

There are handlers for connecting with various web servers:

``

cgihandler.py # discouraged

gaehandler.py # for Google App Engine

fcgihandler.py # for FastCGI

wsgihandler.py # for WSGI

isapiwsgihandler.py # for IIS

modpythonhandler.py # deprecated

``:code

("fcgihandler" calls "gluon/contrib/gateways/fcgi.py" developed by Allan Saddi) and

gluon/contrib/rss2.py

``

gluon/contrib/simplejson/

``:code

**Google Wallet** ``googlewallet``:cite

provides "pay now" buttons which link Google as payment processor:

``

gluon/contrib/google_wallet.py

``:code

**Stripe.com** ``stripe``:cite provides a simple API for accepting credit card payments:

``

gluon/contrib/stripe.py

``:code

**AuthorizeNet** ``authorizenet``:cite provides API to accept credit card payments via Authorize.net network

``

gluon/contrib/AuthorizeNet.py

``:code

``

gluon/contrib/DowCommerce.py

``:code

``

gluon/contrib/paymentech.py

``:code

**PAM**``PAM``:cite authentication API created by Chris AtLee:

``

gluon/contrib/pam.py

``:code

A Bayesian classifier to populate the database with dummy data for testing purposes:

``

gluon/contrib/populate.py

``:code

A file with API for running on Heroku.com : ``heroku``:inxx

``

gluon/contrib/heroku.py

``:code

Applications developed in web2py are composed of the following parts:

- **databases** store SQLite databases and additional table information.

- **cache** store cached application items.

- **modules** are other optional Python modules.

- **private** files are accessed by the controllers but not directly by the developer.

- **uploads** files are accessed by the models but not directly by the developer (e.g., files uploaded by users of the application).

- **tests** is a directory for storing test scripts, fixtures and mocks.

Models, views, controllers, languages, and static files are accessible via the web administration [design] interface. ABOUT, README, and errors are also accessible via the administration interface through the corresponding menu items. Sessions, cache, modules and private files are accessible to the applications but not via the administration interface.

Everything is neatly organized in a clear directory structure that is replicated for every installed web2py application, although the user never needs to access the filesystem directly:

``about``:inxx ``license``:inxx ``cache``:inxx ``controllers``:inxx ``databases``:inxx ``errors``:inxx ``languages``:inxx ``models``:inxx ``modules``:inxx ``private``:inxx ``session``:inxx ``static``:inxx ``tests``:inxx ``uploads``:inxx ``views``:inxx ``__init__.py``:inxx

``

__init__.py ABOUT LICENSE models views

controllers modules private tests cron

cache errors upload sessions static

``:code

"__init__.py" is an empty file which is required in order to allow Python (and web2py) to import the modules in the ``modules`` directory.

### API

Models, controllers, and views are executed in an environment where the following objects are already imported for us:

**Global Objects:** ``request``:inxx ``response``:inxx ``session``:inxx ``cache``:inxx

``

request, response, session, cache

``:code

**Internationalization:** ``T``:inxx ``internationalization``:inxx

``

T

``:code

**Navigation:** ``redirect``:inxx ``HTTP``:inxx

``

IS_UPLOAD_FILENAME, IS_UPPER, IS_URL

**Database:** ``DAL``:inxx

``

DAL, Field

``:code

For backward compatibility ``SQLDB=DAL`` and ``SQLField=Field``. We encourage you to use the new syntax ``DAL`` and ``Field``, instead of the old syntax.

Other objects and modules are defined in the libraries, but they are not automatically imported since they are not used as often.

The core API entities in the web2py execution environment are ``request``, ``response``, ``session``, ``cache``, ``URL``, ``HTTP``, ``redirect`` and ``T`` and are discussed below.

A few objects and functions, including **Auth**, **Crud** and **Service**, are defined in "gluon/tools.py" and they need to be imported as necessary:

``

from gluon.tools import Auth, Crud, Service

``:code

#### Accessing the API from Python modules

``

from gluon import *

``

In fact, any Python module, even if not imported by a web2py application, can import the web2py API as long as web2py is in the ``sys.path``.

There is one caveat, though. Web2py defines some global objects (request, response, session, cache, T) that can only exist when an HTTP request is present (or is faked). Therefore, modules can access them only if they are called from an application. For this reasons they are placed into a container caller ``current``, which is a thread local object. Here is an example.

Create a module "/myapp/modules/test.py" that contains:

``

from gluon import *

def ip(): return current.request.client

``

Now from a controller in "myapp" you can do

``

import test

def index():