Django gives you a few ways to control how database transactions are managed.
Managing database transactions¶
Django’s default transaction behavior¶
Django’s default behavior is to run in autocommit mode. Each query is immediately committed to the database, unless a transaction is active. See below for details.
Django’s TestCase class also wraps each test in a transaction for performance reasons.
Previous version of Django featured a more complicated default behavior.
Tying transactions to HTTP requests¶
A common way to handle transactions on the web is to wrap each request in a transaction. Set ATOMIC_REQUESTS to True in the configuration of each database for which you want to enable this behavior.
It works like this. Before calling a view function, Django starts a transaction. If the response is produced without problems, Django commits the transaction. If the view produces an exception, Django rolls back the transaction.
You may perfom partial commits and rollbacks in your view code, typically with the atomic() context manager. However, at the end of the view, either all the changes will be committed, or none of them.
While the simplicity of this transaction model is appealing, it also makes it inefficient when traffic increases. Opening a transaction for every view has some overhead. The impact on performance depends on the query patterns of your application and on how well your database handles locking.
Per-request transactions and streaming responses
When a view returns a StreamingHttpResponse, reading the contents of the response will often execute code to generate the content. Since the view has already returned, such code runs outside of the transaction.
Generally speaking, it isn’t advisable to write to the database while generating a streaming response, since there’s no sensible way to handle errors after starting to send the response.
In practice, this feature simply wraps every view function in the atomic() decorator described below.
Note that only the execution of your view is enclosed in the transactions. Middleware runs outside of the transaction, and so does the rendering of template responses.
When ATOMIC_REQUESTS is enabled, it’s still possible to prevent views from running in a transaction.
This decorator will negate the effect of ATOMIC_REQUESTS for a given view:
from django.db import transaction @transaction.non_atomic_requests def my_view(request): do_stuff() @transaction.non_atomic_requests(using='other') def my_other_view(request): do_stuff_on_the_other_database()
It only works if it’s applied to the view itself.
Django used to provide this feature via TransactionMiddleware, which is now deprecated.
Controlling transactions explicitly¶
Django provides a single API to control database transactions.
- atomic(using=None, savepoint=True)¶
Atomicity is the defining property of database transactions. atomic allows us to create a block of code within which the atomicity on the database is guaranteed. If the block of code is successfully completed, the changes are committed to the database. If there is an exception, the changes are rolled back.
atomic blocks can be nested. In this case, when an inner block completes successfully, its effects can still be rolled back if an exception is raised in the outer block at a later point.
atomic is usable both as a decorator:
from django.db import transaction @transaction.atomic def viewfunc(request): # This code executes inside a transaction. do_stuff()
and as a context manager:
from django.db import transaction def viewfunc(request): # This code executes in autocommit mode (Django's default). do_stuff() with transaction.atomic(): # This code executes inside a transaction. do_more_stuff()
Wrapping atomic in a try/except block allows for natural handling of integrity errors:
from django.db import IntegrityError, transaction @transaction.atomic def viewfunc(request): create_parent() try: with transaction.atomic(): generate_relationships() except IntegrityError: handle_exception() add_children()
In this example, even if generate_relationships() causes a database error by breaking an integrity constraint, you can execute queries in add_children(), and the changes from create_parent() are still there. Note that any operations attempted in generate_relationships() will already have been rolled back safely when handle_exception() is called, so the exception handler can also operate on the database if necessary.
Avoid catching exceptions inside atomic!
When exiting an atomic block, Django looks at whether it’s exited normally or with an exception to determine whether to commit or roll back. If you catch and handle exceptions inside an atomic block, you may hide from Django the fact that a problem has happened. This can result in unexpected behavior.
This is mostly a concern for DatabaseError and its subclasses such as IntegrityError. After such an error, the transaction is broken and Django will perform a rollback at the end of the atomic block. If you attempt to run database queries before the rollback happens, Django will raise a TransactionManagementError. You may also encounter this behavior when an ORM-related signal handler raises an exception.
The correct way to catch database errors is around an atomic block as shown above. If necessary, add an extra atomic block for this purpose. This pattern has another advantage: it delimits explicitly which operations will be rolled back if an exception occurs.
If you catch exceptions raised by raw SQL queries, Django’s behavior is unspecified and database-dependent.
In order to guarantee atomicity, atomic disables some APIs. Attempting to commit, roll back, or change the autocommit state of the database connection within an atomic block will raise an exception.
atomic takes a using argument which should be the name of a database. If this argument isn’t provided, Django uses the "default" database.
Under the hood, Django’s transaction management code:
- opens a transaction when entering the outermost atomic block;
- creates a savepoint when entering an inner atomic block;
- releases or rolls back to the savepoint when exiting an inner block;
- commits or rolls back the transaction when exiting the outermost block.
You can disable the creation of savepoints for inner blocks by setting the savepoint argument to False. If an exception occurs, Django will perform the rollback when exiting the first parent block with a savepoint if there is one, and the outermost block otherwise. Atomicity is still guaranteed by the outer transaction. This option should only be used if the overhead of savepoints is noticeable. It has the drawback of breaking the error handling described above.
You may use atomic when autocommit is turned off. It will only use savepoints, even for the outermost block, and it will raise an exception if the outermost block is declared with savepoint=False.
Open transactions have a performance cost for your database server. To minimize this overhead, keep your transactions as short as possible. This is especially important if you’re using atomic() in long-running processes, outside of Django’s request / response cycle.
Why Django uses autocommit¶
In the SQL standards, each SQL query starts a transaction, unless one is already active. Such transactions must then be explicitly committed or rolled back.
This isn’t always convenient for application developers. To alleviate this problem, most databases provide an autocommit mode. When autocommit is turned on and no transaction is active, each SQL query gets wrapped in its own transaction. In other words, not only does each such query start a transaction, but the transaction also gets automatically committed or rolled back, depending on whether the query succeeded.
PEP 249, the Python Database API Specification v2.0, requires autocommit to be initially turned off. Django overrides this default and turns autocommit on.
To avoid this, you can deactivate the transaction management, but it isn’t recommended.
Before Django 1.6, autocommit was turned off, and it was emulated by forcing a commit after write operations in the ORM.
Deactivating transaction management¶
You can totally disable Django’s transaction management for a given database by setting AUTOCOMMIT to False in its configuration. If you do this, Django won’t enable autocommit, and won’t perform any commits. You’ll get the regular behavior of the underlying database library.
This requires you to commit explicitly every transaction, even those started by Django or by third-party libraries. Thus, this is best used in situations where you want to run your own transaction-controlling middleware or do something really strange.
This used to be controlled by the TRANSACTIONS_MANAGED setting.
Always prefer atomic() if possible at all. It accounts for the idiosyncrasies of each database and prevents invalid operations.
The low level APIs are only useful if you’re implementing your own transaction management.
Django provides a straightforward API in the django.db.transaction module to manage the autocommit state of each database connection.
- set_autocommit(autocommit, using=None)¶
These functions take a using argument which should be the name of a database. If it isn’t provided, Django uses the "default" database.
Autocommit is initially turned on. If you turn it off, it’s your responsibility to restore it.
Once you turn autocommit off, you get the default behavior of your database adapter, and Django won’t help you. Although that behavior is specified in PEP 249, implementations of adapters aren’t always consistent with one another. Review the documentation of the adapter you’re using carefully.
Django will refuse to turn autocommit off when an atomic() block is active, because that would break atomicity.
A transaction is an atomic set of database queries. Even if your program crashes, the database guarantees that either all the changes will be applied, or none of them.
Django doesn’t provide an API to start a transaction. The expected way to start a transaction is to disable autocommit with set_autocommit().
Once you’re in a transaction, you can choose either to apply the changes you’ve performed until this point with commit(), or to cancel them with rollback(). These functions are defined in django.db.transaction.
These functions take a using argument which should be the name of a database. If it isn’t provided, Django uses the "default" database.
Django will refuse to commit or to rollback when an atomic() block is active, because that would break atomicity.
A savepoint is a marker within a transaction that enables you to roll back part of a transaction, rather than the full transaction. Savepoints are available with the SQLite (≥ 3.6.8), PostgreSQL, Oracle and MySQL (when using the InnoDB storage engine) backends. Other backends provide the savepoint functions, but they’re empty operations – they don’t actually do anything.
Savepoints aren’t especially useful if you are using autocommit, the default behavior of Django. However, once you open a transaction with atomic(), you build up a series of database operations awaiting a commit or rollback. If you issue a rollback, the entire transaction is rolled back. Savepoints provide the ability to perform a fine-grained rollback, rather than the full rollback that would be performed by transaction.rollback().
When the atomic() decorator is nested, it creates a savepoint to allow partial commit or rollback. You’re strongly encouraged to use atomic() rather than the functions described below, but they’re still part of the public API, and there’s no plan to deprecate them.
Each of these functions takes a using argument which should be the name of a database for which the behavior applies. If no using argument is provided then the "default" database is used.
Savepoints are controlled by three functions in django.db.transaction:
Creates a new savepoint. This marks a point in the transaction that is known to be in a “good” state. Returns the savepoint ID (sid).
- savepoint_commit(sid, using=None)¶
Releases savepoint sid. The changes performed since the savepoint was created become part of the transaction.
- savepoint_rollback(sid, using=None)¶
Rolls back the transaction to savepoint sid.
These functions do nothing if savepoints aren’t supported or if the database is in autocommit mode.
In addition, there’s a utility function:
Resets the counter used to generate unique savepoint IDs.
The following example demonstrates the use of savepoints:
from django.db import transaction # open a transaction @transaction.atomic def viewfunc(request): a.save() # transaction now contains a.save() sid = transaction.savepoint() b.save() # transaction now contains a.save() and b.save() if want_to_keep_b: transaction.savepoint_commit(sid) # open transaction still contains a.save() and b.save() else: transaction.savepoint_rollback(sid) # open transaction now contains only a.save()
Savepoints may be used to recover from a database error by performing a partial rollback. If you’re doing this inside an atomic() block, the entire block will still be rolled back, because it doesn’t know you’ve handled the situation at a lower level! To prevent this, you can control the rollback behavior with the following functions.
- set_rollback(rollback, using=None)¶
Setting the rollback flag to True forces a rollback when exiting the innermost atomic block. This may be useful to trigger a rollback without raising an exception.
Setting it to False prevents such a rollback. Before doing that, make sure you’ve rolled back the transaction to a known-good savepoint within the current atomic block! Otherwise you’re breaking atomicity and data corruption may occur.
Savepoints in SQLite¶
While SQLite ≥ 3.6.8 supports savepoints, a flaw in the design of the sqlite3 module makes them hardly usable.
When autocommit is enabled, savepoints don’t make sense. When it’s disabled, sqlite3 commits implicitly before savepoint statements. (In fact, it commits before any statement other than SELECT, INSERT, UPDATE, DELETE and REPLACE.) This bug has two consequences:
Transactions in MySQL¶
If you’re using MySQL, your tables may or may not support transactions; it depends on your MySQL version and the table types you’re using. (By “table types,” we mean something like “InnoDB” or “MyISAM”.) MySQL transaction peculiarities are outside the scope of this article, but the MySQL site has information on MySQL transactions.
If your MySQL setup does not support transactions, then Django will always function in autocommit mode: statements will be executed and committed as soon as they’re called. If your MySQL setup does support transactions, Django will handle transactions as explained in this document.
Handling exceptions within PostgreSQL transactions¶
This section is relevant only if you’re implementing your own transaction management. This problem cannot occur in Django’s default mode and atomic() handles it automatically.
Inside a transaction, when a call to a PostgreSQL cursor raises an exception (typically IntegrityError), all subsequent SQL in the same transaction will fail with the error “current transaction is aborted, queries ignored until end of transaction block”. Whilst simple use of save() is unlikely to raise an exception in PostgreSQL, there are more advanced usage patterns which might, such as saving objects with unique fields, saving using the force_insert/force_update flag, or invoking custom SQL.
There are several ways to recover from this sort of error.
The first option is to roll back the entire transaction. For example:
a.save() # Succeeds, but may be undone by transaction rollback try: b.save() # Could throw exception except IntegrityError: transaction.rollback() c.save() # Succeeds, but a.save() may have been undone
Calling transaction.rollback() rolls back the entire transaction. Any uncommitted database operations will be lost. In this example, the changes made by a.save() would be lost, even though that operation raised no error itself.
You can use savepoints to control the extent of a rollback. Before performing a database operation that could fail, you can set or update the savepoint; that way, if the operation fails, you can roll back the single offending operation, rather than the entire transaction. For example:
a.save() # Succeeds, and never undone by savepoint rollback sid = transaction.savepoint() try: b.save() # Could throw exception transaction.savepoint_commit(sid) except IntegrityError: transaction.savepoint_rollback(sid) c.save() # Succeeds, and a.save() is never undone
In this example, a.save() will not be undone in the case where b.save() raises an exception.
Changes from Django 1.5 and earlier¶
The features described below were deprecated in Django 1.6 and will be removed in Django 1.8. They’re documented in order to ease the migration to the new transaction management APIs.
The following functions, defined in django.db.transaction, provided a way to control transactions on a per-function or per-code-block basis. They could be used as decorators or as context managers, and they accepted a using argument, exactly like atomic().
Enable Django’s default autocommit behavior.
Transactions will be committed as soon as you call model.save(), model.delete(), or any other function that writes to the database.
Use a single transaction for all the work done in a function.
If the function returns successfully, then Django will commit all work done within the function at that point. If the function raises an exception, though, Django will roll back the transaction.
Tells Django you’ll be managing the transaction on your own.
Whether you are writing or simply reading from the database, you must commit() or rollback() explicitly or Django will raise a TransactionManagementError exception. This is required when reading from the database because SELECT statements may call functions which modify tables, and thus it is impossible to know if any data has been modified.
The three functions described above relied on a concept called “transaction states”. This mechanism was deprecated in Django 1.6, but it’s still available until Django 1.8.
At any time, each database connection is in one of these two states:
- auto mode: autocommit is enabled;
- managed mode: autocommit is disabled.
Internally, Django keeps a stack of states. Activations and deactivations must be balanced.
For example, commit_on_success() switches to managed mode when entering the block of code it controls; when exiting the block, it commits or rollbacks, and switches back to auto mode.
So commit_on_success() really has two effects: it changes the transaction state and it defines a transaction block. Nesting will give the expected results in terms of transaction state, but not in terms of transaction semantics. Most often, the inner block will commit, breaking the atomicity of the outer block.
In Django 1.6, TransactionMiddleware is deprecated and replaced by ATOMIC_REQUESTS. While the general behavior is the same, there are two differences.
With the previous API, it was possible to switch to autocommit or to commit explicitly anywhere inside a view. Since ATOMIC_REQUESTS relies on atomic() which enforces atomicity, this isn’t allowed any longer. However, at the toplevel, it’s still possible to avoid wrapping an entire view in a transaction. To achieve this, decorate the view with non_atomic_requests() instead of autocommit().
The transaction middleware applied not only to view functions, but also to middleware modules that came after it. For instance, if you used the session middleware after the transaction middleware, session creation was part of the transaction. ATOMIC_REQUESTS only applies to the view itself.
Starting with Django 1.6, atomic() is the only supported API for defining a transaction. Unlike the deprecated APIs, it’s nestable and always guarantees atomicity.
In most cases, it will be a drop-in replacement for commit_on_success().
During the deprecation period, it’s possible to use atomic() within autocommit(), commit_on_success() or commit_manually(). However, the reverse is forbidden, because nesting the old decorators / context managers breaks atomicity.
Django 1.6 introduces an explicit API for mananging autocommit.
To disable autocommit temporarily, instead of:
with transaction.commit_manually(): # do stuff
you should now use:
transaction.set_autocommit(False) try: # do stuff finally: transaction.set_autocommit(True)
To enable autocommit temporarily, instead of:
with transaction.autocommit(): # do stuff
you should now use:
transaction.set_autocommit(True) try: # do stuff finally: transaction.set_autocommit(False)
Unless you’re implementing a transaction management framework, you shouldn’t ever need to do this.
Since version 1.6, Django uses database-level autocommit in auto mode. Previously, it implemented application-level autocommit by triggering a commit after each ORM write.
As a consequence, each database query (for instance, an ORM read) started a transaction that lasted until the next ORM write. Such “automatic transactions” no longer exist in Django 1.6.
There are four known scenarios where this is backwards-incompatible.
Note that managed mode isn’t affected at all. This section assumes auto mode. See the description of modes above.
Sequences of custom SQL queries¶
If you’re executing several custom SQL queries in a row, each one now runs in its own transaction, instead of sharing the same “automatic transaction”. If you need to enforce atomicity, you must wrap the sequence of queries in atomic().
To check for this problem, look for calls to cursor.execute(). They’re usually followed by a call to transaction.commit_unless_managed(), which isn’t useful any more and should be removed.
Select for update¶
If you were relying on “automatic transactions” to provide locking between select_for_update() and a subsequent write operation — an extremely fragile design, but nonetheless possible — you must wrap the relevant code in atomic(). Since Django 1.6.3, executing a query with select_for_update() in autocommit mode will raise a TransactionManagementError.
Using a high isolation level¶
If you were using the “repeatable read” isolation level or higher, and if you relied on “automatic transactions” to guarantee consistency between successive reads, the new behavior might be backwards-incompatible. To enforce consistency, you must wrap such sequences in atomic().
MySQL defaults to “repeatable read” and SQLite to “serializable”; they may be affected by this problem.
At the “read committed” isolation level or lower, “automatic transactions” have no effect on the semantics of any sequence of ORM operations.
PostgreSQL and Oracle default to “read committed” and aren’t affected, unless you changed the isolation level.
Using unsupported database features¶
With triggers, views, or functions, it’s possible to make ORM reads result in database modifications. Django 1.5 and earlier doesn’t deal with this case and it’s theoretically possible to observe a different behavior after upgrading to Django 1.6 or later. In doubt, use atomic() to enforce integrity.