Add support for just-in-time SQL code generation for databases.

Review Request #11089 — Created July 20, 2020 and submitted

Information

Django Evolution
master

Reviewers

The way Django Evolution worked before was that it'd compute a set of
operations that needed to be performed to get the database into the
state needed for the evolutions, and it'd ask the database evolution
backends to generate suitable SQL for later execution.

This generally works, but there are cases in which the SQL needs to be a
bit more dynamic than that, making use of state that doesn't exist until
it comes time to execute that SQL statement. This will specifically be
required for SQLite primary key modifications in modern versions of
SQLite and Django.

Now, rather than just returning hard-coded SQL statements, an
SQLResult's list of statements can now contain a callback function
that takes a cursor and returns new statements. This will be called when
it'd otherwise be time to execute the SQL statement for that entry in
the list.

To make this happen, some part of the execution/logging code had to
change. Previously, we'd normalize the list of SQL statements for
output, and then normalize a different way for execution (the main
differences between that we'd apply parameters to format strings for
logging, but pass them separately to the database backend for
execution). Since we need the callbacks to execute only once, these
operations had to merge.

To do this, there's now a private function dedicated to taking a list of
SQL statements/callbacks and yielding results one-by-one. There's then a
function, run_sql(), which provides the main logic for both execution
and capturing for logging, allowing both to happen in one pass. The
original execute_sql() and write_sql() methods wrap this, behaving
as before but, in the case of execute_sql(), allowing the caller to
capture as part of the standard execution process.

Unit tests pass for all supported versions of Django and all databases.

Summary ID
Add support for just-in-time SQL code generation for databases.
The way Django Evolution worked before was that it'd compute a set of operations that needed to be performed to get the database into the state needed for the evolutions, and it'd ask the database evolution backends to generate suitable SQL for later execution. This generally works, but there are cases in which the SQL needs to be a bit more dynamic than that, making use of state that doesn't exist until it comes time to execute that SQL statement. This will specifically be required for SQLite primary key modifications in modern versions of SQLite and Django. Now, rather than just returning hard-coded SQL statements, an `SQLResult`'s list of statements can now contain a callback function that takes a cursor and returns new statements. This will be called when it'd otherwise be time to execute the SQL statement for that entry in the list. To make this happen, some part of the execution/logging code had to change. Previously, we'd normalize the list of SQL statements for output, and then normalize a different way for execution (the main differences between that we'd apply parameters to format strings for logging, but pass them separately to the database backend for execution). Since we need the callbacks to execute only once, these operations had to merge. To do this, there's now a private function dedicated to taking a list of SQL statements/callbacks and yielding results one-by-one. There's then a function, `run_sql()`, which provides the main logic for both execution and capturing for logging, allowing both to happen in one pass. The original `execute_sql()` and `write_sql()` methods wrap this, behaving as before but, in the case of `execute_sql()`, allowing the caller to capture as part of the standard execution process.
c33e6b7af0baac85df43ca69a60b23ebf54ef864
david
  1. Ship It!
  2. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to master (2998abc)