Fix issues upgrading SCMTool IDs on non-pristine setups.

Review Request #12380 — Created June 17, 2022 and submitted

Information

Review Board
release-5.0.x

Reviewers

While trying to upgrade my system to the new registry-backed SCMTool
support, a number of issues with the new upgrade process for SCMTools
0were discovered, in part due to the complexity and history of my local
database.

These issues include:

  1. Pre-upgrade only recorded state to upgrade if the
    repository_scmtool_id was not yet in the database, preventing a
    re-attempt at upgrading from working on failure (and outright
    skipping the pre-upgrade work if running evolve --execute before
    running a managed upgrade).

  2. SCMTools that could not be loaded (missing package) would cause the
    pre-upgrade steps to crash and no IDs to be migrated.

  3. Integration configurations that lacked a saved conditions field
    would cause a lookup on conditions to try to load the integration
    class in order to retrieve the default, and as these are provided by
    unloaded extensions, this would cause the pre-upgrade steps to
    crash.

This change fixes all three scenarios.

Pre-upgrade steps for SCMTools are now run if either the evolution is
missing from the database or if we can find any repositories lacking
an ID. This enables re-attempts in both the premature evolve --execute
case and in the upgrade re-attempt case.

We now check for exceptions on getting the SCMTool IDs. If we fail to
load any, we'll list them and follow up with some helpful instructions.
These repositories will remain in their pre-upgrade state until resolved
(an upcoming change will improve upon this bad state).

For updates to integration configuration, the code is now careful to
work directly with the settings dictionary rather than the wrapper
methods, to avoid an attempted load of the integration. As an
optimization, the queryset also avoids loading any more state than
needed for the upgrade.

Tested an upgrade on a clean database.

Tested an upgrade on my messy database, with missing SCMTools and
integration configurations both without explicit conditions saved
and without a ConditionsField.

Verified that the IDs got migrated over even when errors were
encountered.

After resolving the errors, I verified that the upgrade could be
re-attempted and the remaining SCMTool IDs were looked up and
migrated over.

Summary ID
Fix issues upgrading SCMTool IDs on non-pristine setups.
While trying to upgrade my system to the new registry-backed SCMTool support, a number of issues with the new upgrade process for SCMTools 0were discovered, in part due to the complexity and history of my local database. These issues include: 1. Pre-upgrade only recorded state to upgrade if the `repository_scmtool_id` was not yet in the database, preventing a re-attempt at upgrading from working on failure (and outright skipping the pre-upgrade work if running `evolve --execute` before running a managed upgrade). 2. SCMTools that could not be loaded (missing package) would cause the pre-upgrade steps to crash and no IDs to be migrated. 3. Integration configurations that lacked a saved `conditions` field would cause a lookup on `conditions` to try to load the integration class in order to retrieve the default, and as these are provided by unloaded extensions, this would cause the pre-upgrade steps to crash. This change fixes all three scenarios. Pre-upgrade steps for SCMTools are now run if either the evolution is missing from the database *or* if we can find any repositories lacking an ID. This enables re-attempts in both the premature `evolve --execute` case and in the upgrade re-attempt case. We now check for exceptions on getting the SCMTool IDs. If we fail to load any, we'll list them and follow up with some helpful instructions. These repositories will remain in their pre-upgrade state until resolved (an upcoming change will improve upon this bad state). For updates to integration configuration, the code is now careful to work directly with the `settings` dictionary rather than the wrapper methods, to avoid an attempted load of the integration. As an optimization, the queryset also avoids loading any more state than needed for the upgrade.
1c87f7d931eebc34345eb9f3eefc5095ea053008
Description From Last Updated

This blank line isn't necessary.

daviddavid

I feel like we should print what tools are missing.

daviddavid

undefined name 'e' Column: 28 Error code: F821

reviewbotreviewbot
david
  1. 
      
  2. reviewboard/upgrade.py (Diff revision 1)
     
     
    Show all issues

    This blank line isn't necessary.

  3. reviewboard/upgrade.py (Diff revision 1)
     
     

    Does this actually work when the scmtool_id field doesn't yet exist?

    1. No, but if has_evolution is False, we won't hit that code.

      I could wrap the whole thing in an exception as an extra measure of protection.

  4. reviewboard/upgrade.py (Diff revision 1)
     
     
     
     
     
     
    Show all issues

    I feel like we should print what tools are missing.

  5. 
      
chipx86
Review request changed
Change Summary:
  • Removed a blank line.
  • Added extra protection in case a database query fails when determining whether to upgrade tools.
  • Changed the error output for tools that fail to load. This is built to be expanded in an upcoming change.
Commits:
Summary ID
Fix issues upgrading SCMTool IDs on non-pristine setups.
While trying to upgrade my system to the new registry-backed SCMTool support, a number of issues with the new upgrade process for SCMTools 0were discovered, in part due to the complexity and history of my local database. These issues include: 1. Pre-upgrade only recorded state to upgrade if the `repository_scmtool_id` was not yet in the database, preventing a re-attempt at upgrading from working on failure (and outright skipping the pre-upgrade work if running `evolve --execute` before running a managed upgrade). 2. SCMTools that could not be loaded (missing package) would cause the pre-upgrade steps to crash and no IDs to be migrated. 3. Integration configurations that lacked a saved `conditions` field would cause a lookup on `conditions` to try to load the integration class in order to retrieve the default, and as these are provided by unloaded extensions, this would cause the pre-upgrade steps to crash. This change fixes all three scenarios. Pre-upgrade steps for SCMTools are now run if either the evolution is missing from the database *or* if we can find any repositories lacking an ID. This enables re-attempts in both the premature `evolve --execute` case and in the upgrade re-attempt case. We now check for exceptions on getting the SCMTool IDs. If we fail to load any, we'll list them and follow up with some helpful instructions. These repositories will remain in their pre-upgrade state until resolved (an upcoming change will improve upon this bad state). For updates to integration configuration, the code is now careful to work directly with the `settings` dictionary rather than the wrapper methods, to avoid an attempted load of the integration. As an optimization, the queryset also avoids loading any more state than needed for the upgrade.
b546e75de0b11264a5ca4fec539097d255aa89b7
Fix issues upgrading SCMTool IDs on non-pristine setups.
While trying to upgrade my system to the new registry-backed SCMTool support, a number of issues with the new upgrade process for SCMTools 0were discovered, in part due to the complexity and history of my local database. These issues include: 1. Pre-upgrade only recorded state to upgrade if the `repository_scmtool_id` was not yet in the database, preventing a re-attempt at upgrading from working on failure (and outright skipping the pre-upgrade work if running `evolve --execute` before running a managed upgrade). 2. SCMTools that could not be loaded (missing package) would cause the pre-upgrade steps to crash and no IDs to be migrated. 3. Integration configurations that lacked a saved `conditions` field would cause a lookup on `conditions` to try to load the integration class in order to retrieve the default, and as these are provided by unloaded extensions, this would cause the pre-upgrade steps to crash. This change fixes all three scenarios. Pre-upgrade steps for SCMTools are now run if either the evolution is missing from the database *or* if we can find any repositories lacking an ID. This enables re-attempts in both the premature `evolve --execute` case and in the upgrade re-attempt case. We now check for exceptions on getting the SCMTool IDs. If we fail to load any, we'll list them and follow up with some helpful instructions. These repositories will remain in their pre-upgrade state until resolved (an upcoming change will improve upon this bad state). For updates to integration configuration, the code is now careful to work directly with the `settings` dictionary rather than the wrapper methods, to avoid an attempted load of the integration. As an optimization, the queryset also avoids loading any more state than needed for the upgrade.
2a7290c6b4553c6adfbc38a31a94c95717cb91a8

Checks run (1 failed, 1 succeeded)

flake8 failed.
JSHint passed.

flake8

chipx86
david
  1. Ship It!
  2. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-5.0.x (b5337d4)