• 
      

    Fix message threading when using Amazon SES as a mail provider.

    Review Request #11727 — Created July 16, 2021 and submitted

    Information

    Review Board
    release-4.0.x

    Reviewers

    Amazon SES has some non-standard behavior around Message IDs. Normally,
    the sender (or an intermediary) will specify a Message ID for tracking
    and In-Reply-To headers, which can be used for bounce management or
    automated threading. We depend on this for our threading model, storing
    our generated Message IDs and attaching them to any replies.

    SES overrides any provided Message ID, which impacts our threading. They
    do provide a mechanism to get part of the overridden Message ID, by
    using a non-standard addition to the SMTP spec, which is to grab the
    final 250 Ok after sending the e-mail and parsing out what's currently
    a UUID after the Ok.

    Python's smtplib does not support getting this information. Nor does
    Django's e-mail code. In order to access it, we provide our own SMTP
    backend, which is a subclass of Python's, and override the method used
    to get the result of sending an e-mail and store that for later
    retrieval.

    We then subclass Django's backend and pull out this result. If the SMTP
    hostname is an SES endpoint, we parse out the region and activate
    post-sending logic, pulling in that reply, checking if it appears to
    have the first part of a Message ID (the UUID), and then build on the
    domain (which will be email.amazonses.com for us-east-1, or
    <region>.amazonses.com for anything else) and store it back in the
    message, where it can be retrieved.

    This works for now, though as Amazon's behavior is non-standard and they
    don't offer official guidance, it's always possible this logic will need
    to be adjusted down the road.

    For anything other than SES, the new backend should not introduce any
    new behavior.

    Unit tests pass on Python 2 and 3.

    Tested with a customer who was running into threading issues with SES.
    We confirmed that this patch fixes it.

    Summary ID
    Fix message threading when using Amazon SES as a mail provider.
    Amazon SES has some non-standard behavior around Message IDs. Normally, the sender (or an intermediary) will specify a Message ID for tracking and `In-Reply-To` headers, which can be used for bounce management or automated threading. We depend on this for our threading model, storing our generated Message IDs and attaching them to any replies. SES overrides any provided Message ID, which impacts our threading. They do provide a mechanism to get part of the overridden Message ID, by using a non-standard addition to the SMTP spec, which is to grab the final `250 Ok` after sending the e-mail and parsing out what's currently a UUID after the `Ok`. Python's `smtplib` does not support getting this information. Nor does Django's e-mail code. In order to access it, we provide our own SMTP backend, which is a subclass of Python's, and override the method used to get the result of sending an e-mail and store that for later retrieval. We then subclass Django's backend and pull out this result. If the SMTP hostname is an SES endpoint, we parse out the region and activate post-sending logic, pulling in that reply, checking if it appears to have the first part of a Message ID (the UUID), and then build on the domain (which will be `email.amazonses.com` for us-east-1, or `<region>.amazonses.com` for anything else) and store it back in the message, where it can be retrieved. This works for now, though as Amazon's behavior is non-standard and they don't offer official guidance, it's always possible this logic will need to be adjusted down the road. For anything other than SES, the new backend should not introduce any new behavior.
    50c0153f80d901f4ba4d42e2fbf0e5c7e7f4b977
    Description From Last Updated

    Can you add a module docstring?

    daviddavid

    F401 'reviewboard.notifications.email.backend.SMTPConnection' imported but unused

    reviewbotreviewbot
    Checks run (1 failed, 1 succeeded)
    flake8 failed.
    JSHint passed.

    flake8

    david
    1. 
        
    2. Show all issues

      Can you add a module docstring?

    3. 
        
    chipx86
    Review request changed
    Status:
    Completed
    Change Summary:
    Pushed to release-4.0.x (5bc8e19)