• 
      

    Add support of gitlab-ci

    Review Request #14211 — Created Oct. 24, 2024 and submitted

    Information

    rbintegrations
    release-4.x

    Reviewers

    Add support of gitlab-ci

    Added integration and saw that it calls the pipeline on gitlab.

    Summary ID
    Add support of gitlab-ci
    f911e29b30630c4abe1a2585fbff3fae550a560e
    Description From Last Updated

    Also it would be a good idea to provide a gitlab component for patching like this. templates/ref.yml spec: --- .reviewboard: …

    misery misery

    're' imported but unused Column: 1 Error code: F401

    reviewbot reviewbot

    're' imported but unused Column: 1 Error code: F401

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line too long (97 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (92 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (86 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (87 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    This will need a Version Added: with the target version. We can take care of this when we land it, …

    chipx86 chipx86

    We'll need a from __future__ import annotations import here.

    chipx86 chipx86

    gitlab is a third-pary library, so it'll go in the same import group as django (with import ... listed above …

    chipx86 chipx86

    I think we'll want a clean_gitlab_vars method that performs a JSON validation.

    chipx86 chipx86

    Formatting for both these help texts should be the same as the one in gitlab_old_pipeline.

    chipx86 chipx86

    This needs a -> dict[str, Any] type hint.

    chipx86 chipx86

    This can just be super().clean(), since we're on Python 3.

    chipx86 chipx86

    Let's also catch a plain Exception with logging, so we can handle any unforeseen issues.

    chipx86 chipx86

    This can live in clean_gitlab_vars.

    chipx86 chipx86

    This belongs in the third-party import group with djblets.

    chipx86 chipx86

    This needs a Version Added.

    chipx86 chipx86

    This can be dict[str, str], which means we can also get rid of the Dict import.

    chipx86 chipx86

    Since this is a new module, the methods don't need Version Added. The class's will cover it.

    chipx86 chipx86

    Since BuildPrepData is located in another module, we'll need to specify the full module path to it in the docs.

    chipx86 chipx86

    Same note as before.

    chipx86 chipx86

    For dictionaries, we use this format: variables = [ { 'key': 'REVIEWBOARD_SERVER', 'value': patch_info['reviewboard_server'], }, ... ] Note also the …

    chipx86 chipx86

    We use a blank line between statements and blocks.

    chipx86 chipx86

    Here as well.

    chipx86 chipx86

    This should use the same format shown above.

    chipx86 chipx86

    Looks like left-over debug code. Same below.

    chipx86 chipx86

    Same note about dictionary formatting.

    chipx86 chipx86

    This module needs a docstring with a Version Added.

    chipx86 chipx86

    We need a Version Added and an annotations import.

    chipx86 chipx86

    This needs a Version Added.

    chipx86 chipx86

    These should use single quotes and should sort keys alphabetically, to ease lookup and maintenance.

    chipx86 chipx86

    This should include typing for request and the return type HttpResponseBase.

    chipx86 chipx86

    Blank line between statements and blocks. Same below. Along with that, single quotes instead of double quotes, here and below.

    chipx86 chipx86

    We prefer this format for conditionals: if (payload is None or payload.get('object_kind') != 'pipeline' or 'object_attributes' not in payload): ... …

    chipx86 chipx86

    For dictionary comprehensions, we use this format: v = { item['key'] : item['value'] for item in payload['variables'] }

    chipx86 chipx86

    Blank line between statements and conditionals here as well.

    chipx86 chipx86

    We'll want typing for any new methods, for both arguments and return types.

    chipx86 chipx86

    This is missing a trailing period.

    chipx86 chipx86

    The trailing ) should be on the same parameter line.

    chipx86 chipx86

    We should localize all errors. Here and below.

    chipx86 chipx86

    Same note as above with logging. We also prefer for logging variables to be on a separate line from the …

    chipx86 chipx86

    When saving, we should take to pass in update_fields=(...) with every field we explicitly want to save. This avoids side-effects …

    chipx86 chipx86

    We'll need to cap this dependency so there aren't any surprises down the road.

    chipx86 chipx86

    line too long (83 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (82 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (83 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (81 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (81 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (81 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    das wird wohl in der nächsten python-gitlab funktionieren... https://github.com/caillouc/python-gitlab/commit/a4c0444124774b548504c63d39f8bc9a14121933

    misery misery

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 13 Error code: W503

    reviewbot reviewbot

    line break before binary operator Column: 17 Error code: W503

    reviewbot reviewbot

    line too long (90 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line too long (82 > 79 characters) Column: 80 Error code: E501

    reviewbot reviewbot

    line break before binary operator Column: 17 Error code: W503

    reviewbot reviewbot
    misery
    Review request changed
    Summary:
    WIP Add support of gitlab-ci
    Add support of gitlab-ci
    Description:
    ~  

    WIP Add support of gitlab-ci

      ~

    Add support of gitlab-ci

    Testing Done:
      +

    Added integration and saw that it calls the pipeline on gitlab.

    Commits:
    Summary ID
    WIP Add support of gitlab-ci
    0cd1f7e13ffea73695adcbf8fd0ed540fd28de27
    Add support of gitlab-ci
    1bc87e573cddb172e5f295ce2aa7fc8be7bae10c

    Checks run (1 failed, 1 succeeded)

    flake8 failed.
    JSHint passed.

    flake8

    misery
    Review request changed
    Commits:
    Summary ID
    Add support of gitlab-ci
    1bc87e573cddb172e5f295ce2aa7fc8be7bae10c
    Add support of gitlab-ci
    1bc87e573cddb172e5f295ce2aa7fc8be7bae10c

    Checks run (1 failed, 1 succeeded)

    flake8 failed.
    JSHint passed.

    flake8

    misery
    misery
    1. 
        
    2. The best way to notify RB from gitlab would be a webook from gitlab to RB. I wrote a tiny wsgi application that get's "pipeline event" [1] for the project/group that calls rbt. A better approach would be to include the ability to RB itself without that wrapper script.

      #!/usr/bin/python
      
      import json
      import logging
      import uwsgi
      import subprocess
      
      log = logging.getLogger("gitlab")
      
      logging.basicConfig(
          format="%(asctime)s %(levelname)s: %(message)s", level="INFO"
      )
      
      
      def health():
          return None
      
      
      def get_state(state):
          if state == "pending":
              return ("build queued...", "pending")
      
          if state == "passed":
              return ("build succeeded.", "done-success")
      
          if state == "running":
              return ("build started...", "pending")
      
          if state == "canceling":
              return ("canceling build...", "pending")
      
          if state == "canceled":
              return ("build canceled", "error")
      
          return ("build failed", "error")
      
      
      def process_rbtools(v, c, token):
          url = c.get("url")
          url_text = "See pipeline"
      
          description, state = get_state(c.get("detailed_status"))
          cmd = [
              "rbt",
              "status-update",
              "set",
              "--url",
              url,
              "--url-text",
              url_text,
              "-r",
              v.get("REVIEWBOARD_REVIEW_ID"),
              "-s",
              v.get("REVIEWBOARD_STATUS_UPDATE_ID"),
              "--server",
              v.get("REVIEWBOARD_SERVER"),
              "--api-token",
              token,
              "--state",
              state,
              "--description",
              description,
              "--disable-cookie-storage",
              "--disable-cache-storage",
          ]
      
          log.info(cmd)
          subprocess.check_call(cmd)
          return uwsgi.SPOOL_OK
      
      
      def process_pipeline(c, token):
          if c is not None:
              v = c.get("variables")
              if v is not None:
                  v = {item["key"]: item["value"] for item in v}
                  if "REVIEWBOARD_SERVER" in v:
                      return process_rbtools(v, c, token)
          return uwsgi.SPOOL_OK
      
      
      def process_data(data):
          try:
              c = json.loads(data.get("body"))
          except (TypeError, ValueError) as e:
              log.info("Broken json: {0}".format(e))
              return uwsgi.SPOOL_OK
      
          # log.debug(json.dumps(c, indent=4, sort_keys=True))
      
          if c.get("object_kind") == "pipeline" and "object_attributes" in c:
              return process_pipeline(c.get("object_attributes"), data.get(b"token"))
      
          return uwsgi.SPOOL_OK
      
      
      def process_spooler(data):
          try:
              return process_data(data)
          except BaseException:
              log.exception("Cannot process data")
      
          return uwsgi.SPOOL_RETRY
      
      
      def application(environ, start_response):
          result = []
          headers = [("Content-Type", "text/plain")]
      
          request_body = []
          try:
              request_body_size = environ.get("CONTENT_LENGTH", 0)
              if request_body_size:
                  request_body = environ["wsgi.input"].read(int(request_body_size))
          except ValueError as e:
              log.warn("Cannot read body: {0}".format(e))
              headers.append(("Content-Length", str(len(result))))
              start_response("400 Bad Request", headers)
              return result
      
          status = "200 OK"
          if len(request_body) > 0 and "HTTP_X_GITLAB_TOKEN" in environ:
              uwsgi.spool(
                  {
                      b"body": request_body,
                      b"token": environ["HTTP_X_GITLAB_TOKEN"].encode("utf-8"),
                  }
              )
          else:
              healthResult = health()
              if healthResult is None:
                  status = "204 No Content"
              else:
                  status = "503 Service Unavailable"
                  result = [healthResult.encode("utf-8")]
      
          headers.append(("Content-Length", str(len(result))))
          start_response(status, headers)
          return result
      
      
      uwsgi.spooler = process_spooler
      

      [1] https://docs.gitlab.com/user/project/integrations/webhook_events/#pipeline-events

    3. Show all issues

      Also it would be a good idea to provide a gitlab component for patching like this.

      templates/ref.yml

      spec:
      ---
      
      .reviewboard:
          reviewboard_before_script:
             - |
                   if [ "$REVIEWBOARD_SERVER" != "" ]; then
                      apk add rbtools git
                      rbt patch --server ${REVIEWBOARD_SERVER} --diff-revision ${REVIEWBOARD_DIFF_REVISION} ${REVIEWBOARD_REVIEW_ID}
                   fi
      

      templates/global_before_script.yml

      spec:
      ---
      
      include:
        - component: https://github.com/reviewboard/rbintegrations/ref@main
      
      before_script:
          - !reference [.reviewboard, reviewboard_before_script]
      

      customer .gitlab-ci.yml

      include:
        - component: https://github.com/reviewboard/rbintegrations/ref@main
      
      first-job:
        script:
          - echo Hello
      
    4. 
        
    misery
    misery
    misery
    misery
    Review request changed
    Commits:
    Summary ID
    Add support of gitlab-ci
    07a041231afc9a85712e81b73539e7c56347cc9f
    Add support of gitlab-ci
    fac64c4771f050aa05b5535b46f27e2526ab0c62

    Checks run (1 failed, 1 succeeded)

    flake8 failed.
    JSHint passed.

    flake8

    misery
    misery
    misery
    Review request changed
    Commits:
    Summary ID
    Add support of gitlab-ci
    55e66ce2f3d443172107b2b0b26d1016078939e6
    Add support of gitlab-ci
    69d3ba54b4e7d0f63e5e140af85a0d08a7ca7c8a

    Checks run (1 failed, 1 succeeded)

    flake8 failed.
    JSHint passed.

    flake8

    misery
    misery
    misery
    chipx86
    1. Thanks for the change, and for your patience while we worked on a launch!

      The logic all looks great. There's style guideline stuff, which is the bulk of the review. We're happy to finish off the work if you'd like to pass it to us, but of course we're also happy to work with you through iterations of the code :)

      Definitely want to get this in for the next release.

      1. Feel free to adopt this. I will add "input variables" of gitlab 17.11 in the next weeks if our internal instance is updated. Do you want to merge this soon? So I can add another request for the input-feature.

        https://about.gitlab.com/releases/2025/04/17/gitlab-17-11-released/#cicd-pipeline-inputs

    2. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
      Show all issues

      This will need a Version Added: with the target version.

      We can take care of this when we land it, but I'm noting this so it doesn't get missed.

      Same wiht the __init__.py and other modules.

    3. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
       
       
      Show all issues

      We'll need a from __future__ import annotations import here.

    4. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
      Show all issues

      gitlab is a third-pary library, so it'll go in the same import group as django (with import ... listed above from ...).

    5. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
       
       
      Show all issues

      I think we'll want a clean_gitlab_vars method that performs a JSON validation.

    6. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
       
       
       
       
       
       
       
       
       
       
      Show all issues

      Formatting for both these help texts should be the same as the one in gitlab_old_pipeline.

    7. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
      Show all issues

      This needs a -> dict[str, Any] type hint.

    8. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
      Show all issues

      This can just be super().clean(), since we're on Python 3.

    9. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
       
       
       
       
       
       
       
      Show all issues

      Let's also catch a plain Exception with logging, so we can handle any unforeseen issues.

    10. rbintegrations/gitlabci/forms.py (Diff revision 14)
       
       
       
       
       
       
       
      Show all issues

      This can live in clean_gitlab_vars.

    11. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
      Show all issues

      This belongs in the third-party import group with djblets.

    12. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
      Show all issues

      This needs a Version Added.

    13. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
      Show all issues

      This can be dict[str, str], which means we can also get rid of the Dict import.

    14. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
       
      Show all issues

      Since this is a new module, the methods don't need Version Added. The class's will cover it.

    15. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
      Show all issues

      Since BuildPrepData is located in another module, we'll need to specify the full module path to it in the docs.

    16. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
      Show all issues

      Same note as before.

    17. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
      Show all issues

      For dictionaries, we use this format:

      variables = [
          {
              'key': 'REVIEWBOARD_SERVER',
              'value': patch_info['reviewboard_server'],
          },
          ...
      ]
      

      Note also the single quotes. We use double quotes only when containing text needing a single quote.

    18. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
       
      Show all issues

      We use a blank line between statements and blocks.

    19. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
       
      Show all issues

      Here as well.

    20. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
      Show all issues

      This should use the same format shown above.

    21. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
      Show all issues

      Looks like left-over debug code. Same below.

    22. rbintegrations/gitlabci/integration.py (Diff revision 14)
       
       
       
      Show all issues

      Same note about dictionary formatting.

    23. rbintegrations/gitlabci/urls.py (Diff revision 14)
       
       
      Show all issues

      This module needs a docstring with a Version Added.

    24. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
       
      Show all issues

      We need a Version Added and an annotations import.

    25. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
      Show all issues

      This needs a Version Added.

    26. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
       
      Show all issues

      These should use single quotes and should sort keys alphabetically, to ease lookup and maintenance.

    27. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
      Show all issues

      This should include typing for request and the return type HttpResponseBase.

    28. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
      Show all issues

      Blank line between statements and blocks. Same below.

      Along with that, single quotes instead of double quotes, here and below.

    29. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
       
       
       
      Show all issues

      We prefer this format for conditionals:

      if (payload is None or
          payload.get('object_kind') != 'pipeline' or
          'object_attributes' not in payload):
          ...
      

      Same below.

    30. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
      Show all issues

      For dictionary comprehensions, we use this format:

      v = {
          item['key'] : item['value']
          for item in payload['variables']
      }
      
    31. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
      Show all issues

      Blank line between statements and conditionals here as well.

    32. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
      Show all issues

      We'll want typing for any new methods, for both arguments and return types.

    33. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
      Show all issues

      This is missing a trailing period.

    34. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
       
       
       
       
      Show all issues

      The trailing ) should be on the same parameter line.

    35. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
      Show all issues

      We should localize all errors. Here and below.

    36. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
       
       
      Show all issues

      Same note as above with logging. We also prefer for logging variables to be on a separate line from the message, to make it easier to manage the text.

    37. rbintegrations/gitlabci/views.py (Diff revision 14)
       
       
      Show all issues

      When saving, we should take to pass in update_fields=(...) with every field we explicitly want to save. This avoids side-effects with other state that may be in the database (e.g., extra_data) and avoids re-saving fields that haven't changed.

    38. setup.py (Diff revision 14)
       
       
      Show all issues

      We'll need to cap this dependency so there aren't any surprises down the road.

    39. 
        
    misery
    misery
    Review request changed
    Commits:
    Summary ID
    Add support of gitlab-ci
    4f20e6d7d519951c160f2a1a866f48bf9ad2dce8
    Add support of gitlab-ci
    f911e29b30630c4abe1a2585fbff3fae550a560e

    Checks run (1 failed, 1 succeeded)

    flake8 failed.
    JSHint passed.

    flake8

    misery
    1. 
        
    2. rbintegrations/gitlabci/integration.py (Diff revision 16)
       
       
       
       
       
      Show all issues

      das wird wohl in der nächsten python-gitlab funktionieren...

      https://github.com/caillouc/python-gitlab/commit/a4c0444124774b548504c63d39f8bc9a14121933

    3. 
        
    david
    Review request changed
    Status:
    Completed