Add support of gitlab-ci

Review Request #14211 — Created Oct. 24, 2024 and updated

Information

rbintegrations
release-4.x

Reviewers

Add support of gitlab-ci

Added integration and saw that it calls the pipeline on gitlab.

Summary ID
Add support of gitlab-ci
f911e29b30630c4abe1a2585fbff3fae550a560e
Description From Last Updated

Also it would be a good idea to provide a gitlab component for patching like this. templates/ref.yml spec: --- .reviewboard: …

miserymisery

're' imported but unused Column: 1 Error code: F401

reviewbotreviewbot

're' imported but unused Column: 1 Error code: F401

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line too long (97 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (92 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (86 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (87 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

This will need a Version Added: with the target version. We can take care of this when we land it, …

chipx86chipx86

We'll need a from __future__ import annotations import here.

chipx86chipx86

gitlab is a third-pary library, so it'll go in the same import group as django (with import ... listed above …

chipx86chipx86

I think we'll want a clean_gitlab_vars method that performs a JSON validation.

chipx86chipx86

Formatting for both these help texts should be the same as the one in gitlab_old_pipeline.

chipx86chipx86

This needs a -> dict[str, Any] type hint.

chipx86chipx86

This can just be super().clean(), since we're on Python 3.

chipx86chipx86

Let's also catch a plain Exception with logging, so we can handle any unforeseen issues.

chipx86chipx86

This can live in clean_gitlab_vars.

chipx86chipx86

This belongs in the third-party import group with djblets.

chipx86chipx86

This needs a Version Added.

chipx86chipx86

This can be dict[str, str], which means we can also get rid of the Dict import.

chipx86chipx86

Since this is a new module, the methods don't need Version Added. The class's will cover it.

chipx86chipx86

Since BuildPrepData is located in another module, we'll need to specify the full module path to it in the docs.

chipx86chipx86

Same note as before.

chipx86chipx86

For dictionaries, we use this format: variables = [ { 'key': 'REVIEWBOARD_SERVER', 'value': patch_info['reviewboard_server'], }, ... ] Note also the …

chipx86chipx86

We use a blank line between statements and blocks.

chipx86chipx86

Here as well.

chipx86chipx86

This should use the same format shown above.

chipx86chipx86

Looks like left-over debug code. Same below.

chipx86chipx86

Same note about dictionary formatting.

chipx86chipx86

This module needs a docstring with a Version Added.

chipx86chipx86

We need a Version Added and an annotations import.

chipx86chipx86

This needs a Version Added.

chipx86chipx86

These should use single quotes and should sort keys alphabetically, to ease lookup and maintenance.

chipx86chipx86

This should include typing for request and the return type HttpResponseBase.

chipx86chipx86

Blank line between statements and blocks. Same below. Along with that, single quotes instead of double quotes, here and below.

chipx86chipx86

We prefer this format for conditionals: if (payload is None or payload.get('object_kind') != 'pipeline' or 'object_attributes' not in payload): ... …

chipx86chipx86

For dictionary comprehensions, we use this format: v = { item['key'] : item['value'] for item in payload['variables'] }

chipx86chipx86

Blank line between statements and conditionals here as well.

chipx86chipx86

We'll want typing for any new methods, for both arguments and return types.

chipx86chipx86

This is missing a trailing period.

chipx86chipx86

The trailing ) should be on the same parameter line.

chipx86chipx86

We should localize all errors. Here and below.

chipx86chipx86

Same note as above with logging. We also prefer for logging variables to be on a separate line from the …

chipx86chipx86

When saving, we should take to pass in update_fields=(...) with every field we explicitly want to save. This avoids side-effects …

chipx86chipx86

We'll need to cap this dependency so there aren't any surprises down the road.

chipx86chipx86

line too long (83 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (82 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (83 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (81 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (81 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (81 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

das wird wohl in der nächsten python-gitlab funktionieren... https://github.com/caillouc/python-gitlab/commit/a4c0444124774b548504c63d39f8bc9a14121933

miserymisery

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 13 Error code: W503

reviewbotreviewbot

line break before binary operator Column: 17 Error code: W503

reviewbotreviewbot

line too long (90 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line too long (82 > 79 characters) Column: 80 Error code: E501

reviewbotreviewbot

line break before binary operator Column: 17 Error code: W503

reviewbotreviewbot
misery
Review request changed
Summary:
WIP Add support of gitlab-ci
Add support of gitlab-ci
Description:
~  

WIP Add support of gitlab-ci

  ~

Add support of gitlab-ci

Testing Done:
  +

Added integration and saw that it calls the pipeline on gitlab.

Commits:
Summary ID
WIP Add support of gitlab-ci
0cd1f7e13ffea73695adcbf8fd0ed540fd28de27
Add support of gitlab-ci
1bc87e573cddb172e5f295ce2aa7fc8be7bae10c
Diff:

Revision 2 (+972 -575)

Show changes

README.rst
package-lock.json
setup.py
docs/releasenotes/4.0.1.rst
docs/releasenotes/conf.py
docs/releasenotes/index.rst
rbintegrations/__init__.py
rbintegrations/extension.py
28 more

Checks run (1 failed, 1 succeeded)

flake8 failed.
JSHint passed.

flake8

misery
Review request changed
Commits:
Summary ID
Add support of gitlab-ci
1bc87e573cddb172e5f295ce2aa7fc8be7bae10c
Add support of gitlab-ci
1bc87e573cddb172e5f295ce2aa7fc8be7bae10c
Diff:

Revision 3 (+700)

Show changes

setup.py
rbintegrations/extension.py
rbintegrations/gitlabci/__init__.py
rbintegrations/gitlabci/forms.py
rbintegrations/gitlabci/integration.py
rbintegrations/static/images/gitlabci/icon.png
rbintegrations/static/images/gitlabci/icon@2x.png

Checks run (1 failed, 1 succeeded)

flake8 failed.
JSHint passed.

flake8

misery
misery
  1. 
      
  2. The best way to notify RB from gitlab would be a webook from gitlab to RB. I wrote a tiny wsgi application that get's "pipeline event" [1] for the project/group that calls rbt. A better approach would be to include the ability to RB itself without that wrapper script.

    #!/usr/bin/python
    
    import json
    import logging
    import uwsgi
    import subprocess
    
    log = logging.getLogger("gitlab")
    
    logging.basicConfig(
        format="%(asctime)s %(levelname)s: %(message)s", level="INFO"
    )
    
    
    def health():
        return None
    
    
    def get_state(state):
        if state == "pending":
            return ("build queued...", "pending")
    
        if state == "passed":
            return ("build succeeded.", "done-success")
    
        if state == "running":
            return ("build started...", "pending")
    
        if state == "canceling":
            return ("canceling build...", "pending")
    
        if state == "canceled":
            return ("build canceled", "error")
    
        return ("build failed", "error")
    
    
    def process_rbtools(v, c, token):
        url = c.get("url")
        url_text = "See pipeline"
    
        description, state = get_state(c.get("detailed_status"))
        cmd = [
            "rbt",
            "status-update",
            "set",
            "--url",
            url,
            "--url-text",
            url_text,
            "-r",
            v.get("REVIEWBOARD_REVIEW_ID"),
            "-s",
            v.get("REVIEWBOARD_STATUS_UPDATE_ID"),
            "--server",
            v.get("REVIEWBOARD_SERVER"),
            "--api-token",
            token,
            "--state",
            state,
            "--description",
            description,
            "--disable-cookie-storage",
            "--disable-cache-storage",
        ]
    
        log.info(cmd)
        subprocess.check_call(cmd)
        return uwsgi.SPOOL_OK
    
    
    def process_pipeline(c, token):
        if c is not None:
            v = c.get("variables")
            if v is not None:
                v = {item["key"]: item["value"] for item in v}
                if "REVIEWBOARD_SERVER" in v:
                    return process_rbtools(v, c, token)
        return uwsgi.SPOOL_OK
    
    
    def process_data(data):
        try:
            c = json.loads(data.get("body"))
        except (TypeError, ValueError) as e:
            log.info("Broken json: {0}".format(e))
            return uwsgi.SPOOL_OK
    
        # log.debug(json.dumps(c, indent=4, sort_keys=True))
    
        if c.get("object_kind") == "pipeline" and "object_attributes" in c:
            return process_pipeline(c.get("object_attributes"), data.get(b"token"))
    
        return uwsgi.SPOOL_OK
    
    
    def process_spooler(data):
        try:
            return process_data(data)
        except BaseException:
            log.exception("Cannot process data")
    
        return uwsgi.SPOOL_RETRY
    
    
    def application(environ, start_response):
        result = []
        headers = [("Content-Type", "text/plain")]
    
        request_body = []
        try:
            request_body_size = environ.get("CONTENT_LENGTH", 0)
            if request_body_size:
                request_body = environ["wsgi.input"].read(int(request_body_size))
        except ValueError as e:
            log.warn("Cannot read body: {0}".format(e))
            headers.append(("Content-Length", str(len(result))))
            start_response("400 Bad Request", headers)
            return result
    
        status = "200 OK"
        if len(request_body) > 0 and "HTTP_X_GITLAB_TOKEN" in environ:
            uwsgi.spool(
                {
                    b"body": request_body,
                    b"token": environ["HTTP_X_GITLAB_TOKEN"].encode("utf-8"),
                }
            )
        else:
            healthResult = health()
            if healthResult is None:
                status = "204 No Content"
            else:
                status = "503 Service Unavailable"
                result = [healthResult.encode("utf-8")]
    
        headers.append(("Content-Length", str(len(result))))
        start_response(status, headers)
        return result
    
    
    uwsgi.spooler = process_spooler
    

    [1] https://docs.gitlab.com/user/project/integrations/webhook_events/#pipeline-events

  3. Show all issues

    Also it would be a good idea to provide a gitlab component for patching like this.

    templates/ref.yml

    spec:
    ---
    
    .reviewboard:
        reviewboard_before_script:
           - |
                 if [ "$REVIEWBOARD_SERVER" != "" ]; then
                    apk add rbtools git
                    rbt patch --server ${REVIEWBOARD_SERVER} --diff-revision ${REVIEWBOARD_DIFF_REVISION} ${REVIEWBOARD_REVIEW_ID}
                 fi
    

    templates/global_before_script.yml

    spec:
    ---
    
    include:
      - component: https://github.com/reviewboard/rbintegrations/ref@main
    
    before_script:
        - !reference [.reviewboard, reviewboard_before_script]
    

    customer .gitlab-ci.yml

    include:
      - component: https://github.com/reviewboard/rbintegrations/ref@main
    
    first-job:
      script:
        - echo Hello
    
  4. 
      
misery
misery
misery
misery
Review request changed
Commits:
Summary ID
Add support of gitlab-ci
07a041231afc9a85712e81b73539e7c56347cc9f
Add support of gitlab-ci
fac64c4771f050aa05b5535b46f27e2526ab0c62
Diff:

Revision 8 (+1054)

Show changes

setup.py
rbintegrations/extension.py
rbintegrations/gitlabci/__init__.py
rbintegrations/gitlabci/forms.py
rbintegrations/gitlabci/integration.py
rbintegrations/gitlabci/urls.py
rbintegrations/gitlabci/views.py
rbintegrations/static/images/gitlabci/icon.png
1 more

Checks run (1 failed, 1 succeeded)

flake8 failed.
JSHint passed.

flake8

misery
misery
misery
Review request changed
Commits:
Summary ID
Add support of gitlab-ci
55e66ce2f3d443172107b2b0b26d1016078939e6
Add support of gitlab-ci
69d3ba54b4e7d0f63e5e140af85a0d08a7ca7c8a
Diff:

Revision 11 (+1204)

Show changes

setup.py
rbintegrations/extension.py
rbintegrations/gitlabci/__init__.py
rbintegrations/gitlabci/forms.py
rbintegrations/gitlabci/integration.py
rbintegrations/gitlabci/urls.py
rbintegrations/gitlabci/views.py
rbintegrations/static/images/gitlabci/icon.png
1 more

Checks run (1 failed, 1 succeeded)

flake8 failed.
JSHint passed.

flake8

misery
misery
misery
chipx86
  1. Thanks for the change, and for your patience while we worked on a launch!

    The logic all looks great. There's style guideline stuff, which is the bulk of the review. We're happy to finish off the work if you'd like to pass it to us, but of course we're also happy to work with you through iterations of the code :)

    Definitely want to get this in for the next release.

    1. Feel free to adopt this. I will add "input variables" of gitlab 17.11 in the next weeks if our internal instance is updated. Do you want to merge this soon? So I can add another request for the input-feature.

      https://about.gitlab.com/releases/2025/04/17/gitlab-17-11-released/#cicd-pipeline-inputs

  2. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
    Show all issues

    This will need a Version Added: with the target version.

    We can take care of this when we land it, but I'm noting this so it doesn't get missed.

    Same wiht the __init__.py and other modules.

  3. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
     
     
    Show all issues

    We'll need a from __future__ import annotations import here.

  4. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
    Show all issues

    gitlab is a third-pary library, so it'll go in the same import group as django (with import ... listed above from ...).

  5. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
     
     
    Show all issues

    I think we'll want a clean_gitlab_vars method that performs a JSON validation.

  6. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
     
     
     
     
     
     
     
     
     
     
    Show all issues

    Formatting for both these help texts should be the same as the one in gitlab_old_pipeline.

  7. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
    Show all issues

    This needs a -> dict[str, Any] type hint.

  8. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
    Show all issues

    This can just be super().clean(), since we're on Python 3.

  9. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
     
     
     
     
     
     
     
    Show all issues

    Let's also catch a plain Exception with logging, so we can handle any unforeseen issues.

  10. rbintegrations/gitlabci/forms.py (Diff revision 14)
     
     
     
     
     
     
     
    Show all issues

    This can live in clean_gitlab_vars.

  11. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
    Show all issues

    This belongs in the third-party import group with djblets.

  12. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
    Show all issues

    This needs a Version Added.

  13. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
    Show all issues

    This can be dict[str, str], which means we can also get rid of the Dict import.

  14. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
     
    Show all issues

    Since this is a new module, the methods don't need Version Added. The class's will cover it.

  15. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
    Show all issues

    Since BuildPrepData is located in another module, we'll need to specify the full module path to it in the docs.

  16. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
    Show all issues

    Same note as before.

  17. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
    Show all issues

    For dictionaries, we use this format:

    variables = [
        {
            'key': 'REVIEWBOARD_SERVER',
            'value': patch_info['reviewboard_server'],
        },
        ...
    ]
    

    Note also the single quotes. We use double quotes only when containing text needing a single quote.

  18. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
     
    Show all issues

    We use a blank line between statements and blocks.

  19. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
     
    Show all issues

    Here as well.

  20. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
    Show all issues

    This should use the same format shown above.

  21. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
    Show all issues

    Looks like left-over debug code. Same below.

  22. rbintegrations/gitlabci/integration.py (Diff revision 14)
     
     
     
    Show all issues

    Same note about dictionary formatting.

  23. rbintegrations/gitlabci/urls.py (Diff revision 14)
     
     
    Show all issues

    This module needs a docstring with a Version Added.

  24. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
     
    Show all issues

    We need a Version Added and an annotations import.

  25. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
    Show all issues

    This needs a Version Added.

  26. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
    Show all issues

    These should use single quotes and should sort keys alphabetically, to ease lookup and maintenance.

  27. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
    Show all issues

    This should include typing for request and the return type HttpResponseBase.

  28. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
    Show all issues

    Blank line between statements and blocks. Same below.

    Along with that, single quotes instead of double quotes, here and below.

  29. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
     
     
     
    Show all issues

    We prefer this format for conditionals:

    if (payload is None or
        payload.get('object_kind') != 'pipeline' or
        'object_attributes' not in payload):
        ...
    

    Same below.

  30. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
    Show all issues

    For dictionary comprehensions, we use this format:

    v = {
        item['key'] : item['value']
        for item in payload['variables']
    }
    
  31. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
    Show all issues

    Blank line between statements and conditionals here as well.

  32. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
    Show all issues

    We'll want typing for any new methods, for both arguments and return types.

  33. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
    Show all issues

    This is missing a trailing period.

  34. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
     
     
     
     
    Show all issues

    The trailing ) should be on the same parameter line.

  35. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
    Show all issues

    We should localize all errors. Here and below.

  36. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
     
     
    Show all issues

    Same note as above with logging. We also prefer for logging variables to be on a separate line from the message, to make it easier to manage the text.

  37. rbintegrations/gitlabci/views.py (Diff revision 14)
     
     
    Show all issues

    When saving, we should take to pass in update_fields=(...) with every field we explicitly want to save. This avoids side-effects with other state that may be in the database (e.g., extra_data) and avoids re-saving fields that haven't changed.

  38. setup.py (Diff revision 14)
     
     
    Show all issues

    We'll need to cap this dependency so there aren't any surprises down the road.

  39. 
      
misery
misery
Review request changed
Commits:
Summary ID
Add support of gitlab-ci
4f20e6d7d519951c160f2a1a866f48bf9ad2dce8
Add support of gitlab-ci
f911e29b30630c4abe1a2585fbff3fae550a560e
Diff:

Revision 16 (+1704)

Show changes

setup.py
rbintegrations/extension.py
rbintegrations/gitlabci/__init__.py
rbintegrations/gitlabci/forms.py
rbintegrations/gitlabci/integration.py
rbintegrations/gitlabci/urls.py
rbintegrations/gitlabci/views.py
rbintegrations/static/images/gitlabci/icon.png
1 more

Checks run (1 failed, 1 succeeded)

flake8 failed.
JSHint passed.

flake8

misery
  1. 
      
  2. rbintegrations/gitlabci/integration.py (Diff revision 16)
     
     
     
     
     
    Show all issues

    das wird wohl in der nächsten python-gitlab funktionieren...

    https://github.com/caillouc/python-gitlab/commit/a4c0444124774b548504c63d39f8bc9a14121933

  3. 
      
Loading...