Add support of gitlab-ci
Review Request #14211 — Created Oct. 24, 2024 and updated
Add support of gitlab-ci
Added integration and saw that it calls the pipeline on gitlab.
Summary | ID |
---|---|
f911e29b30630c4abe1a2585fbff3fae550a560e |
Description | From | Last Updated |
---|---|---|
Also it would be a good idea to provide a gitlab component for patching like this. templates/ref.yml spec: --- .reviewboard: … |
|
|
're' imported but unused Column: 1 Error code: F401 |
![]() |
|
're' imported but unused Column: 1 Error code: F401 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line too long (97 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (92 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (86 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (87 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
This will need a Version Added: with the target version. We can take care of this when we land it, … |
|
|
We'll need a from __future__ import annotations import here. |
|
|
gitlab is a third-pary library, so it'll go in the same import group as django (with import ... listed above … |
|
|
I think we'll want a clean_gitlab_vars method that performs a JSON validation. |
|
|
Formatting for both these help texts should be the same as the one in gitlab_old_pipeline. |
|
|
This needs a -> dict[str, Any] type hint. |
|
|
This can just be super().clean(), since we're on Python 3. |
|
|
Let's also catch a plain Exception with logging, so we can handle any unforeseen issues. |
|
|
This can live in clean_gitlab_vars. |
|
|
This belongs in the third-party import group with djblets. |
|
|
This needs a Version Added. |
|
|
This can be dict[str, str], which means we can also get rid of the Dict import. |
|
|
Since this is a new module, the methods don't need Version Added. The class's will cover it. |
|
|
Since BuildPrepData is located in another module, we'll need to specify the full module path to it in the docs. |
|
|
Same note as before. |
|
|
For dictionaries, we use this format: variables = [ { 'key': 'REVIEWBOARD_SERVER', 'value': patch_info['reviewboard_server'], }, ... ] Note also the … |
|
|
We use a blank line between statements and blocks. |
|
|
Here as well. |
|
|
This should use the same format shown above. |
|
|
Looks like left-over debug code. Same below. |
|
|
Same note about dictionary formatting. |
|
|
This module needs a docstring with a Version Added. |
|
|
We need a Version Added and an annotations import. |
|
|
This needs a Version Added. |
|
|
These should use single quotes and should sort keys alphabetically, to ease lookup and maintenance. |
|
|
This should include typing for request and the return type HttpResponseBase. |
|
|
Blank line between statements and blocks. Same below. Along with that, single quotes instead of double quotes, here and below. |
|
|
We prefer this format for conditionals: if (payload is None or payload.get('object_kind') != 'pipeline' or 'object_attributes' not in payload): ... … |
|
|
For dictionary comprehensions, we use this format: v = { item['key'] : item['value'] for item in payload['variables'] } |
|
|
Blank line between statements and conditionals here as well. |
|
|
We'll want typing for any new methods, for both arguments and return types. |
|
|
This is missing a trailing period. |
|
|
The trailing ) should be on the same parameter line. |
|
|
We should localize all errors. Here and below. |
|
|
Same note as above with logging. We also prefer for logging variables to be on a separate line from the … |
|
|
When saving, we should take to pass in update_fields=(...) with every field we explicitly want to save. This avoids side-effects … |
|
|
We'll need to cap this dependency so there aren't any surprises down the road. |
|
|
line too long (83 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (82 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (83 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (81 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (81 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (81 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
das wird wohl in der nächsten python-gitlab funktionieren... https://github.com/caillouc/python-gitlab/commit/a4c0444124774b548504c63d39f8bc9a14121933 |
|
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 13 Error code: W503 |
![]() |
|
line break before binary operator Column: 17 Error code: W503 |
![]() |
|
line too long (90 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line too long (82 > 79 characters) Column: 80 Error code: E501 |
![]() |
|
line break before binary operator Column: 17 Error code: W503 |
![]() |
- Summary:
-
WIP Add support of gitlab-ciAdd support of gitlab-ci
- Description:
-
~ WIP Add support of gitlab-ci
~ Add support of gitlab-ci
- Testing Done:
-
+ Added integration and saw that it calls the pipeline on gitlab.
- Commits:
-
Summary ID 0cd1f7e13ffea73695adcbf8fd0ed540fd28de27 1bc87e573cddb172e5f295ce2aa7fc8be7bae10c
Checks run (1 failed, 1 succeeded)
flake8
- Commits:
-
Summary ID 1bc87e573cddb172e5f295ce2aa7fc8be7bae10c 1bc87e573cddb172e5f295ce2aa7fc8be7bae10c
- Commits:
-
Summary ID 1bc87e573cddb172e5f295ce2aa7fc8be7bae10c 0307c1bd8d6cee04d6390640cde4dc21258db4d5
Checks run (2 succeeded)
-
-
The best way to notify RB from gitlab would be a webook from gitlab to RB. I wrote a tiny wsgi application that get's "pipeline event" [1] for the project/group that calls rbt. A better approach would be to include the ability to RB itself without that wrapper script.
#!/usr/bin/python import json import logging import uwsgi import subprocess log = logging.getLogger("gitlab") logging.basicConfig( format="%(asctime)s %(levelname)s: %(message)s", level="INFO" ) def health(): return None def get_state(state): if state == "pending": return ("build queued...", "pending") if state == "passed": return ("build succeeded.", "done-success") if state == "running": return ("build started...", "pending") if state == "canceling": return ("canceling build...", "pending") if state == "canceled": return ("build canceled", "error") return ("build failed", "error") def process_rbtools(v, c, token): url = c.get("url") url_text = "See pipeline" description, state = get_state(c.get("detailed_status")) cmd = [ "rbt", "status-update", "set", "--url", url, "--url-text", url_text, "-r", v.get("REVIEWBOARD_REVIEW_ID"), "-s", v.get("REVIEWBOARD_STATUS_UPDATE_ID"), "--server", v.get("REVIEWBOARD_SERVER"), "--api-token", token, "--state", state, "--description", description, "--disable-cookie-storage", "--disable-cache-storage", ] log.info(cmd) subprocess.check_call(cmd) return uwsgi.SPOOL_OK def process_pipeline(c, token): if c is not None: v = c.get("variables") if v is not None: v = {item["key"]: item["value"] for item in v} if "REVIEWBOARD_SERVER" in v: return process_rbtools(v, c, token) return uwsgi.SPOOL_OK def process_data(data): try: c = json.loads(data.get("body")) except (TypeError, ValueError) as e: log.info("Broken json: {0}".format(e)) return uwsgi.SPOOL_OK # log.debug(json.dumps(c, indent=4, sort_keys=True)) if c.get("object_kind") == "pipeline" and "object_attributes" in c: return process_pipeline(c.get("object_attributes"), data.get(b"token")) return uwsgi.SPOOL_OK def process_spooler(data): try: return process_data(data) except BaseException: log.exception("Cannot process data") return uwsgi.SPOOL_RETRY def application(environ, start_response): result = [] headers = [("Content-Type", "text/plain")] request_body = [] try: request_body_size = environ.get("CONTENT_LENGTH", 0) if request_body_size: request_body = environ["wsgi.input"].read(int(request_body_size)) except ValueError as e: log.warn("Cannot read body: {0}".format(e)) headers.append(("Content-Length", str(len(result)))) start_response("400 Bad Request", headers) return result status = "200 OK" if len(request_body) > 0 and "HTTP_X_GITLAB_TOKEN" in environ: uwsgi.spool( { b"body": request_body, b"token": environ["HTTP_X_GITLAB_TOKEN"].encode("utf-8"), } ) else: healthResult = health() if healthResult is None: status = "204 No Content" else: status = "503 Service Unavailable" result = [healthResult.encode("utf-8")] headers.append(("Content-Length", str(len(result)))) start_response(status, headers) return result uwsgi.spooler = process_spooler
[1] https://docs.gitlab.com/user/project/integrations/webhook_events/#pipeline-events
-
Also it would be a good idea to provide a gitlab component for patching like this.
templates/ref.yml
spec: --- .reviewboard: reviewboard_before_script: - | if [ "$REVIEWBOARD_SERVER" != "" ]; then apk add rbtools git rbt patch --server ${REVIEWBOARD_SERVER} --diff-revision ${REVIEWBOARD_DIFF_REVISION} ${REVIEWBOARD_REVIEW_ID} fi
templates/global_before_script.yml
spec: --- include: - component: https://github.com/reviewboard/rbintegrations/ref@main before_script: - !reference [.reviewboard, reviewboard_before_script]
customer .gitlab-ci.yml
include: - component: https://github.com/reviewboard/rbintegrations/ref@main first-job: script: - echo Hello
- Commits:
-
Summary ID 0307c1bd8d6cee04d6390640cde4dc21258db4d5 a1f1316f27e283b2f1819dee845250d15da76f4a
Checks run (2 succeeded)
- Commits:
-
Summary ID a1f1316f27e283b2f1819dee845250d15da76f4a 1b8d10f23e35d52f6d142e845c14b9b6625f4c30
Checks run (2 succeeded)
- Commits:
-
Summary ID 1b8d10f23e35d52f6d142e845c14b9b6625f4c30 07a041231afc9a85712e81b73539e7c56347cc9f
Checks run (2 succeeded)
- Commits:
-
Summary ID 07a041231afc9a85712e81b73539e7c56347cc9f fac64c4771f050aa05b5535b46f27e2526ab0c62 - Diff:
-
Revision 8 (+1054)
- Commits:
-
Summary ID fac64c4771f050aa05b5535b46f27e2526ab0c62 7cc427b0261e253904db5854bb251d80e73695bc - Diff:
-
Revision 9 (+1054)
Checks run (2 succeeded)
- Commits:
-
Summary ID 7cc427b0261e253904db5854bb251d80e73695bc 55e66ce2f3d443172107b2b0b26d1016078939e6 - Diff:
-
Revision 10 (+1080)
Checks run (2 succeeded)
- Commits:
-
Summary ID 55e66ce2f3d443172107b2b0b26d1016078939e6 69d3ba54b4e7d0f63e5e140af85a0d08a7ca7c8a - Diff:
-
Revision 11 (+1204)
Checks run (1 failed, 1 succeeded)
flake8
- Commits:
-
Summary ID 69d3ba54b4e7d0f63e5e140af85a0d08a7ca7c8a 6fb99337c738eb67f9f3ac06dce3aed6ae50cf3d - Diff:
-
Revision 12 (+1216)
Checks run (2 succeeded)
- Commits:
-
Summary ID 6fb99337c738eb67f9f3ac06dce3aed6ae50cf3d 3b0c9287ab38b0d4fc4c22ea35ff942e847dcfc4 - Diff:
-
Revision 13 (+1218)
Checks run (2 succeeded)
- Commits:
-
Summary ID 3b0c9287ab38b0d4fc4c22ea35ff942e847dcfc4 fdb658c64258a4d418c2747b15fc02e7a0f37fad - Diff:
-
Revision 14 (+1218)
Checks run (2 succeeded)
-
Thanks for the change, and for your patience while we worked on a launch!
The logic all looks great. There's style guideline stuff, which is the bulk of the review. We're happy to finish off the work if you'd like to pass it to us, but of course we're also happy to work with you through iterations of the code :)
Definitely want to get this in for the next release.
-
This will need a
Version Added:
with the target version.We can take care of this when we land it, but I'm noting this so it doesn't get missed.
Same wiht the
__init__.py
and other modules. -
-
gitlab
is a third-pary library, so it'll go in the same import group asdjango
(withimport ...
listed abovefrom ...
). -
-
-
-
-
-
-
-
-
-
-
Since
BuildPrepData
is located in another module, we'll need to specify the full module path to it in the docs. -
-
For dictionaries, we use this format:
variables = [ { 'key': 'REVIEWBOARD_SERVER', 'value': patch_info['reviewboard_server'], }, ... ]
Note also the single quotes. We use double quotes only when containing text needing a single quote.
-
-
-
-
-
-
-
-
-
-
-
Blank line between statements and blocks. Same below.
Along with that, single quotes instead of double quotes, here and below.
-
We prefer this format for conditionals:
if (payload is None or payload.get('object_kind') != 'pipeline' or 'object_attributes' not in payload): ...
Same below.
-
For dictionary comprehensions, we use this format:
v = { item['key'] : item['value'] for item in payload['variables'] }
-
-
-
-
-
-
Same note as above with logging. We also prefer for logging variables to be on a separate line from the message, to make it easier to manage the text.
-
When saving, we should take to pass in
update_fields=(...)
with every field we explicitly want to save. This avoids side-effects with other state that may be in the database (e.g.,extra_data
) and avoids re-saving fields that haven't changed. -
- Commits:
-
Summary ID fdb658c64258a4d418c2747b15fc02e7a0f37fad 4f20e6d7d519951c160f2a1a866f48bf9ad2dce8 - Diff:
-
Revision 15 (+1286)
Checks run (2 succeeded)
- Commits:
-
Summary ID 4f20e6d7d519951c160f2a1a866f48bf9ad2dce8 f911e29b30630c4abe1a2585fbff3fae550a560e - Diff:
-
Revision 16 (+1704)
Checks run (1 failed, 1 succeeded)
flake8
-
-
das wird wohl in der nächsten python-gitlab funktionieren...
https://github.com/caillouc/python-gitlab/commit/a4c0444124774b548504c63d39f8bc9a14121933