Allow and abstract private extra data in the API
Review Request #7674 — Created Oct. 2, 2015 and submitted
The webapi allows for arbitrary data to be set in
extra_data
for certain API resources. Currently, all of that information is accessible through the API which is not always desirable.
The change in progress will allow users to specify privateextra_data
which will be abstracted from the publicly api available information.Change prohibits any kind of modification to private variables (denoted by a leading double-undersore
__
) in resources containing extra_data, as well as removing the private fields when a resource is requested.Added
APIExtraDataHook
to allow extension writers to give access states to extra_data fields.
Added new PUT, POST and GET tests for
__
denoted private extra_data.
Ran all tests.Added new tests for testing the
APIExtraDataHook
.
Description | From | Last Updated |
---|---|---|
Col: 35 E703 statement ends with a semicolon |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 19 E231 missing whitespace after ';' |
reviewbot | |
Col: 32 E713 test for membership should be 'not in' |
reviewbot | |
Col: 35 E703 statement ends with a semicolon |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 19 E231 missing whitespace after ';' |
reviewbot | |
Col: 32 E713 test for membership should be 'not in' |
reviewbot | |
Col: 35 E703 statement ends with a semicolon |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 19 E231 missing whitespace after ';' |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 39 E203 whitespace before ':' |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Col: 1 W293 blank line contains whitespace |
reviewbot | |
When we wrap conditionals, instead of using \, we like to surround the whole thing in parens. |
david | |
Long docstrings should have a single summary line, followed by a blank line, followed by more detailed description. Please also … |
david | |
This function doesn't seem to return anything? |
david | |
redefinition of unused 'test_post_with_extra_fields' from line 7 |
reviewbot | |
Col: 39 E203 whitespace before ':' |
reviewbot | |
Col: 39 E203 whitespace before ':' |
reviewbot | |
Col: 21 W503 line break before binary operator |
reviewbot | |
This funciton body should still be empty. |
brennie | |
This method should be called serialize_extra_data. That way, when serializing any field called extra_data, the web API will know to … |
brennie | |
Given what I said above, we shouldnt need this function. |
brennie | |
This should be a method on the class because it doesn't need closure around any variables. |
brennie | |
Blank line between these. |
brennie | |
Instead of POSTing, can we set the extra_data to a nested structure and do a GET test instead? |
brennie | |
How about: "Strip private fields from an extra data object." |
brennie | |
through the provided object |
brennie | |
"The object ..." |
brennie | |
You don't need the hasattr call. This function will only be called when there is an extra_data field. Also, if … |
brennie | |
Col: 19 W291 trailing whitespace |
reviewbot | |
The not should be aligned with the self. (I know the pep8 tool will complain here, but we ignore this … |
chipx86 | |
So, this is going to actually modify the object itself, which is bad if the object then gets saved later. … |
chipx86 | |
No comma needed before the "as well as" |
CH chronicleyu | |
Should end in a period. |
brennie | |
Should be on previous line. |
brennie | |
Blank line between these. |
brennie | |
"Return" (imperitive mood) |
brennie | |
Reformat this as: key_path (tuple): A tuple of strings... |
brennie | |
unicode |
brennie | |
Missing a type. in the case of unions, (type or type or type ... or type). |
brennie | |
'APIExtraDataAccessHook' imported but unused |
reviewbot | |
Blank line between these. |
brennie | |
This should probably be: super(APIExtraDataAccessHookTests, self).tearDown() |
mike_conley | |
This should go with the standard library imports. |
brennie | |
Single quotes |
brennie | |
This should probably be set in __init__. |
brennie | |
I believe you mean (self.EXTRA_DATA_ACCESS_PUBLIC, None) |
brennie | |
"Return" |
brennie | |
See above formatting comment. |
brennie | |
unicode |
brennie | |
No comma here. |
brennie | |
We can avoid a backslash here: url, mimetype, data, objs = self.setup_basic_post_test( self.user, False, None, True) |
brennie | |
Should be: unicode: The acess state of the provided field. |
brennie | |
Blank line between these. |
brennie | |
Blank line between these. |
brennie | |
Don't do import inside of a class. Do it inside defs if you need it locally |
brennie | |
'ExtensionInfo' imported but unused |
reviewbot | |
Col: 80 E501 line too long (93 > 79 characters) |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 33 E203 whitespace before ':' |
reviewbot | |
Col: 34 E203 whitespace before ':' |
reviewbot | |
Col: 35 E203 whitespace before ':' |
reviewbot | |
Needs a docstring. |
brennie | |
Col: 80 E501 line too long (87 > 79 characters) |
reviewbot | |
local variable 'root' is assigned to but never used |
reviewbot | |
Col: 9 E303 too many blank lines (2) |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
Can you seperate the constants from the overridden class attributes with a newline? |
brennie | |
Each of these should have a doc comment, e.g.: #: A public ``extra_data`` access state indicates everyone(?) can retrieve and … |
brennie | |
Remove this line. |
brennie | |
This code is very highly nested. Can we break this out into another function, e.g., _process_extra_data ? |
brennie | |
Just unicode: .... |
brennie | |
We probably don't want to call callback twice. Instead, we should cache the return value of it, e.g. value = … |
brennie | |
Blank line between these. |
brennie | |
If there is no access state, shouldn't it be public or something? I might be misunderstanding this. It should probably … |
brennie | |
How about "The object from which private fields will be stripped" |
brennie | |
Needs a docstring. |
brennie | |
Doing list(six.iteritems(d)) will remove all the benefits of doing six.iteritems in the first place. In Python 2, d.items() generates a … |
brennie | |
We avoid the Python ternary expression. Rewrite this as: if parent_path: constructed_path = parent_path + (k,) else: constructed_path = k |
brennie | |
Needs a docstring. |
brennie | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 33 E203 whitespace before ':' |
reviewbot | |
Col: 34 E203 whitespace before ':' |
reviewbot | |
Col: 35 E203 whitespace before ':' |
reviewbot | |
Col: 80 E501 line too long (87 > 79 characters) |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
local variable 'rsp' is assigned to but never used |
reviewbot | |
This should live in strip_private_data. If a subclass overwrites it and wants to implement the behaviour, it should use super. |
brennie | |
Can we rename k to field_name and v to value? |
brennie | |
My previous comment here wasn't clear and I didn't notice the dictionary modification while iterating issue. However, instead of using … |
brennie | |
What if you get rid of registration entirely? From looking at djblets/extensions/manager.py, it looks like if it doesn't find one, … |
mike_conley | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 33 E203 whitespace before ':' |
reviewbot | |
Col: 34 E203 whitespace before ':' |
reviewbot | |
Col: 35 E203 whitespace before ':' |
reviewbot | |
Col: 9 E303 too many blank lines (2) |
reviewbot | |
local variable 'accesshook' is assigned to but never used |
reviewbot | |
local variable 'extension' is assigned to but never used |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
local variable 'rsp' is assigned to but never used |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 33 E203 whitespace before ':' |
reviewbot | |
Col: 34 E203 whitespace before ':' |
reviewbot | |
Col: 35 E203 whitespace before ':' |
reviewbot | |
Col: 9 E303 too many blank lines (2) |
reviewbot | |
local variable 'accesshook' is assigned to but never used |
reviewbot | |
'ReviewRequestResource' imported but unused |
reviewbot | |
Col: 80 E501 line too long (85 > 79 characters) |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
local variable 'rsp' is assigned to but never used |
reviewbot | |
Col: 25 E225 missing whitespace around operator |
reviewbot | |
Col: 29 E127 continuation line over-indented for visual indent |
reviewbot | |
Col: 31 E231 missing whitespace after ':' |
reviewbot | |
Col: 33 E203 whitespace before ':' |
reviewbot | |
Col: 34 E203 whitespace before ':' |
reviewbot | |
Col: 35 E203 whitespace before ':' |
reviewbot | |
local variable 'accesshook' is assigned to but never used |
reviewbot | |
local variable 'list_mimetype' is assigned to but never used |
reviewbot | |
Col: 19 E702 multiple statements on one line (semicolon) |
reviewbot | |
local variable 'rsp' is assigned to but never used |
reviewbot | |
'augment_method_from' imported but unused |
reviewbot | |
Col: 27 E702 multiple statements on one line (semicolon) |
reviewbot | |
"A hook." |
brennie | |
Missing period. |
brennie | |
This should be the first funciton. |
brennie | |
this.class.path should be the actual class path, e.g. reviewboard.webapi.base.WebAPIResource |
brennie | |
This is probably unnecessary. |
brennie | |
Second line needs a colon. Comments like thsould be of the form of: #: Single line summary #: #: Multi-line … |
brennie | |
Same here. |
brennie | |
Same here. |
brennie | |
Since this is returning a boolean, its name should reflect that. Can we rename this to _should_processes_extra_data? |
brennie | |
Needs Args and Returns |
brennie | |
Needs docstring. |
brennie | |
Needs docstring |
brennie | |
Just "key path" |
brennie | |
Blank line between a statement and the start (or end) of a block. |
brennie | |
``extra_data`` Just two backslashes. Also, how about: A clone of the ``extra_data`` stripped of its private fields. |
brennie | |
Space between if and (. |
brennie | |
Can we add a helper function self._is_private(key) that just does return self.get_extra_data_field_state(path) == self.EXTRA_DATA_ACCESS_PRIVATE ? It will make this easier … |
brennie | |
Needs Args and Returns. |
brennie | |
Can you add some examples of nested extra data? e.g. extra_fields = { '__private_key': 'private_data', 'public_key': { '__another_private_key': 'foo', 'another_public_key': … |
brennie | |
Col: 16 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 26 E127 continuation line over-indented for visual indent |
reviewbot | |
This should be a lot more fleshed out, highlighting what kind of things can be set and how this impacts … |
chipx86 | |
You'll have to build the docs and see if these resolve with the line breaks. You can also prefix a … |
chipx86 | |
No need for enumerate here. That's only useful when you need the index, which you're not using. |
chipx86 | |
Needs a doc string. |
chipx86 | |
Must be in alphabetical order. |
chipx86 | |
I would declare this outside of setUp, because otherwise, the entire class is being re-created every time this is run. … |
chipx86 | |
Not too useful for the test. |
chipx86 | |
This needs to be a tuple. As it is, this is going to be treated as a tuple of ('G', … |
chipx86 | |
Docstrings aren't really necessary for test objects like this one. We'll never generate docs from them, and they're not useful … |
chipx86 | |
Should be formatted like: data = { 'extra_data': ... } |
chipx86 | |
I know other classes are doing all this. Instead of repeating all that, create a mixin to keep it all … |
chipx86 | |
This shouldn't be here. |
chipx86 | |
Should be one statement. |
chipx86 | |
Should be one statement. However, this sort of thing doesn't belong in setUp(), and instead should be in the tests … |
chipx86 | |
Multi-line dicts shoudl always be in the form of: extra_fields = { key: value, key: value, } Also, this can … |
chipx86 | |
Your tests here are doing too much. They're claiming to test registration, but then they're testing usage. Instead, you should … |
chipx86 | |
In this case, you should start arguments on the opening line. |
chipx86 | |
Should be part of the statement below. Having this in its own variable just reduces readability. |
chipx86 | |
Same as above. |
chipx86 | |
Same as above. |
chipx86 | |
Second line is indented too far. |
chipx86 | |
Let's pull these out into a dedicated class: ExtraDataAccessLevel. That will also help with the documentation, which won't have to … |
chipx86 | |
These must still follow the single-line summary, multi-line description format. |
chipx86 | |
Blank line between these. |
chipx86 | |
"if an ..." |
chipx86 | |
The descriptions belong on the following line, indented 4 spaces. |
chipx86 | |
No blank line here. Description should be on the following line. |
chipx86 | |
The second line should probably be indented 4 spaces, since otherwise it's a standalone condition per line. |
chipx86 | |
"Register" How about: "Register a function for determining access to an extra_data key." Same general idea in functions below. |
chipx86 | |
This is repeating the summary. Instead, flesh it out with more information about the purpose of registering such a thing. |
chipx86 | |
This should probably check that this wasn't already registered, and raise a ValueError with a suitable error message if so … |
chipx86 | |
This is repeating the summary. |
chipx86 | |
This can fail with a ValueError. Catch that and return a new ValueError with a more suitable error message (and … |
chipx86 | |
This should mention "extra_data", like the others. |
chipx86 | |
When adjusting the types above, this will be an int. |
chipx86 | |
Blank line between these. |
chipx86 | |
Too many backticks. Should just be 2. |
chipx86 | |
This needs a more specific name, and documentation. |
chipx86 | |
Public functions go above private ones. |
chipx86 | |
"Serialize" (no 's'), and "resource's". |
chipx86 | |
Description on the next line. |
chipx86 | |
The return type should come first, rather than being mentioned in parens. |
chipx86 | |
Your unit tests are only testing the __ ability. Instead, for each suitable HTTP method, you need to test: __ … |
chipx86 | |
Col: 80 E501 line too long (83 > 79 characters) |
reviewbot | |
Col: 80 E501 line too long (95 > 79 characters) |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 13 E128 continuation line under-indented for visual indent |
reviewbot | |
Col: 12 E401 multiple imports on one line |
reviewbot | |
Col: 21 E127 continuation line over-indented for visual indent |
reviewbot | |
Col: 30 E127 continuation line over-indented for visual indent |
reviewbot | |
Col: 30 E127 continuation line over-indented for visual indent |
reviewbot | |
Col: 1 E303 too many blank lines (3) |
reviewbot | |
Col: 29 W292 no newline at end of file |
reviewbot | |
Col: 80 E501 line too long (83 > 79 characters) |
reviewbot | |
Col: 80 E501 line too long (95 > 79 characters) |
reviewbot | |
Col: 21 E127 continuation line over-indented for visual indent |
reviewbot | |
Col: 1 E303 too many blank lines (3) |
reviewbot | |
Col: 80 E501 line too long (83 > 79 characters) |
reviewbot | |
Col: 80 E501 line too long (95 > 79 characters) |
reviewbot | |
This should be done in setUp(), i.e., def setUp(self): super(ExtensionManagerMixin, self).setUp() self.manager = ExtensionManager('') |
brennie | |
Col: 21 E127 continuation line over-indented for visual indent |
reviewbot | |
Can we call this register_extra_data_access_callback ? |
brennie | |
Likewise unregister_extra_data_access_callback |
brennie | |
Col: 80 E501 line too long (83 > 79 characters) |
reviewbot | |
Col: 80 E501 line too long (95 > 79 characters) |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 5 E303 too many blank lines (2) |
reviewbot | |
Col: 28 E712 comparison to True should be 'if cond is True:' or 'if cond:' |
reviewbot | |
Col: 80 E501 line too long (83 > 79 characters) |
reviewbot | |
Col: 9 E303 too many blank lines (2) |
reviewbot | |
Col: 21 E127 continuation line over-indented for visual indent |
reviewbot | |
Too many spaces after "An". |
chipx86 | |
The PRIVATE, READ_ONLY, etc. don't mean much at this point when reading through the docs. I'd try rethinking how you … |
chipx86 | |
This isn't where they live now. |
chipx86 | |
This needs to point to the class owning the method. It should link properly when building the docs. |
chipx86 | |
provides -> provided |
CH chronicleyu | |
Two blank lines between sections. |
chipx86 | |
The source code should start 3 characters in, aligned with the code-block. |
chipx86 | |
No parens here. Instead, use \\. |
chipx86 | |
This isn't very readable. It should be: access_hook = APIExtraDataAccessHook( self, review_request_resource, [ (('private',), ExtraDataAccessLevel.ACCESS_STATE_PRIVATE), ]) (Note that it's not … |
chipx86 | |
This isn't valid Python, and we absolutely don't want to be documenting that monkey-patching a hook is valid. Instead, show … |
chipx86 | |
Typo on the hook variable name. |
chipx86 | |
No blank line at the end of the file. |
chipx86 | |
The wrapping is pretty funky in a lot of these. It should wrap close to the 79 character column. |
chipx86 | |
Col: 80 E501 line too long (86 > 79 characters) |
reviewbot | |
This should be: ```python """ Example: ... code-block:: python resource.extra_Data = { ... """ |
chipx86 | |
Col: 80 E501 line too long (98 > 79 characters) |
reviewbot | |
How does this look? It's a tuple, but we're comparing against the field_set. Can you show me an example of … |
chipx86 | |
None is a possible result, and should be documented. |
chipx86 | |
Docstring summaries cannot wrap. |
chipx86 | |
Blank line between these. |
chipx86 | |
No parens, and space after except. |
chipx86 | |
Alphabetical order (in the import ... list). |
chipx86 | |
This should have a docstring. |
chipx86 | |
Combine this to one statement. |
chipx86 | |
Should be: return 200, { 'test': self.extra_data, } |
chipx86 | |
There's no reason to have this function. |
chipx86 | |
Docstring summaries can't wrap, but in this case, don't bother with documenting this. It's not useful, since it's just a … |
chipx86 | |
Blank line between a class's docstring and its body. |
chipx86 | |
Here, too. |
chipx86 | |
Blank line between these. |
chipx86 | |
No trailing periods on the unit test docstrings (here and others). |
chipx86 | |
This (and others like it) are formatted awkwardly. Instead, try: APIExtraDataAccessHook( self.extension, self.resource, [ (('public',), ExtraDataAccessLevel.ACCESS_STATE_PUBLIC) ]) |
chipx86 | |
While assertEquals and assertNotEquals are valid aliases, we only use assertEqual and assertNotEqual. Make sure all the tests are using … |
chipx86 | |
A bunch of these are missing docstrings. |
chipx86 | |
Col: 21 E127 continuation line over-indented for visual indent |
reviewbot | |
When viewing a log, this isn't going to mean much. I wouldn't log this. The caller will see the error. … |
chipx86 | |
callbable -> callable |
CH chronicleyu | |
This should provide an example of what's expected in key_path. Maybe have an "Example" section, like I document elsewhere. |
chipx86 | |
If this is meant only for internal use, it should have a _ prefix. |
chipx86 | |
Second line should not be indented. It should be on the same level as the prior line. |
chipx86 | |
serialize functions traditionally go higher up, below has_access_permissions. |
chipx86 | |
Is it worth having this method? It's just a simple comparison, and is only used in one place. Best to … |
chipx86 | |
Must inherit from object. Should also be placed in alphabetical order. |
chipx86 | |
Blank line between a class's docstring and its body. |
chipx86 | |
These must follow traditional docstring semantics of a summary line (max one sentence), blank line, description. |
chipx86 | |
Col: 80 E501 line too long (86 > 79 characters) |
reviewbot | |
Col: 80 E501 line too long (98 > 79 characters) |
reviewbot | |
Col: 21 E127 continuation line over-indented for visual indent |
reviewbot | |
Col: 80 E501 line too long (86 > 79 characters) |
reviewbot | |
Col: 80 E501 line too long (98 > 79 characters) |
reviewbot | |
Col: 21 E127 continuation line over-indented for visual indent |
reviewbot | |
indentation is off |
brennie |
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py
-
-
-
-
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py
-
-
- Description:
-
--WIP--
The webapi allows for arbitrary data to be set in
extra_data
for certain API resources. Currently, all of that information is accessible through the API which is not always desirable.The change in progress will allow users to specify private extra data which will be abstracted from the publicly api available information. + + Associated djiblets change: https://reviews.reviewboard.org/r/7675/
- Description:
-
--WIP--
The webapi allows for arbitrary data to be set in
extra_data
for certain API resources. Currently, all of that information is accessible through the API which is not always desirable.The change in progress will allow users to specify private extra data which will be abstracted from the publicly api available information. + Change prohibits any kind of modification to private variables (denoted by a leading double-undersore
__
) in resources containing extra_data.+ Associated djiblets change: https://reviews.reviewboard.org/r/7675/
- Groups:
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/resources/review_request_draft.py
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py
-
Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py
- Description:
-
--WIP--
The webapi allows for arbitrary data to be set in
extra_data
for certain API resources. Currently, all of that information is accessible through the API which is not always desirable.~ The change in progress will allow users to specify private extra data which will be abstracted from the publicly api available information. ~ The change in progress will allow users to specify private extra_data
which will be abstracted from the publicly api available information.~ Change prohibits any kind of modification to private variables (denoted by a leading double-undersore
__
) in resources containing extra_data.~ Change prohibits any kind of modification to private variables (denoted by a leading double-undersore
__
) in resources containing extra_data, as well as removing the private fields when a resource is requested.- - Associated djiblets change: https://reviews.reviewboard.org/r/7675/
-
Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py
-
Looks good, but there's a subtle bug that can manifest with the approach. Shouldn't be terrible to fix, though.
You should also reach out to Steven MacLeod in Slack about this, because I know he had things he needed beyond a
__
prefix. Essentially, we'll want to allow subclasses to be able to make a determination as to whether a key should be private, even without the__
(so that later we can allow extensions to strip these out). -
The
not
should be aligned with theself.
(I know the pep8 tool will complain here, but we ignore this one because it's ridiculous, and not even a PEP-8 issue, more a personal preference of the tool author.)
-
So, this is going to actually modify the object itself, which is bad if the object then gets saved later. These functions should instead generate new dictionaries, returning those, instead of modifying the provided dictionary.
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
- Description:
-
--WIP--
The webapi allows for arbitrary data to be set in
extra_data
for certain API resources. Currently, all of that information is accessible through the API which is not always desirable.The change in progress will allow users to specify private extra_data
which will be abstracted from the publicly api available information.Change prohibits any kind of modification to private variables (denoted by a leading double-undersore
__
) in resources containing extra_data, as well as removing the private fields when a resource is requested.+ + Added
APIExtraDataHook
to allow extension writers to give access states to extra_data fields. - Testing Done:
-
~ Added new PUT, POST and GET tests for private extra_data.
~ Added new PUT, POST and GET tests for
__
denoted private extra_data.Ran all tests. + + Still need to write hook tests
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Each of these should have a doc comment, e.g.:
#: A public ``extra_data`` access state indicates everyone(?) can retrieve and edit it. EXTRA_DATA_ACCESS_PUBLIC = 'PUBLIC' #: A readonly public ....
For longer doc comments, they should be formatted as docstrings, e.g.
#: Single line summary. #: #: Multi-line description. #: Description continued. thing = Foo()
-
-
This code is very highly nested. Can we break this out into another function, e.g.,
_process_extra_data
? -
-
We probably don't want to call
callback
twice. Instead, we should cache the return value of it, e.g.value = callback(key_path) if value is not None: return value
-
-
If there is no access state, shouldn't it be public or something?
I might be misunderstanding this. It should probably be documented.
-
-
-
Doing
list(six.iteritems(d))
will remove all the benefits of doingsix.iteritems
in the first place.In Python 2,
d.items()
generates a list of key-value pairs. However, that is a waste of memory unless you are explicitly going to use it as a list. In the case of iteration, we generally want to usesix.iteritems(d)
. This uses a generator toyield
all the key-value pairs instead of constructing a list (and wasting memory by doing so).Doing
list(six.iteritems(d))
will construct a list from the generator, wasting as much memory as just doingd.items()
, but also being less efficient because there are going to be an an extraO(n)
function calls. -
We avoid the Python ternary expression.
Rewrite this as:
if parent_path: constructed_path = parent_path + (k,) else: constructed_path = k
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
-
-
-
-
-
-
-
-
This should live in
strip_private_data
. If a subclass overwrites it and wants to implement the behaviour, it should use super. -
My previous comment here wasn't clear and I didn't notice the dictionary modification while iterating issue. However, instead of using
.items()
(which will blow up on Py3 because it is an iterator there), we can take a modified approach.Instead of passing a deep copy of
extra_data
tostrip_private_data
, we can instead pass the originalextra_data
and make the copy there. Then we can iterate through the originalextra_data
's keys and modify our copy.In essence:
def strip_private_data(self, extra_data, parent_path=None): clone = copy.copy(extra_data) for field_name, value in six.iteritems(extra_data): if parent_path: path = parent_path + (field_name,) else: path = field_name if (field_name.startswith(PRIVATE_KEY_PREFIX) or self.get_extra_data_field_state(constructed_path) == self.EXTRA_DATA_ACCESS_PRIVATE): del clone[field_name] elif isinstance(value, dict): clone[field_name] = self.strip_private_data(clone[field_name], path) return clone def seralize_extra_data_field: if obj.extra_data is not None: return self.strip_private_data(obj.extra_data) return None
We no longer require doing a
deepcopy
of theextra_data
becuase we do a shallow copy at each level.
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
-
-
-
-
-
-
-
-
-
-
What if you get rid of
registration
entirely? From looking at djblets/extensions/manager.py, it looks like if it doesn't find one, it'll create one itself. Looking at the other tests in that folder as well (like the "ExtensionManagerTest") the TestExtension doesn't define aregistration
member. So try leaving that out?
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
-
-
-
-
-
-
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
-
-
-
-
-
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
- Description:
-
- --WIP--
- The webapi allows for arbitrary data to be set in
extra_data
for certain API resources. Currently, all of that information is accessible through the API which is not always desirable.The change in progress will allow users to specify private extra_data
which will be abstracted from the publicly api available information.Change prohibits any kind of modification to private variables (denoted by a leading double-undersore
__
) in resources containing extra_data, as well as removing the private fields when a resource is requested.Added
APIExtraDataHook
to allow extension writers to give access states to extra_data fields. - Testing Done:
-
Added new PUT, POST and GET tests for
__
denoted private extra_data.Ran all tests. ~ Still need to write hook tests
~ Added new tests for testing the
APIExtraDataHook
.
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
-
-
-
-
-
Second line needs a colon. Comments like thsould be of the form of:
#: Single line summary #: #: Multi-line description
just like a docstring.
-
-
-
Since this is returning a boolean, its name should reflect that.
Can we rename this to
_should_processes_extra_data
? -
-
-
-
-
-
``extra_data``
Just two backslashes.
Also, how about:
A clone of the ``extra_data`` stripped of its private fields.
-
-
Can we add a helper function
self._is_private(key)
that just doesreturn self.get_extra_data_field_state(path) == self.EXTRA_DATA_ACCESS_PRIVATE
?It will make this easier to read.
-
-
Can you add some examples of nested extra data?
e.g.
extra_fields = { '__private_key': 'private_data', 'public_key': { '__another_private_key': 'foo', 'another_public_key': 'bar', }, }
And verify that works as intended.
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py
-
This is looking great. I do have a lot of comments (I know that wall of yellow looks scary), but most are wording changes in docs or little things that need changing. There are a couple bigger-ticket items for how some stuff is structured/named, but you'll plow through those fast. The biggest is that the test coverage isn't complete, so I'll want to see some more things there (I outline this below). Having that coverage will help prove your implementation.
-
This should be a lot more fleshed out, highlighting what kind of things can be set and how this impacts the API. This will be used as a base for documentation on the hook.
-
You'll have to build the docs and see if these resolve with the line breaks.
You can also prefix a module path like this with
~
so that the generated docs only show the last component of the path. -
-
-
-
I would declare this outside of
setUp
, because otherwise, the entire class is being re-created every time this is run. Top-level is fine, if you name the class right. -
-
-
Docstrings aren't really necessary for test objects like this one. We'll never generate docs from them, and they're not useful for the test.
-
-
I know other classes are doing all this. Instead of repeating all that, create a mixin to keep it all in one place.
-
-
-
Should be one statement.
However, this sort of thing doesn't belong in
setUp()
, and instead should be in the tests that use them. The reason being that as the test suite grows, having these pre-generated data sets will become less useful and more confusing (a contributor won't know if they should be using it, adding more pre-generated data sets for their tests, etc.). -
Multi-line dicts shoudl always be in the form of:
extra_fields = { key: value, key: value, }
Also, this can definitely be merged with the statement below.
-
Your tests here are doing too much. They're claiming to test registration, but then they're testing usage.
Instead, you should have a test for each of these conditions:
- Registration (just check that the resulting function is properly registered)
- Unregistration (just check that the formerly-registered function is now unregistered)
- The hook usage with a key set for the public state.
- The hook usage with a key set for the read-only state (and attempting to modify).
- The hook usage with a key set for the private state.
Fine-grained tests are better than larger, more encompassing tests, because when things go wrong, we can see if only a subset of the tests fail. This gives the developer a better idea of what broke, without having to start debugging the code.
-
-
-
-
-
-
Let's pull these out into a dedicated class:
ExtraDataAccessLevel
.That will also help with the documentation, which won't have to describe that it's an "extra_data access state" over and over.
I'd also just simply use integers for the level types. They're smaller for storage, and faster to compare.
-
-
-
-
-
-
The second line should probably be indented 4 spaces, since otherwise it's a standalone condition per line.
-
"Register"
How about: "Register a function for determining access to an extra_data key."
Same general idea in functions below.
-
This is repeating the summary. Instead, flesh it out with more information about the purpose of registering such a thing.
-
This should probably check that this wasn't already registered, and raise a
ValueError
with a suitable error message if so (which will also need to be documented).It should also check that what's provided is a
callable
. -
-
This can fail with a
ValueError
. Catch that and return a newValueError
with a more suitable error message (and document it).It should also check that what's provided is a
callable
. -
-
-
-
-
-
-
-
-
-
Your unit tests are only testing the
__
ability. Instead, for each suitable HTTP method, you need to test:__
prefixes on paths- Registered extra_data access functions returning "public".
- Registered extra_data access functions returning "read-only".
- Registered extra_data access functions returning "private".
You also need unit tests covering the register/unregister functions with the various success/fail conditions, and
get_extra_data_field_state
with the various conditions. These should go in a more dedicated unit test function, rather than being in this mixin, because we don't need to test those functions a hundred times.
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst
-
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst
-
-
-
-
-
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst
-
-
-
-
-
-
The
PRIVATE
,READ_ONLY
, etc. don't mean much at this point when reading through the docs. I'd try rethinking how you could write this to better describe the various states. Maybe say something like "The access states can be one of:" with a list mapping the:py:data:
of each access state name to a description. -
-
-
-
-
-
This isn't very readable. It should be:
access_hook = APIExtraDataAccessHook( self, review_request_resource, [ (('private',), ExtraDataAccessLevel.ACCESS_STATE_PRIVATE), ])
(Note that it's not
WebAPIResource
, and that your example must import this). -
This isn't valid Python, and we absolutely don't want to be documenting that monkey-patching a hook is valid. Instead, show an example with a subclass that overrides that function.
You should take all the stuff you're documenting in the hook and temporarily stick it in a unit test. It should be able to run correctly. That's how you know it's not going to lead users down the wrong path.
Also, ideally, this would demonstrate something with the
key_path
. -
-
-
-
-
How does this look? It's a tuple, but we're comparing against the
field_set
. Can you show me an example of how that works? Don't we need to iterate through each value inkey_path
? -
-
-
-
-
-
-
-
-
-
Docstring summaries can't wrap, but in this case, don't bother with documenting this. It's not useful, since it's just a test object for the test suite.
-
-
-
-
-
This (and others like it) are formatted awkwardly. Instead, try:
APIExtraDataAccessHook( self.extension, self.resource, [ (('public',), ExtraDataAccessLevel.ACCESS_STATE_PUBLIC) ])
-
-
When viewing a log, this isn't going to mean much. I wouldn't log this. The caller will see the error.
Same with the ones below.
-
This should provide an example of what's expected in
key_path
. Maybe have an "Example" section, like I document elsewhere. -
-
-
-
Is it worth having this method? It's just a simple comparison, and is only used in one place. Best to avoid the function call if we can.
-
-
-
These must follow traditional docstring semantics of a summary line (max one sentence), blank line, description.
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst
-
-
-
-
Tool: Pyflakes Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst Tool: PEP8 Style Checker Processed Files: reviewboard/webapi/base.py reviewboard/webapi/tests/mixins_extra_data.py reviewboard/extensions/hooks.py reviewboard/extensions/tests.py Ignored Files: docs/manual/extending/extensions/hooks/index.rst docs/manual/extending/extensions/hooks/api-extra-data-access-hook.rst
-
-
-