Deprecate execute(), allow it to return strings or bytes and make stripping whitespace from the output optional.

Review Request #13283 — Created Sept. 22, 2023 and submitted

Information

RBTools
release-5.x

Reviewers

Our rbtools.hooks.common.execute() function uses subprocess.Popen and
Popen.communicate to execute commands and read their output. Between Python 2
and 3, subprocess.Popen(...).communicate() changed from returning a tuple of
strings to a tuple of bytes. This breaks some of our hook util methods in
rbtools.hooks.git where we try to perform string operations on the output
of execute(), but we receive bytes instead of strings.

Python 3.7 added a text parameter to Popen to control whether file
objects, stdin, stdout, and stderr are opened in text mode or binary mode.
We make use of that parameter by adding our own text parameter to execute()
to control whether to return the stdout output as a string or bytes. This
defaults to False to match the default behavior of Popen in Python 3.

This also makes it optional to strip whitespace from the output, instead of
always stripping it. This is useful when dealing with commands where we
need to preserve the whitespace in the output, such as with git diff.

This changes also deprecates the rbtools.hooks.common.execute() function
in favour of rbtools.utils.process.run_process(), which is more
future-proof and has better type safety.

  • Ran into an error when running the git pre-receive hook, which came
    from the clash between string and bytes. Applied this fix and ran
    the hook again, saw that the error was gone.
  • Ran unit tests.
Summary ID
Ensure that the execute method returns strings and not bytes.
b8e4ba4563a448db38ce518b78da3ebd71691f7b
Description From Last Updated

While we're here, let's change this from "Executes and returns" to "Execute and return"

daviddavid

Might want to specify here that this only applies when text=True.

daviddavid

We should probably use the old behavior by default, where this would be defaulted to falsy (technically None, but False …

chipx86chipx86

Missing , optional.

chipx86chipx86

Missing , optional.

chipx86chipx86

Let's use a PendingRemovalInRBToolsWarning warning. It's less harsh, and avoids a timeline. The reason being that this method is needed …

chipx86chipx86
david
  1. 
      
  2. rbtools/hooks/common.py (Diff revision 1)
     
     
    Show all issues

    While we're here, let's change this from "Executes and returns" to "Execute and return"

  3. 
      
maubin
david
  1. Ship It!
  2. 
      
maubin
david
  1. 
      
  2. rbtools/hooks/common.py (Diff revision 3)
     
     
     
    Show all issues

    Might want to specify here that this only applies when text=True.

  3. 
      
maubin
david
  1. Ship It!
  2. 
      
chipx86
  1. This change looks fine, but we should probably deprecate this method in favor of the modern run_process() code. We'll still need to keep this method around for existing hook scripts for now, though.

    1. Sounds good, I'll deprecate it in this change.

  2. 
      
maubin
david
  1. Ship It!
  2. 
      
chipx86
  1. 
      
  2. rbtools/hooks/common.py (Diff revision 5)
     
     
    Show all issues

    We should probably use the old behavior by default, where this would be defaulted to falsy (technically None, but False should be fine). Popen() defaults to binary streams.

    1. Ah, I was confused on what the "old behavior" was. In Python 2 Popen() returns text, but since we've ditched Python 2 since RBTools 4 anyways, I should just be considering its behavior in Python 3. I'll fix that up, and also make it so that we strip the bytes output by default too, since this was also the old behavior.

  3. rbtools/hooks/common.py (Diff revision 5)
     
     
    Show all issues

    Missing , optional.

  4. rbtools/hooks/common.py (Diff revision 5)
     
     
    Show all issues

    Missing , optional.

  5. rbtools/hooks/common.py (Diff revision 5)
     
     
     
     
    Show all issues

    Let's use a PendingRemovalInRBToolsWarning warning. It's less harsh, and avoids a timeline. The reason being that this method is needed for existing hooks being used in the wild today, and until we have a full new hook implementation and can get people onto it, we should avoid telling them to make changes to their hooks.

    Hooks also have to be careful with stdout/stderr, as that can signal things to some SCM command line tools or servers.

  6. 
      
maubin
chipx86
  1. Ship It!
  2. 
      
maubin
Review request changed
Status:
Completed
Change Summary:
Pushed to release-5.x (3a6fddf)