git-p4: Fix file renames, deletes and files with whiltespace in name

Review Request #12258 — Created April 25, 2022 and submitted — Latest diff uploaded

Information

RBTools
master

Reviewers

Previously, the filename was gathered from the diff --git ... line,
but this is problematic in the case where the filename has spaces. The
parser was simply splitting on space.

The solution is to get the filename from the --- ... line, resetting
when we see the diff ... line. This is simpler as we can assume
anything following the --- is the filename. We also ensure that we
correctly identify file renames by reading the old
filename and new filename from these lines.

Further we include the headers for renames as generated by the
perforce rbtools client (Moved from: and Moved to:). There's one special
case here where previously a 100% rename (with no diff) would
silently fail to send anything to the server. In this case, we now send
something the server recognises as a file rename operation, special
casing like we do in the perforce.py client.

In a git-p4 repo with files containing spaces:
1. edit one file
2. rename one file
3. create a new file with space in it
4. delete a file with space in it

I compared the generated diffs from a pure perforce repo and a git-p4 repo of the same locatoin and ensured that we generate the same for git-p4 as we do for perforce on rbt post.

Here are the outputs from adding, renaming (without changes), renaming (with changes) and deleting files:

Image
Image
Image

The one remaining area that's differnt is that git-p4 diff generation does not handle binary files being added/modified/deleted/etc. but this is enough for one patch.

unit tests

I used a docker conatiner to run them. Here's the dockerfile:

FROM ubi8

ENV YUMARGSBASE="--disablerepo=* --enablerepo=ubi-8-*"
ENV YUMARGS="${YUMARGSBASE} --enablerepo=epel*"

RUN  dnf ${YUMARGSBASE} -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm &&\
  yum ${YUMARGS} update -y && \
  yum ${YUMARGS} install -y git \
                            gcc-c++ \
                            make \
                            python39 \
                            diffutils \
                            apr-devel \
                            apr-util-devel \
                            unzip \
                            zlib-devel \
                            lz4-devel \
                            python39-devel && \
  yum ${YUMARGS} clean all

ARG SUBVERSION=1.14.2
ARG SQLITE=3081101

RUN cd $HOME &&  \
    curl -LO https://dlcdn.apache.org/subversion/subversion-${SUBVERSION}.tar.gz && \
    curl -LO https://www.sqlite.org/2015/sqlite-amalgamation-${SQLITE}.zip && \
    tar zxvf subversion-${SUBVERSION}.tar.gz && \
    rm subversion-${SUBVERSION}.tar.gz && \
    unzip sqlite-amalgamation-${SQLITE}.zip && \
    rm sqlite-amalgamation-${SQLITE}.zip && \
    mv sqlite-amalgamation-${SQLITE} subversion-${SUBVERSION}/sqlite-amalgamation && \
    cd subversion-${SUBVERSION} && \
    ./configure --with-utf8proc=internal && \
    make -j 8 && \
    make install && \
    cd $HOME && \
    rm -rf subversion-${SUBVERSION}

The result was some failures - all for svn failing, though i don't know why this would be:

============================================================================ short test summary info =============================================================================
FAILED rbtools/clients/tests/test_cvs.py::CVSClientTests::test_get_repository_info_with_found - AssertionError: None is not an instance of <class 'rbtools.clients.RepositoryIn...
FAILED rbtools/clients/tests/test_scanning.py::ScanningTests::test_scanning_nested_repos_1 - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/...
FAILED rbtools/clients/tests/test_scanning.py::ScanningTests::test_scanning_nested_repos_2 - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_diff_exclude - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/clients/tests/te...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_diff_exclude_in_subdir - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/client...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_diff_exclude_root_pattern_in_subdir - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/r...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_diff_non_unicode_characters - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/c...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_diff_non_unicode_filename_repository_url - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbto...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_get_commit_message_committed_revision - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_get_commit_message_committed_revisions - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtool...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_get_commit_message_working_copy - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtoo...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_history_scheduled_with_commit_nominal - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_history_scheduled_with_commit_special_case_exclude - Exception: Failed to execute command: ['svn', 'co', 'file://...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_history_scheduled_with_commit_special_case_non_local_mods - Exception: Failed to execute command: ['svn', 'co', '...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_invalid_spec - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbto...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_no_args - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/c...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_non_unicode_log - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/r...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_one_arg_two_revisions - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbt...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_one_revision - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbto...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_one_revision_changelist - Exception: Failed to execute command: ['svn', 'co', 'file:///root/r...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_one_revision_nonexistant_changelist - Exception: Failed to execute command: ['svn', 'co', 'fi...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_one_revision_url - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_two_arguments - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbt...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_parse_revision_spec_two_revisions_url - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_rename_diff_mangling_bug_4546 - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_same_diff_multiple_methods - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/cl...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_show_copies_as_adds_disabled - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/...
ERROR rbtools/clients/tests/test_svn.py::SVNClientTests::test_show_copies_as_adds_enabled - Exception: Failed to execute command: ['svn', 'co', 'file:///root/rbtools/rbtools/c...
================================================== 3 failed, 221 passed, 42 skipped, 2 warnings, 25 errors in 78.68s (0:01:18) ===================================================

The skiipped tess are all bzr and clearcase (and some mercurial for some reason):

[root@1bda7694ea8b rbtools]# python3 tests/runtests.py 
============================================================================== test session starts ===============================================================================
platform linux -- Python 3.9.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /root/rbtools, configfile: setup.cfg, testpaths: rbtools
plugins: kgb-7.0, env-0.6.2
collected 291 items                                                                                                                                                              

rbtools/api/tests/test_capabilities.py ...                                                                                                                                 [  1%]
rbtools/api/tests/test_factory.py ......                                                                                                                                   [  3%]
rbtools/api/tests/test_http_request.py ........                                                                                                                            [  5%]
rbtools/api/tests/test_resource.py ....................                                                                                                                    [ 12%]
rbtools/clients/tests/test_bzr.py ssssssssssssssssssss                                                                                                                     [ 19%]
rbtools/clients/tests/test_clearcase.py sssssssssssssss                                                                                                                    [ 24%]
rbtools/clients/tests/test_cvs.py F.                                                                                                                                       [ 25%]
rbtools/clients/tests/test_git.py ...............................................                                                                                          [ 41%]
rbtools/clients/tests/test_mercurial.py .....................................sssssss                                                                                       [ 56%]
rbtools/clients/tests/test_p4.py .....................                                                                                                                     [ 63%]
rbtools/clients/tests/test_scanning.py FF                                                                                                                                  [ 64%]
rbtools/clients/tests/test_svn.py ....EEEEEEEEEEEEEEEEEEEEEEEEE                                                                                                            [ 74%]
rbtools/commands/tests/test_alias.py .......                                                                                                                               [ 76%]
rbtools/commands/tests/test_main.py .............                                                                                                                          [ 81%]
rbtools/commands/tests/test_post.py ..............                                                                                                                         [ 86%]
rbtools/commands/tests/test_setup_repo.py .......                                                                                                                          [ 88%]
rbtools/utils/tests/test_aliases.py .............                                                                                                                          [ 93%]
rbtools/utils/tests/test_checks.py ...                                                                                                                                     [ 94%]
rbtools/utils/tests/test_console.py .......                                                                                                                                [ 96%]
rbtools/utils/tests/test_filesystem.py .....                                                                                                                               [ 98%]
rbtools/utils/tests/test_process.py .                                                                                                                                      [ 98%]
rbtools/utils/tests/test_repository.py ....                                                                                                                                [100%]

Commits

Files