GitLab

This change is to rewrite URL of file to download through buildout.download.Download using macdef definition in netrc file. The rewrite is based on regular expression, captures are substituted after optionally urlencoding them.

This is how macdef definition is used to rewrite URL:

.netrc:

machine HOSTNAME
login foo
password bar

macdef buildout:HOSTNAME
  REGEX_STRING
    TEMPLATE_1 HEADER1=VALUE1 HEADER2=VALUE2 ...
  ANOTHER_REGEX
    TEMPLATE_2 HEADER1=VALUE1 HEADER2=VALUE2 ...
  ...

macdef ...

HEADER1=VALUE1 string are optional, they are used to set the header. Similar to the command curl --header "HEADER1: VALUE1" .... Headers can be repeated as it's needed.

REGEX_STRING is used to match the path and query (if present) of the url we are trying to download. for example: the regex /(.*)/-/raw/([\w\-]+)/(.*) for url 'https://lab.nexedi.com/namespace/project/-/raw/master/README.md'

                /(.*)/-/raw/([\w\-]+)/(.*)
      {0}         {1}          {2}     {3}

TEMPLATE is the new full url with scheme and authority (netloc). All captures are used to format the template and the headers. It's possible to encode string while formatting, for now only quote function from urllib.parse is supported. Captured groups from REGEX_STRING start at {1}; {0} is the base url.

The TEMPLATE line is split using shell-like syntax (with shlex.split). The indentation in netrc file is optional, only there for readability.

Example of template and headers:

{0}/api/v4/projects/{1.quote}/repository/files/{3}/raw?ref={2} Authentication={2}

With regex and template above, the url 'https://lab.nexedi.com/namespace/project/-/raw/master/README.md' is rewritten to: https://lab.nexedi.com/api/v4/projects/namespace%2Fproject/repository/files/README.md/raw?ref=master

{0} is the base URL (https://lab.nexedi.com)

{1} match 'namespace/project'

{2} match 'master' and

{3} match 'README.md'

namespace/project is changed to namespace%2Fproject since it's encoded with quote.

Some uses cases

machine lab.nexedi.com
login ignored
password <ACCESS_TOKEN>

macdef buildout:lab.nexedi.com
  /(.*)/-/raw/([\w\-]+)/(.*)
    {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>

or

macdef buildout:lab.nexedi.com
  /(.*)/raw/([\w\-]+)/(.*)
    {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} Authorization="Bearer <OAUTH-TOKEN>"
  /(.*)/-/raw/([\w\-]+)/(.*)
    {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>

This MR is a workaround of how to access private to raw files on gitlab dicussed here: https://gitlab.com/gitlab-org/gitlab/-/issues/19189