[feat] Allow to rewrite url before download using netrc and macdef
This change is to rewrite URL of file to download through buildout.download.Download using macdef definition in netrc file. The rewrite is based on regular expression, captures are substituted after optionally urlencoding them.
This is how macdef definition is used to rewrite URL:
.netrc:
machine HOSTNAME
login foo
password bar
macdef buildout:HOSTNAME
REGEX_STRING
TEMPLATE_1 HEADER1=VALUE1 HEADER2=VALUE2 ...
ANOTHER_REGEX
TEMPLATE_2 HEADER1=VALUE1 HEADER2=VALUE2 ...
...
macdef ...
HEADER1=VALUE1
string are optional, they are used to set the header. Similar to
the command curl --header "HEADER1: VALUE1" ...
. Headers can
be repeated as it's needed.
REGEX_STRING
is used to match the path and query (if present) of the url we are trying to download.
for example: the regex /(.*)/-/raw/([\w\-]+)/(.*)
for url
'https://lab.nexedi.com/namespace/project/-/raw/master/README.md'
/(.*)/-/raw/([\w\-]+)/(.*)
{0} {1} {2} {3}
TEMPLATE
is the new full url with scheme and authority (netloc). All captures are used to format
the template and the headers. It's possible to encode string while formatting, for now only quote
function from urllib.parse
is supported. Captured groups from REGEX_STRING start at {1}; {0} is the base url.
The TEMPLATE
line is split using shell-like syntax (with shlex.split
). The indentation in netrc
file is optional, only there for readability.
Example of template and headers:
{0}/api/v4/projects/{1.quote}/repository/files/{3}/raw?ref={2} Authentication={2}
With regex and template above, the url 'https://lab.nexedi.com/namespace/project/-/raw/master/README.md' is rewritten to: https://lab.nexedi.com/api/v4/projects/namespace%2Fproject/repository/files/README.md/raw?ref=master
{0} is the base URL (https://lab.nexedi.com)
{1} match 'namespace/project'
{2} match 'master' and
{3} match 'README.md'
namespace/project
is changed to namespace%2Fproject
since it's encoded with quote
.
Some uses cases
machine lab.nexedi.com
login ignored
password <ACCESS_TOKEN>
macdef buildout:lab.nexedi.com
/(.*)/-/raw/([\w\-]+)/(.*)
{0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>
or
macdef buildout:lab.nexedi.com
/(.*)/raw/([\w\-]+)/(.*)
{0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} Authorization="Bearer <OAUTH-TOKEN>"
/(.*)/-/raw/([\w\-]+)/(.*)
{0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>