• Julien Muchembled's avatar
    fixup! [feat] allow to rewrite url before download using netrc and macdef · 389ea203
    Julien Muchembled authored
    (new commit message)
    
    This adds a generic mechanism in the Download API to rewrite the URL
    to be downloaded, possibly with extra headers. Substitued groups from
    the matching regular expression can be optionally quoted.
    
    .netrc:
    
    macdef buildout:HOSTNAME
      RULE_1_REGEX
        RULE_1_NEW_URL HEADER1=VALUE1 HEADER2=VALUE2 ...
      RULE_2_REGEX
        RULE_2_NEW_URL ...
      ...
    
    macdef buildout:OTHER_HOSTNAME
      ...
    
    A rewriting rule is defined by a pair of lines with optional indentation
    (only there for readability).
    
    The first line of a rule is a regex that matches fully against
    the path?query part of the URL.
    
    If the second line is empty, the request isn't changed.
    Else parts are parsed using shell-like syntax:
    - the first one must produce the full URL to download
    - the next ones are optional headers to send
    - each part is subject to regex substitution using the
      Python Format Specification Mini-Language:
      captured grouped from the first line are numbered starting from 1,
      {0} is the base URL (scheme://location) and
      the optional `quote` attribute returns the urlencoded value
    
    A use case is to work around
      https://gitlab.com/gitlab-org/gitlab/-/issues/19189
    for example with the following .netrc:
    
        macdef buildout:lab.nexedi.com
          /(.+)/-/raw/([^/]+)/(.+)
            {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>
    389ea203
download.py 13.7 KB