• Rémy Coutable's avatar
    Merge branch 'robots-snippets' into 'master' · ebc4299d
    Rémy Coutable authored
    Update robots.txt disallow to allow users/groups starting with "s"
    
    ## What does this MR do?
    
    Update robots.txt "Disallow: /s" line to "Disallow: /s/"
    
    ## Why was this MR needed?
    
    Currently any Project that is under a username or group name that starts with "s" cannot be indexed because of this disallow line.
    
    For example on https://gitlab.com looking at the projects with the most stars there is this project: https://gitlab.com/somerobots/Trident
    
    If you do a Google search for "site:gitlab.com Trident", the project comes up but with "A description for this result is not available because of this site's robots.txt"
    
    Adding the trailing slash will still disallow all snippet short urls, but start allowing projects where the URI begins with "s"
    
    See merge request !6481
    ebc4299d
To find the state of this project's repository at the time of any of these versions, check out the tags.
CHANGELOG 123 KB