Skip to content

S
slapos.buildout
Commit 57345f87 authored by Łukasz Nowak's avatar Łukasz Nowak
Browse files
Options

Reimplement tests. 

Use simple download recipe instead of relying on egg downloading.
parent e8187176
Hide whitespace changes
Inline Side-by-side
Showing with 211 additions and 203 deletions
src/zc/buildout/networkcache.txt
View file @ 57345f87
......@@ -45,6 +45,11 @@ Now let's prepare networkcache server:
>>> nc_data = tmpdir('nc_data')
>>> nc_url = start_nc(nc_data)
And remote server:
>>> remote_server_data = tmpdir('remote_server_data')
>>> remote_server_url = start_server(remote_server_data)
Networkcache in buildout is optional, and during normal run no information
is shown:
......@@ -55,19 +60,57 @@ is shown:
... ''')
>>> print system(buildout)
But whenever networkcache configuration is enabled, it is possible to use it:
For some tests simple download recipe will be needed:
>>> mkdir(sample_buildout, 'download')
>>> write(sample_buildout, 'download', 'download.py',
... """
... import logging, os, zc.buildout
...
... class Download:
...
... def __init__(self, buildout, name, options):
... self.name, self.options, self.buildout = name, options, buildout
...
... def install(self):
... download = zc.buildout.download.Download(
... self.buildout['buildout'])
... path, is_temp = download(self.options['url'],
... md5sum=self.options.get('md5sum'))
... logging.getLogger(self.name).info(
... 'Downloaded %s', self.options['url'])
... return []
...
... update = install
... """)
>>> write(sample_buildout, 'download', 'setup.py',
... """
... from setuptools import setup
...
... setup(
... name = "download",
... entry_points = {'zc.buildout': ['default = download:Download']},
... )
... """)
>>> write(sample_buildout, 'download', 'README.txt', " ")
Let's create file on remote server:
>>> write(remote_server_data, 'hello.txt', 'Hello my friend')
It will be downloaded and stored into networkcache:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -79,39 +122,30 @@ But whenever networkcache configuration is enabled, it is possible to use it:
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: 404 : File not found
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demoneeded*'))]
[None]
>>> remove('.installed.cfg')
Develop: '/sample-buildout/download'
Installing download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
And run it again, data will be downloaded from networkcache:
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Networkcache download cache: 'http://localhost:29630/shacache', directory 'http://localhost:29630/shadir'
Networkcache upload cache: 'http://localhost:29630/shacache', directory 'http://localhost:29630/shadir'
Develop: '/tmp/tmpOkGTHmbuildoutSetUp/_TEST_/sample-buildout/download'
Updating download.
Downloading http://localhost:25390/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost:25390/hello.txt
Now lets clean up buildout directory:
Lets clean networkcache:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demoneeded*'))]
[None]
>>> remove('.installed.cfg')
>>> rmdir(nc_data)
>>> mkdir(nc_data)
As one can see, in case if upload is not working, there is no problem with running buildout:
......@@ -120,12 +154,12 @@ As one can see, in case if upload is not working, there is no problem with runni
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demo
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -137,24 +171,19 @@ As one can see, in case if upload is not working, there is no problem with runni
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://127.0.0.1:1/sshacache', directory 'http://127.0.0.1:1/sshadir'
Installing eggs.
Getting distribution for 'demo'.
Downloading demo-0.4c1-py2.7.egg from network cache.
Failed to download from network cache demo-0.4c1-py2.7.egg: 404 : File not found
Uploading http://localhost/demo-0.4c1-py2.7.egg into network cache.
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
Uploading http://localhost/hello.txt into network cache.
Fail to upload file. [Errno 111] Connection refused
Got demo 0.4c1.
Getting distribution for 'demoneeded'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
download: Downloaded http://localhost/hello.txt
Now lets clean up buildout directory:
Lets clean networkcache:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None, None]
>>> remove('.installed.cfg')
>>> rmdir(nc_data)
>>> mkdir(nc_data)
Upload is optional, and then no upload will be tried:
......@@ -163,12 +192,12 @@ Upload is optional, and then no upload will be tried:
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demo
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -178,22 +207,12 @@ Upload is optional, and then no upload will be tried:
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demo'.
Downloading demo-0.4c1-py2.7.egg from network cache.
Failed to download from network cache demo-0.4c1-py2.7.egg: 404 : File not found
Got demo 0.4c1.
Getting distribution for 'demoneeded'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None, None]
>>> remove('.installed.cfg')
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
download: Downloaded http://localhost/hello.txt
In case if download cache is failing, original resources are used:
......@@ -205,12 +224,12 @@ message not perfect error is shown like <urlopen...Connection Refused>)
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demo
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = http://127.0.0.1:1/shacache
......@@ -222,32 +241,20 @@ message not perfect error is shown like <urlopen...Connection Refused>)
Networkcache enabled.
Networkcache download cache: 'http://127.0.0.1:1/shacache', directory 'http://127.0.0.1:1/shadir'
Networkcache upload cache: 'http://127.0.0.1:1/sshacache', directory 'http://127.0.0.1:1/sshadir'
Installing eggs.
Getting distribution for 'demo'.
Downloading demo-0.4c1-py2.7.egg from network cache.
Failed to download from network cache demo-0.4c1-py2.7.egg: <urlopen error [Errno 111] Connection refused>
Uploading http://localhost/demo-0.4c1-py2.7.egg into network cache.
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: <urlopen error [Errno 111] Connection refused>
Uploading http://localhost/hello.txt into network cache.
Fail to upload file. [Errno 111] Connection refused
Got demo 0.4c1.
Getting distribution for 'demoneeded'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: <urlopen error [Errno 111] Connection refused>
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Fail to upload file. [Errno 111] Connection refused
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None, None]
>>> remove('.installed.cfg')
download: Downloaded http://localhost/hello.txt
Creating the signature files:
Lets create the signature files:
>>> signature_data = tmpdir('signature_data')
>>> keydir = tmpdir('keydir')
>>> import os
>>> public_certificate_path = os.path.join(keydir, 'public.pem')
>>> private_key_path = os.path.join(keydir, 'private.pem')
>>> ignored = system(os.path.join(sample_buildout, 'bin', 'generate-signature-key') + ' --signature-certificate-file=' + public_certificate_path + ' --signature-private-key-file=' + private_key_path)
......@@ -255,59 +262,51 @@ Creating the signature files:
- private.pem
- public.pem
First put not signed file in networkcache:
Enable signature, so it should not download any content which is not trusted :
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(private_key_path)s
... signature-certificate-file = %(public_certificate_path)s
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: Could not find a trustable entry.
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Now enable signature, so it should not download any content which is not trusted :
Once it has a trusted content available into the server it should be abled to download it:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -321,20 +320,29 @@ Once it has a trusted content available into the server it should be abled to do
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: Could not find a trustable entry.
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
Once it has a trusted content available into the server it should be abled to download it:
Now lets clean up buildout directory:
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost:26283/shacache', directory 'http://localhost:26283/shadir'
Networkcache upload cache: 'http://localhost:26283/shacache', directory 'http://localhost:26283/shadir'
Develop: '/tmp/tmpO55kN1buildoutSetUp/_TEST_/sample-buildout/download'
Updating download.
Downloading http://localhost:22168/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost:22168/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Now lets create new signature files:
>>> new_keydir = tmpdir('newkeydir')
>>> new_public_certificate_path = os.path.join(new_keydir, 'public2.pem')
>>> new_private_key_path = os.path.join(new_keydir, 'private2.pem')
......@@ -344,148 +352,147 @@ Now lets create new signature files:
- public2.pem
There is a new certificate file, so it should not download the old content:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(new_private_key_path)s
... signature-certificate-file = %(new_public_certificate_path)s
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: Could not find a trustable entry.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: Could not find a trustable entry.
download: Downloaded http://localhost/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
If it can not find a trustable entry into server, it should upload the files using new signature:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(new_private_key_path)s
... signature-certificate-file = %(new_public_certificate_path)s
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: Could not find a trustable entry.
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: Could not find a trustable entry.
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Buildout can download the content using the old certificatei, because it still available in the cache server:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(private_key_path)s
... signature-certificate-file = %(public_certificate_path)s
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost/hello.txt
Buildout can download the content using the new certificate, because it still available in the cache server:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... signature-certificate-file = %(new_public_certificate_path)s
... signature-private-key-file = %(private_key_path)s
... signature-certificate-file = %(public_certificate_path)s
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost/hello.txt
Todo:
* downloading eggs
* downloading extensions?
* showing that wrong md5sum will fail to upload
###############
# PARSING URL #
###############
Check if the parsing url method works correctly, in low-level:
>>> get_filename_from_url("http://localhost/lib/patch/?id=700c7d5382b01f94e7141")
'id=700c7d5382b01f94e7141'
>>> get_filename_from_url("http://localhost/tarball-5.1.tar.gz?id=700")
......@@ -495,6 +502,7 @@ Check if the parsing url method works correctly, in low-level:
Check if the networkcache upload method is using the correct method to
parse the original url and get the file name:
>>> tmp_dir = tmpdir('tmp_dir')
>>> write(tmp_dir, 'tmp_file', 'Content of temp file.')
>>> get_filename_from_upload_network_cached(
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7