Commit bd331b1f authored by Łukasz Nowak's avatar Łukasz Nowak

Reimplement tests.

Use simple download recipe instead of relying on egg downloading.
parent c44ab168
......@@ -45,6 +45,11 @@ Now let's prepare networkcache server:
>>> nc_data = tmpdir('nc_data')
>>> nc_url = start_nc(nc_data)
And remote server:
>>> remote_server_data = tmpdir('remote_server_data')
>>> remote_server_url = start_server(remote_server_data)
Networkcache in buildout is optional, and during normal run no information
is shown:
......@@ -55,19 +60,57 @@ is shown:
... ''')
>>> print system(buildout)
But whenever networkcache configuration is enabled, it is possible to use it:
For some tests simple download recipe will be needed:
>>> mkdir(sample_buildout, 'download')
>>> write(sample_buildout, 'download', 'download.py',
... """
... import logging, os, zc.buildout
...
... class Download:
...
... def __init__(self, buildout, name, options):
... self.name, self.options, self.buildout = name, options, buildout
...
... def install(self):
... download = zc.buildout.download.Download(
... self.buildout['buildout'])
... path, is_temp = download(self.options['url'],
... md5sum=self.options.get('md5sum'))
... logging.getLogger(self.name).info(
... 'Downloaded %s', self.options['url'])
... return []
...
... update = install
... """)
>>> write(sample_buildout, 'download', 'setup.py',
... """
... from setuptools import setup
...
... setup(
... name = "download",
... entry_points = {'zc.buildout': ['default = download:Download']},
... )
... """)
>>> write(sample_buildout, 'download', 'README.txt', " ")
Let's create file on remote server:
>>> write(remote_server_data, 'hello.txt', 'Hello my friend')
It will be downloaded and stored into networkcache:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -79,39 +122,30 @@ But whenever networkcache configuration is enabled, it is possible to use it:
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: 404 : File not found
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demoneeded*'))]
[None]
>>> remove('.installed.cfg')
Develop: '/sample-buildout/download'
Installing download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
And run it again, data will be downloaded from networkcache:
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Networkcache download cache: 'http://localhost:29630/shacache', directory 'http://localhost:29630/shadir'
Networkcache upload cache: 'http://localhost:29630/shacache', directory 'http://localhost:29630/shadir'
Develop: '/tmp/tmpOkGTHmbuildoutSetUp/_TEST_/sample-buildout/download'
Updating download.
Downloading http://localhost:25390/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost:25390/hello.txt
Now lets clean up buildout directory:
Lets clean networkcache:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demoneeded*'))]
[None]
>>> remove('.installed.cfg')
>>> rmdir(nc_data)
>>> mkdir(nc_data)
As one can see, in case if upload is not working, there is no problem with running buildout:
......@@ -120,12 +154,12 @@ As one can see, in case if upload is not working, there is no problem with runni
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demo
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -137,24 +171,19 @@ As one can see, in case if upload is not working, there is no problem with runni
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://127.0.0.1:1/sshacache', directory 'http://127.0.0.1:1/sshadir'
Installing eggs.
Getting distribution for 'demo'.
Downloading demo-0.4c1-py2.7.egg from network cache.
Failed to download from network cache demo-0.4c1-py2.7.egg: 404 : File not found
Uploading http://localhost/demo-0.4c1-py2.7.egg into network cache.
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
Uploading http://localhost/hello.txt into network cache.
Fail to upload file. [Errno 111] Connection refused
Got demo 0.4c1.
Getting distribution for 'demoneeded'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
download: Downloaded http://localhost/hello.txt
Now lets clean up buildout directory:
Lets clean networkcache:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None, None]
>>> remove('.installed.cfg')
>>> rmdir(nc_data)
>>> mkdir(nc_data)
Upload is optional, and then no upload will be tried:
......@@ -163,12 +192,12 @@ Upload is optional, and then no upload will be tried:
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demo
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -178,22 +207,12 @@ Upload is optional, and then no upload will be tried:
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demo'.
Downloading demo-0.4c1-py2.7.egg from network cache.
Failed to download from network cache demo-0.4c1-py2.7.egg: 404 : File not found
Got demo 0.4c1.
Getting distribution for 'demoneeded'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None, None]
>>> remove('.installed.cfg')
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
download: Downloaded http://localhost/hello.txt
In case if download cache is failing, original resources are used:
......@@ -205,12 +224,12 @@ message not perfect error is shown like <urlopen...Connection Refused>)
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demo
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = http://127.0.0.1:1/shacache
......@@ -222,32 +241,20 @@ message not perfect error is shown like <urlopen...Connection Refused>)
Networkcache enabled.
Networkcache download cache: 'http://127.0.0.1:1/shacache', directory 'http://127.0.0.1:1/shadir'
Networkcache upload cache: 'http://127.0.0.1:1/sshacache', directory 'http://127.0.0.1:1/sshadir'
Installing eggs.
Getting distribution for 'demo'.
Downloading demo-0.4c1-py2.7.egg from network cache.
Failed to download from network cache demo-0.4c1-py2.7.egg: <urlopen error [Errno 111] Connection refused>
Uploading http://localhost/demo-0.4c1-py2.7.egg into network cache.
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: <urlopen error [Errno 111] Connection refused>
Uploading http://localhost/hello.txt into network cache.
Fail to upload file. [Errno 111] Connection refused
Got demo 0.4c1.
Getting distribution for 'demoneeded'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: <urlopen error [Errno 111] Connection refused>
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Fail to upload file. [Errno 111] Connection refused
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None, None]
>>> remove('.installed.cfg')
download: Downloaded http://localhost/hello.txt
Creating the signature files:
Lets create the signature files:
>>> signature_data = tmpdir('signature_data')
>>> keydir = tmpdir('keydir')
>>> import os
>>> public_certificate_path = os.path.join(keydir, 'public.pem')
>>> private_key_path = os.path.join(keydir, 'private.pem')
>>> ignored = system(os.path.join(sample_buildout, 'bin', 'generate-signature-key') + ' --signature-certificate-file=' + public_certificate_path + ' --signature-private-key-file=' + private_key_path)
......@@ -255,59 +262,51 @@ Creating the signature files:
- private.pem
- public.pem
First put not signed file in networkcache:
Enable signature, so it should not download any content which is not trusted :
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(private_key_path)s
... signature-certificate-file = %(public_certificate_path)s
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: Could not find a trustable entry.
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: 404 : Not Found
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Now enable signature, so it should not download any content which is not trusted :
Once it has a trusted content available into the server it should be abled to download it:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
......@@ -321,20 +320,29 @@ Once it has a trusted content available into the server it should be abled to do
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: Could not find a trustable entry.
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
Once it has a trusted content available into the server it should be abled to download it:
Now lets clean up buildout directory:
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost:26283/shacache', directory 'http://localhost:26283/shadir'
Networkcache upload cache: 'http://localhost:26283/shacache', directory 'http://localhost:26283/shadir'
Develop: '/tmp/tmpO55kN1buildoutSetUp/_TEST_/sample-buildout/download'
Updating download.
Downloading http://localhost:22168/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost:22168/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Now lets create new signature files:
>>> new_keydir = tmpdir('newkeydir')
>>> new_public_certificate_path = os.path.join(new_keydir, 'public2.pem')
>>> new_private_key_path = os.path.join(new_keydir, 'private2.pem')
......@@ -344,148 +352,147 @@ Now lets create new signature files:
- public2.pem
There is a new certificate file, so it should not download the old content:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(new_private_key_path)s
... signature-certificate-file = %(new_public_certificate_path)s
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: Could not find a trustable entry.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: Could not find a trustable entry.
download: Downloaded http://localhost/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
If it can not find a trustable entry into server, it should upload the files using new signature:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(new_private_key_path)s
... signature-certificate-file = %(new_public_certificate_path)s
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Failed to download from network cache demoneeded-1.2c1.zip: Could not find a trustable entry.
Uploading http://localhost/demoneeded-1.2c1.zip into network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
Failed to download from network cache hello.txt: Could not find a trustable entry.
Uploading http://localhost/hello.txt into network cache.
download: Downloaded http://localhost/hello.txt
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Buildout can download the content using the old certificatei, because it still available in the cache server:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... signature-private-key-file = %(private_key_path)s
... signature-certificate-file = %(public_certificate_path)s
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Now lets clean up buildout directory:
>>> import os, glob
>>> [remove(q) for q in glob.glob(os.path.join(sample_buildout, 'eggs', 'demo*'))]
[None]
>>> remove('.installed.cfg')
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost/hello.txt
Buildout can download the content using the new certificate, because it still available in the cache server:
>>> write(sample_buildout, 'buildout.cfg',
... '''
... [buildout]
...
... networkcache-section = networkcache
... find-links = %(link_server)s
... parts = eggs
... develop = download
... parts = download
...
... [eggs]
... recipe = zc.recipe.egg:eggs
... eggs = demoneeded ==1.2c1
... [download]
... recipe = download
... url = %(remote_server_url)shello.txt
...
... [networkcache]
... download-cache-url = %(nc_url)sshacache
... download-dir-url = %(nc_url)sshadir
... signature-certificate-file = %(new_public_certificate_path)s
... signature-private-key-file = %(private_key_path)s
... signature-certificate-file = %(public_certificate_path)s
... upload-cache-url = %(nc_url)sshacache
... upload-dir-url = %(nc_url)sshadir
... ''' % globals())
>>> print system(buildout)
Networkcache enabled.
Networkcache download cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Networkcache upload cache: '', directory ''
Installing eggs.
Getting distribution for 'demoneeded==1.2c1'.
Downloading demoneeded-1.2c1.zip from network cache.
Got demoneeded 1.2c1.
<BLANKLINE>
Networkcache upload cache: 'http://localhost/shacache', directory 'http://localhost/shadir'
Develop: '/sample-buildout/download'
Updating download.
Downloading http://localhost/hello.txt
Downloading hello.txt from network cache.
download: Downloaded http://localhost/hello.txt
Todo:
* downloading eggs
* downloading extensions?
* showing that wrong md5sum will fail to upload
###############
# PARSING URL #
###############
Check if the parsing url method works correctly, in low-level:
>>> get_filename_from_url("http://localhost/lib/patch/?id=700c7d5382b01f94e7141")
'id=700c7d5382b01f94e7141'
>>> get_filename_from_url("http://localhost/tarball-5.1.tar.gz?id=700")
......@@ -495,6 +502,7 @@ Check if the parsing url method works correctly, in low-level:
Check if the networkcache upload method is using the correct method to
parse the original url and get the file name:
>>> tmp_dir = tmpdir('tmp_dir')
>>> write(tmp_dir, 'tmp_file', 'Content of temp file.')
>>> get_filename_from_upload_network_cached(
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment