Skip to content

slapos
slapos
Commit aca81eb3 authored by Łukasz Nowak's avatar Łukasz Nowak
Browse files
Options

kvm: Implement external-disk 

Support absolute and relative paths. Paths starting with "rbd:" are special,
and they point to the rbd endpoint, served by Ceph.

cache parameter is optional, in case if present, will be used as-is, even with
empty cache.

external-disk-number, external-disk-size and external-disk-format has been
removed from the schema to become hidden, nevertheless it's expected to be
fully supported, as long tests are kept.
parent 0ba172a0
Pipeline #22228 failed with stage
    in 0 seconds
    Hide whitespace changes
    Inline Side-by-side
    Showing with 412 additions and 82 deletions
    software/kvm/README.rst
    View file @ aca81eb3
    ......@@ -72,12 +72,6 @@ KVM instance parameters:
    - hard-drive-url-check-certificate (default: True)
    if virtual-hard-drive-url use self-signed https, then specify if https certificate should be verified or not
    - external-disk-number (default: 0)
    Number of additional disk to attach to this VM. Need slapformat to be configured for this feature.
    - external-disk-size (default: 20)
    - external-disk-format (default: qcow2)
    additional disk format. should be in this list: ['qcow2', 'raw', 'vdi', 'vmdk', 'cloop', 'qed']
    - enable-http-server (default: False)
    Configure server that will help to get some files into the vm from http
    require use-nat = True
    ......@@ -126,3 +120,79 @@ Updating boot-image-url-select
    * update the ``boot-image-url-select`` in:
    * ``instance-kvm-input-schema.json``
    * ``instance-kvm-cluster-input-schema.json``
    Migration to modern external-disk parameter
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Despite ``external-disk-number``, ``external-disk-size`` and
    ``external-disk-format`` are supported fully until unknown moment in the
    future, it's advised to migrate to external-disk parameter as soon as possible,
    as slapos.core ``slapos.cfg`` ``instance_storage_home`` can become obsoleted
    and removed in future versions.
    **Note**: Due to how technically ``instance_storage_home`` is implemented, such
    migration requires full access to the Compute Node hosting given KVM instance.
    Let's imagine that there is a kvm instance which was requested with parameters::
    {
    "external-disk-number": 2,
    "external-disk-size": 10
    }
    After locating the partition on proper Compute Node, the node administrator
    has to find the kvm processing running there with::
    slapos node status slappartNN: | grep kvm-
    The interesting part is the ``PID``, which can be used to find which disk paths
    are configured for the running KVM process with::
    ps axu | grep PID | grep --color DATA
    It will be possible to find two entries pointing to ``DATA`` directory in the
    partition::
    -drive file=/srv/slapgrid/slappartNN/DATA/dataX/kvm_virtual_disk.qcow2,if=virtio
    -drive file=/srv/slapgrid/slappartNN/DATA/dataY/kvm_virtual_disk.qcow2,if=virtio
    **Attention**: Order of the disks is important.
    The administrator shall provide absolute path to both for both disks::
    readlink -f /srv/slapgrid/slappartNN/DATA/dataX/kvm_virtual_disk.qcow2 --> /<instance_storage_home>/dataX/slappartNN/kvm_virtual_disk.qcow2
    readlink -f /srv/slapgrid/slappartNN/DATA/dataY/kvm_virtual_disk.qcow2 --> /<instance_storage_home>/dataY/slappartNN/kvm_virtual_disk.qcow2
    And now it will be safe to use the paths in ``external-disk`` parameter::
    {
    "external-disk": {
    "first": {
    "path": "/<instance_storage_home>/dataX/slappartNN/kvm_virtual_disk.qcow2",
    "index": 1
    },
    "second": {
    "path": "/<instance_storage_home>/dataY/slappartNN/kvm_virtual_disk.qcow2",
    "index": 2
    }
    }
    }
    Of course ``external-disk-number`` and ``external-disk-size`` HAVE TO be removed
    from instance parameters before continuing.
    For now such configuration will lead to no starting kvm process, so after
    parameters are updated in SlapOS Master **and** are processed on the Compute
    Node The administrator shall release the images from automatic detection by
    removing files:
    * ``etc/.data-disk-amount``
    * ``etc/.data-disk-ids``
    from the partition (typically ``/srv/slapgrid/slappartNN/`` directory).
    The failure observed to confirm the situation can be found in
    ``.slappartNN_kvm-HASH.log`` with presence of message like::
    qemu-system-x86_64: -drive file=/<instance_storage_home>/dataX/slappartNN/kvm_virtual_disk.qcow2,if=virtio,cache=writeback: Failed to get "write" lock
    Is another process using the image [/<instance_storage_home>/dataX/slappartNN/kvm_virtual_disk.qcow2]?
    software/kvm/buildout.hash.cfg
    View file @ aca81eb3
    ......@@ -19,11 +19,11 @@ md5sum = 7e90da1f6dac4233e1aa3248f48e357c
    [template-kvm]
    filename = instance-kvm.cfg.jinja2
    md5sum = 4a3f999a8d1705c0cabf58d563953d4d
    md5sum = 44dc93281f2fffe64d014754fae1b38a
    [template-kvm-cluster]
    filename = instance-kvm-cluster.cfg.jinja2.in
    md5sum = 406bf455afe1c71cb0e8cc04dd877f1d
    md5sum = c00608f94ff6cfd8d37a280267c7227f
    [template-kvm-resilient]
    filename = instance-kvm-resilient.cfg.jinja2
    ......@@ -55,7 +55,7 @@ md5sum = a8cf453d20f01c707f02c4b4014580d8
    [template-kvm-run]
    filename = template/template-kvm-run.in
    md5sum = 6c100eec00de5e53f64b075dd69a9865
    md5sum = 2b025e5828901a92d2ed25d6ee9f7412
    [template-kvm-controller]
    filename = template/kvm-controller-run.in
    ......
    software/kvm/instance-kvm-cluster-input-schema.json
    View file @ aca81eb3
    ......@@ -392,34 +392,6 @@
    "type": "boolean",
    "default": true
    },
    "external-disk-number": {
    "title": "Number of additional disk to create for virtual machine",
    "description": "Specify the number of additional disk to create for virtual machine in data folder of SlapOS Node. Requires instance_storage_home to be configured on SlapOS Node.",
    "type": "integer",
    "minimum": 0,
    "default": 0
    },
    "external-disk-size": {
    "title": "Size of additional disk to create for virtual machine, in Gigabytes",
    "description": "Specify the size of additional disk to create for virtual machine in data folder of SlapOS Node. Requires instance_storage_home to be configured on SlapOS Node.",
    "type": "integer",
    "minimum": 5,
    "default": 20
    },
    "external-disk-format": {
    "title": "Type of external disk drive to create by QEMU.",
    "description": "Type of QEMU disk drive, to create.",
    "type": "string",
    "default": "qcow2",
    "enum": [
    "qcow2",
    "raw",
    "vdi",
    "vmdk",
    "cloop",
    "qed"
    ]
    },
    "wipe-disk-ondestroy": {
    "title": "Wipe disks when destroy the VM",
    "description": "Say if disks should be wiped by writing new data over every single bit before delete them. This option is used to securely delete VM disks",
    ......@@ -545,6 +517,58 @@
    "description": "List of whitelisted domain names to be accessed from the VM. They will be resolved to IPs depending on where the VM end up. IPs can be used too.",
    "type": "string",
    "textarea": true
    },
    "external-disk": {
    "title": "External disks",
    "description": "Allows to set paths and types of manually provided external disks. Use index to order disks. Changing this parameter leads to kvm process restart.",
    "type": "object",
    "default": {},
    "patternProperties": {
    ".*": {
    "properties": {
    "path": {
    "title": "Path of the provided image",
    "description": "Absolute or relative path (to the partition root path) of the provided image. Image has to be provided, with read and write permissions of the partition user. Please note, that if the name starts with 'rbd:' it's considered special and won't be used as relative path.",
    "type": "string"
    },
    "format": {
    "title": "Format of a disk image",
    "description": "Format of a disk image. Note: rbd type is experimental.",
    "type": "string",
    "default": "autodetect",
    "enum": [
    "autodetect",
    "qcow2",
    "rbd",
    "raw",
    "vdi",
    "vmdk",
    "cloop",
    "qed"
    ]
    },
    "cache": {
    "title": "Cache value for the disk",
    "description": "Cache value directly passed to the kvm -drive invocation as cache=value.",
    "type": "string",
    "default": "writeback",
    "enum": [
    "writeback",
    "none",
    "unsafe",
    "directsync",
    "writethrough"
    ]
    },
    "index": {
    "title": "Index of a disk",
    "description": "An index value used to order disks for the VM, if not specified, disk order will be random on each run.",
    "type": "integer",
    "default": 0
    }
    }
    }
    }
    }
    },
    "type": "object"
    ......
    software/kvm/instance-kvm-cluster.cfg.jinja2.in
    View file @ aca81eb3
    ......@@ -139,7 +139,7 @@ config-wipe-disk-iterations = {{ dumps(kvm_parameter_dict.get('wipe-disk-iterati
    config-document-host = ${apache-conf:ip}
    config-document-port = ${apache-conf:port}
    config-document-path = ${hash-code:passwd}
    {%- for k in ['boot-image-url-list', 'boot-image-url-select', 'whitelist-domains'] %}
    {%- for k in ['boot-image-url-list', 'boot-image-url-select', 'whitelist-domains', 'external-disk'] %}
    {#- play nice - use parameter only if present #}
    {%- if k in kvm_parameter_dict %}
    {#- play safe - dumps value #}
    ......
    software/kvm/instance-kvm-input-schema.json
    View file @ aca81eb3
    ......@@ -193,34 +193,6 @@
    "type": "boolean",
    "default": true
    },
    "external-disk-number": {
    "title": "Number of additional disk to create for virtual machine",
    "description": "Specify the number of additional disk to create for virtual machine in data folder of SlapOS Node. Requires instance_storage_home to be configured on SlapOS Node.",
    "type": "integer",
    "minimum": 0,
    "default": 0
    },
    "external-disk-size": {
    "title": "Size of additional disk to create for virtual machine, in Gigabytes",
    "description": "Specify the size of additional disk to create for virtual machine in data folder of SlapOS Node. Requires instance_storage_home to be configured on SlapOS Node.",
    "type": "integer",
    "minimum": 1,
    "default": 20
    },
    "external-disk-format": {
    "title": "Type of external disk drive to create by QEMU.",
    "description": "Type of QEMU disk drive, to create.",
    "type": "string",
    "default": "qcow2",
    "enum": [
    "qcow2",
    "raw",
    "vdi",
    "vmdk",
    "cloop",
    "qed"
    ]
    },
    "wipe-disk-ondestroy": {
    "title": "Wipe disks when destroy the VM",
    "description": "Say if disks should be wiped by writing new data over every single bit before delete them. This option is used to securely delete VM disks",
    ......@@ -402,6 +374,58 @@
    "description": "List of whitelisted domain names to be accessed from the VM. They will be resolved to IPs depending on where the VM end up. IPs can be used too.",
    "type": "string",
    "textarea": true
    },
    "external-disk": {
    "title": "External disks",
    "description": "Allows to set paths and types of manually provided external disks. Use index to order disks. Changing this parameter leads to kvm process restart.",
    "type": "object",
    "default": {},
    "patternProperties": {
    ".*": {
    "properties": {
    "path": {
    "title": "Path of the provided image",
    "description": "Absolute or relative path (to the partition root path) of the provided image. Image has to be provided, with read and write permissions of the partition user. Please note, that if the name starts with 'rbd:' it's considered special and won't be used as relative path.",
    "type": "string"
    },
    "format": {
    "title": "Format of a disk image",
    "description": "Format of a disk image. Note: rbd type is experimental.",
    "type": "string",
    "default": "autodetect",
    "enum": [
    "autodetect",
    "qcow2",
    "rbd",
    "raw",
    "vdi",
    "vmdk",
    "cloop",
    "qed"
    ]
    },
    "cache": {
    "title": "Cache value for the disk",
    "description": "Cache value directly passed to the kvm -drive invocation as cache=value.",
    "type": "string",
    "default": "writeback",
    "enum": [
    "writeback",
    "none",
    "unsafe",
    "directsync",
    "writethrough"
    ]
    },
    "index": {
    "title": "Index of a disk",
    "description": "An index value used to order disks for the VM, if not specified, disk order will be random on each run.",
    "type": "integer",
    "default": 0
    }
    }
    }
    }
    }
    }
    }
    software/kvm/instance-kvm.cfg.jinja2
    View file @ aca81eb3
    ......@@ -471,6 +471,9 @@ disk-storage-list =
    external-disk-number = ${slap-parameter:external-disk-number}
    external-disk-size = ${slap-parameter:external-disk-size}
    external-disk-format = ${slap-parameter:external-disk-format}
    # new external-disk parameter
    external-disk = ${slap-parameter:external-disk}
    instance-root = ${buildout:directory}
    {% if enable_http -%}
    httpd-port = ${slap-parameter:httpd-port}
    ......@@ -1091,6 +1094,7 @@ hard-drive-url-check-certificate = True
    external-disk-number = 0
    external-disk-size = 20
    external-disk-format = qcow2
    external-disk = {}
    # Help to get some configuration files into the vm from http
    enable-http-server = False
    ......
    software/kvm/template/template-kvm-run.in
    View file @ aca81eb3
    ......@@ -12,6 +12,7 @@ from random import shuffle
    import glob
    import re
    import json
    import operator
    # XXX: give all of this through parameter, don't use this as template, but as module
    qemu_img_path = {{ repr(parameter_dict["qemu-img-path"]) }}
    ......@@ -46,6 +47,8 @@ pid_file_path = '{{ parameter_dict.get("pid-file-path") }}'
    external_disk_number = {{ parameter_dict.get("external-disk-number") }}
    external_disk_size = {{ parameter_dict.get("external-disk-size") }}
    external_disk_format = {{ repr(parameter_dict["external-disk-format"]) }}
    external_disk = {{ parameter_dict['external-disk'] }}
    instance_root = '{{ parameter_dict['instance-root'] }}'
    disk_storage_dict = {}
    disk_storage_list = """{{ parameter_dict.get("disk-storage-list") }}""".split('\n')
    map_storage_list = []
    ......@@ -305,6 +308,35 @@ for disk in additional_disk_list:
    kvm_argument_list.extend([
    '-drive', 'file=%s,if=%s' % (disk, disk_type)])
    # allow empty index if only one disk is provided
    automatic_index_applied = False
    for entry in external_disk.keys():
    if 'index' not in external_disk[entry]:
    if automatic_index_applied:
    raise ValueError('index is missing and more than one disk is present in external-disk configuration')
    external_disk[entry]['index'] = 0
    automatic_index_applied = True
    # support external-disk parameter
    for disk_info in sorted(external_disk.values(), key=operator.itemgetter('index')):
    if disk_info['path'].startswith('rbd:') or disk_info['path'].startswith('/'):
    path = disk_info['path']
    else:
    path = os.path.join(instance_root, disk_info['path'])
    drive_argument_list = [
    "file=%s" %( path,),
    "if=%s" % (disk_type,),
    "cache=%s" % (disk_info.get('cache', 'writeback'),)
    ]
    if disk_info.get('format', 'autodetect') != 'autodetect':
    drive_argument_list.append(
    "format=%s" % (disk_info['format'],)
    )
    kvm_argument_list += (
    '-drive',
    ",".join(drive_argument_list)
    )
    if auto_ballooning:
    kvm_argument_list.extend(['-device', 'virtio-balloon-pci,id=balloon0'])
    ......
    software/kvm/test/test.py
    View file @ aca81eb3
    ......@@ -1868,8 +1868,25 @@ class TestParameterCluster(TestParameterDefault):
    return 'kvm-cluster'
    class ExternalDiskMixin(KvmMixin):
    def getRunningDriveList(self, kvm_instance_partition):
    _match_drive = re.compile('file.*if=virtio.*').match
    with self.slap.instance_supervisor_rpc as instance_supervisor:
    kvm_pid = next(q for q in instance_supervisor.getAllProcessInfo()
    if 'kvm-' in q['name'])['pid']
    drive_list = []
    for entry in psutil.Process(kvm_pid).cmdline():
    m = _match_drive(entry)
    if m:
    path = m.group(0)
    drive_list.append(
    path.replace(kvm_instance_partition, '${partition}')
    )
    return drive_list
    @skipUnlessKvm
    class TestExternalDisk(InstanceTestCase, KvmMixin):
    class TestExternalDisk(InstanceTestCase, ExternalDiskMixin):
    __partition_reference__ = 'ed'
    kvm_instance_partition_reference = 'ed0'
    ......@@ -1958,21 +1975,6 @@ class TestExternalDisk(InstanceTestCase, KvmMixin):
    super(TestExternalDisk, cls).tearDownClass()
    shutil.rmtree(cls.working_directory)
    def getRunningDriveList(self, kvm_instance_partition):
    _match_drive = re.compile('file=(.+),if=virtio').match
    with self.slap.instance_supervisor_rpc as instance_supervisor:
    kvm_pid = next(q for q in instance_supervisor.getAllProcessInfo()
    if 'kvm-' in q['name'])['pid']
    dirve_list = []
    for entry in psutil.Process(kvm_pid).cmdline():
    m = _match_drive(entry)
    if m:
    path = m.group(1)
    dirve_list.append(
    path.replace(kvm_instance_partition, '${partition}')
    )
    return dirve_list
    def test(self):
    kvm_instance_partition = os.path.join(
    self.slap.instance_directory, self.kvm_instance_partition_reference)
    ......@@ -2005,3 +2007,177 @@ class TestExternalDisk(InstanceTestCase, KvmMixin):
    class TestExternalDiskJson(
    KvmMixinJson, TestExternalDisk):
    pass
    @skipUnlessKvm
    class TestExternalDiskModern(InstanceTestCase, ExternalDiskMixin):
    __partition_reference__ = 'edm'
    kvm_instance_partition_reference = 'edm0'
    @classmethod
    def getInstanceSoftwareType(cls):
    return 'default'
    @classmethod
    def setUpClass(cls):
    super(TestExternalDiskModern, cls).setUpClass()
    def getExternalDiskInstanceParameterDict(self, first, second, third):
    return {
    "external-disk": {
    "second disk": {
    "path": second,
    "index": 2,
    },
    "third disk": {
    "path": third,
    "index": 3,
    "cache": "none"
    },
    "first disk": {
    "path": first,
    "index": 1,
    "format": "qcow"
    },
    }
    }
    def test(self):
    # Disks can't be created in /tmp, as it's specially mounted on testnodes
    # and then KVM can't use them:
    # -drive file=/tmp/tmpX/third_disk,if=virtio,cache=none: Could not open
    # '/tmp/tmpX/third_disk': filesystem does not support O_DIRECT
    self.working_directory = tempfile.mkdtemp(dir=self.slap.instance_directory)
    self.addCleanup(shutil.rmtree, self.working_directory)
    kvm_instance_partition = os.path.join(
    self.slap.instance_directory, self.kvm_instance_partition_reference)
    # find qemu_img from the tested SR via it's partition parameter, as
    # otherwise qemu-kvm would be dependency of test suite
    with open(
    os.path.join(self.computer_partition_root_path, 'buildout.cfg')) as fh:
    qemu_img = [
    q for q in fh.readlines()
    if 'raw qemu_img_executable_location' in q][0].split()[-1]
    self.first_disk = os.path.join(self.working_directory, 'first_disk')
    subprocess.check_call([
    qemu_img, "create", "-f", "qcow", self.first_disk, "1M"])
    second_disk = 'second_disk'
    self.second_disk = os.path.join(kvm_instance_partition, second_disk)
    subprocess.check_call([
    qemu_img, "create", "-f", "qcow2", os.path.join(
    kvm_instance_partition, self.second_disk), "1M"])
    self.third_disk = os.path.join(self.working_directory, 'third_disk')
    subprocess.check_call([
    qemu_img, "create", "-f", "qcow2", self.third_disk, "1M"])
    self.rerequestInstance({'_': json.dumps(
    self.getExternalDiskInstanceParameterDict(
    self.first_disk, second_disk, self.third_disk))})
    self.waitForInstance()
    drive_list = self.getRunningDriveList(kvm_instance_partition)
    self.assertEqual(
    drive_list,
    [
    'file=${partition}/srv/virtual.qcow2,if=virtio,discard=on,'
    'format=qcow2',
    'file=%s/first_disk,if=virtio,cache=writeback,format=qcow' % (
    self.working_directory,),
    'file=${partition}/second_disk,if=virtio,cache=writeback',
    'file=%s/third_disk,if=virtio,cache=none' % (
    self.working_directory,)
    ]
    )
    @skipUnlessKvm
    class TestExternalDiskModernCluster(TestExternalDiskModern):
    kvm_instance_partition_reference = 'edm1'
    @classmethod
    def getInstanceParameterDict(cls):
    return {'_': json.dumps({
    "kvm-partition-dict": {
    "kvm-default": {
    "disable-ansible-promise": True,
    }
    }
    })}
    @classmethod
    def getInstanceSoftwareType(cls):
    return 'kvm-cluster'
    def getExternalDiskInstanceParameterDict(self, *args, **kwargs):
    partition_dict = super(
    TestExternalDiskModernCluster, self
    ).getExternalDiskInstanceParameterDict(*args, **kwargs)
    partition_dict.update({"disable-ansible-promise": True})
    return {
    "kvm-partition-dict": {
    "kvm-default": partition_dict
    }
    }
    @skipUnlessKvm
    class TestExternalDiskModernIndexRequired(InstanceTestCase, ExternalDiskMixin):
    __partition_reference__ = 'edm'
    kvm_instance_partition_reference = 'edm0'
    @classmethod
    def getInstanceSoftwareType(cls):
    return 'default'
    @classmethod
    def setUpClass(cls):
    super(TestExternalDiskModern, cls).setUpClass()
    def getExternalDiskInstanceParameterDict(self, first, second, third):
    return {
    "external-disk": {
    "second disk": {
    "path": second,
    },
    "third disk": {
    "path": third,
    "index": 3,
    },
    "first disk": {
    "path": first,
    "index": 1,
    },
    }
    }
    def test(self):
    # Disks can't be created in /tmp, as it's specially mounted on testnodes
    # and then KVM can't use them:
    # -drive file=/tmp/tmpX/third_disk,if=virtio,cache=none: Could not open
    # '/tmp/tmpX/third_disk': filesystem does not support O_DIRECT
    self.working_directory = tempfile.mkdtemp(dir=self.slap.instance_directory)
    self.addCleanup(shutil.rmtree, self.working_directory)
    kvm_instance_partition = os.path.join(
    self.slap.instance_directory, self.kvm_instance_partition_reference)
    # find qemu_img from the tested SR via it's partition parameter, as
    # otherwise qemu-kvm would be dependency of test suite
    with open(
    os.path.join(self.computer_partition_root_path, 'buildout.cfg')) as fh:
    qemu_img = [
    q for q in fh.readlines()
    if 'raw qemu_img_executable_location' in q][0].split()[-1]
    self.first_disk = os.path.join(self.working_directory, 'first_disk')
    subprocess.check_call([
    qemu_img, "create", "-f", "qcow", self.first_disk, "1M"])
    second_disk = 'second_disk'
    self.second_disk = os.path.join(kvm_instance_partition, second_disk)
    subprocess.check_call([
    qemu_img, "create", "-f", "qcow2", os.path.join(
    kvm_instance_partition, self.second_disk), "1M"])
    self.third_disk = os.path.join(self.working_directory, 'third_disk')
    subprocess.check_call([
    qemu_img, "create", "-f", "qcow2", self.third_disk, "1M"])
    self.rerequestInstance({'_': json.dumps(
    self.getExternalDiskInstanceParameterDict(
    self.first_disk, second_disk, self.third_disk))})
    self.raising_waitForInstance()
    Markdown is supported
    0% or
    You are about to add 0 people to the discussion. Proceed with caution.
    Finish editing this message first!
    Please register or to comment
    GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7