# Extend Software Release Using Webrunner

This document will walk through the steps of extending an existing software release available on SlapOS. It will demonstrate how to add parameters, components and features.

To follow the steps in this document, you should have read understanding SlapOS architecture and be familiar with Buildout. It is also required to have the Webrunner Software added to the SlapOS Master software catalouge and have an instance of the Webrunner with the included helloworld software release deployed inside.

# Table of Content

- Add Parameter to all Software Instances
- Add Feature
- Add Component
- Add Promise

# Add Parameter to all Software Instances

This section will modify the "helloworld" Software Release, a software that creates a simple web server and publishes the text hello <configuration.name>.

This will be done using a Webrunner a development IDE, that contains a "mini SlapOS Master" for deploying a single software (helloworld). In more complex SlapOS networks, a master manages a catalog of softwares which are supplied and instantiated on a network of nodes, some of which may again be Webrunners containing a single software release. Webrunners are useful, because they provide resiliency for this single software by hosting multiple "internal" instances. The deployed software may also be patched and can be accessed via SSH which for some softwares may be required to access the underlying file system.

The software release of "Helloworld" is relatively simple and composed of two files:

- `software.cfg` - the Buildout profile which installs the software release on a computer
- `instance.cfg.in` - the instance profile filled by `software.cfg` on instantiation

`instance.cfg.in` will be used to deploy (configure and run) one (on the Webrunner) or multiple (on a SlapOS network) instances of a software. Note, that after Buildout has filled the `instance.cfg.in` file, there should be a hidden file called `.installed.cfg`. In case of instantiation failing this is a good place to look for what went wrong.

# Check Connection Parameters

Log in to the webrunner and open the Services tab. Check the Connection Information Slap Response to see which parameters the helloworld software release is currently outputting. The connection urls of the webservers in Go, Ruby and Python are also outputted. Opening either one will show the default page with the name parameter.

# Locate Helloworld Software.cfg

Head over to the Editor. Click the fullscreen icon in the toolbar, then locate the Helloworld folder in the Software directory.

# Software.cfg

All `software.cfg` profiles extend the default slapos.cfg (which contains basic slapos deployment tools). In this example it is extended by `../../component/helloweb/buildout.cfg` - the profile to create some programs that say "hello" in three different languages (python, ruby, golang). It also describes how to parse the instance profile `instance.cfg.in` into the rendered file later used for supplying instances of our helloworld software.

Note the comment regarding the MD5 checksum and then either comment out your checksum or replace it with the hash of `[your_instance.cfg.in]`. Otherwise the build will fail when checking the integrity of the file.

One way to customize a software instance is through providing instance parameters when requesting an instance from a SlapOS Master using the instance XML configuration. Let's add a new parameter called title to display the title of a web page alongside the name. Open the `instance.cfg.in` in the same directory.

# Instance.cfg.in

In the `instance.cfg.in` `[instance-parameter]` section, add a new parameter like so:

    # XXX This is the declaration of the default value for the parameter "title"
    # (None here). If commented, the recipe will crash. The "configuration." part
    # of the parameter name means that the value will be the one given (if present)
    # in instance parameters
    configuration.title =

And in the `[publish-connection-parameter]` section, output the new parameter as well:

    # XXX the published parameter "title". By default, its value is the value
    # declared in the section [instance-parameter] above.
    title = Title ${instance-parameter:configuration.title}!

# Rebuild Software Release

Save the modified file in the Webrunner, don't forget to uncomment the MD5 checksum in the `software.cfg`, then hit the green play button in the top right toolbar to rebuild your Webrunner (you may have to get out of fullscreen mode to show the green button again). Rebuilding should take a few minutes and will not complete in case you have made an error when updating the files.

# Successful Rebuild

Once both indicators switch to completed and green status the helloworld software has been rebuilt with the modified configuration files. Head back to Services to check the Connection Information.

# Updated Software Connection Information

The `title` parameter is now displayed as well. As we did not provide a parameter akin to John Doe for the `name`, nothing is displayed.

While this update was done in a Webrunner and therefore only on a single instance, it can just as well be done on a SlapOS network. In this case, the Software Release points to a repository with the `software.cfg` and `instance.cfg.in`. Updating these profiles and then adding a new version entry to the SlapOS Master software catalogue will trigger a pending upgrade of all instances of this software.

# Add Feature

After working with existing sections, the following section will show how to add new sections using different recipes. We will be adding a template file and render it during instantiation.

# Add Template Folder and Template

Head back to the Editor and create a new folder named `/template`. Create a new file named `rendered.in` and open it.

SlapOS software configuration templates are using Jinja2 (more info on the jinja2 structure and syntax). Add the following content to the file and save:

    {% if title %}
    {{ title }}
    -
    {% endif %}
    Title: {{ name }}!

In order to import this template into the software release, the software profile needs to be modified.

# Add Template to Software Profile

Note that there are now different templating formats being used:

- Buildout templates using `object['parameter']` and `${:variable name}`
- Jinja templates using the syntax: `{{ }}`

First, in `software.cfg` add a new entry to the `parts` section called template-get (Note, that get is an arbitrarly chosen name given to our template). Then update the `[instance-profile]` context (see jinja specifications on context) to include a second section to be available in `instance.cfg.in`:

    context =
      section buildout buildout
      section template_get template-get

Note for exposing context parameters, there must always be three parts (type, name and expression) separated by whitespace, that multiple entries go on separate lines and that the name cannot contain "-", because this will be the caller in later requests.

Finally add the corresponding section to the end of the file:

    # Retrieve Jinja2 template to render via url. Location is only used
    # to share information through buildout (corresponds to a default location)
    [template-get]
    recipe = hexagonit.recipe.download
    url = ${:_profile_base_location_}/template/${:filename}
    location = ${buildout:parts-directory}/${:_buildout_section_name_}
    filename = rendered.in
    download-only = true
    extensions = jinja2.ext.do
    context =
      section buildout buildout

Note that this defines the location and filename variables we will require in the next step. Head over to `instance.cfg.in`

# Update Instance.cfg.in

As in the software profile, start by adding a new entry called template-creation to the `[buildout] parts` section along with a new folder after the var and etc folders using:

    # Create folder which will contain the jinja template
    data = ${:home}/srv/data

Then add a new section:

    # XXX Jinja action!
    [template-creation]
    recipe = slapos.recipe.template:jinja2

    # The template value will be replaced via the info given in software.cfg
    template = {{ template_get['location'] }}/{{ template_get['filename'] }}

    # Declare where we want to get the new file rendered by Jinja
    rendered = ${directory:data}/data1.txt
    mode = 700

    # Using context we can define specific values wich will be substituted in the
    # template. "key" means that the value is defined by buildout.
    context =
      key name instance-parameter:configuration.name
      key title instance-parameter:configuration.title

Note, the use of template_get which we defined earlier in the `software.cfg` context of the `[instance-profile]`

Also, it is possible define the template inline.

    # template = inline:
    #  ----------------------------------
    #  Title : {{title}}
    #  ----------------------------------
    #  Hello {{name}}

Save the file and rebuild by pressing the green play button again.

# Verify Template is Available

If everything was done correctly you should find the rendered template at `${template-creation:rendered}`, which should compute to the the `srv/data/data1.txt` on your instance.

There are plenty of pre-existing recipes available online (see the buildout documentation) as well as in the Slapos Git repository.

Should you run into issues, make sure to check the software and instance logs in the Logs section and make sure that sections you have defined (like template-get and template-creation) show up in the logs and can be processed without errors.

# Add Component

It is common having to add a new component to a software release in order to add certain features. There is a number of components available to use from the SlapOS repository.

This section will implement a log rotation feature using logrotate, cron and gzip.

# Add Logrotate To Software.cfg

Start by adding the required components to the `extends` section:

    # Add needed components
      ../../component/dcron/buildout.cfg
      ../../component/logrotate/buildout.cfg
      ../../component/gzip/buildout.cfg

and in the `parts` section, add the deployment entries:

    # Deployment of new components.
      dcron
      logrotate

Logrotate allows rotation of log files and cron will execute them regularly. Gzip is just added to have a more concrete example. Next update the `instance.cfg.in`.

# Configure Components in instance.cfg.in

Start by creating directories for configs and logs. Comment out the current data folder created for the jinja template earlier, then add the following folders to your `[directory]` recipe:

    bin = ${:home}/bin
    srv = ${:home}/srv
    data = ${:srv}/data
    # path of the log directory used by our service (see [hello-world])
    log = ${:var}/log

    # cron directories
    cronentries = ${:etc}/cron.d
    crontabs = ${:etc}/crontabs
    cronstamps = ${:etc}/cronstamps

    # logrotate directories
    backup = $${:srv}/backup
    logrotate-backup = $${:backup}/logrotate
    logrotate-entries = $${:etc}/logrotate.d

Next we'll install and deploy cron

Add a new section for cron:

    # Deploy cron service
    [cron]
    recipe = slapos.cookbook:cron
    dcrond-binary = ${dcron:location}/sbin/crond
    cronentries = ${directory:cronentries}
    crontabs = ${directory:crontabs}
    cronstamps = ${directory:cronstamps}
    catcher = ${cron-simplelogger:wrapper}
    binary = ${directory:service}/crond

    # Add logger to cron
    [cron-simplelogger]
    recipe = slapos.cookbook:simplelogger
    wrapper = ${directory:bin}/cron_simplelogger
    log = ${directory:log}/cron.log

    # Add a cron entry
    [cron-entry-logrotate]
    <= cron
    recipe = slapos.cookbook:cron.d
    name = logrotate
    frequency = 0 * * * *
    # Now that the cron is set up, we can define
    # the command to launch at the defined frequency
    command = $${logrotate:wrapper}

First, we deploy cron using cron's recipe. In this recipe we use as a catcher for logs with a wrapper defined in the cron-simplelogger section. These buildout sections where copied from the apache-frontend software profile. It means that it easy to reuse sections from other profiles if they provide features matching your needs.

Now that the cron is set up, we only need to add the command to launch logrotate at the defined frequency:

    command = $${logrotate:wrapper}

The last step is adding the `logrotate` itself:

    # Deploy Logrotate (see ../../slapos/recipe/logrogate.py)
    [logrotate]
    recipe = slapos.cookbook:logrotate

    # Binaries
    logrotate-binary = ${logrotate:location}/usr/sbin/logrotate
    gzip-binary = ${gzip:location}/bin/gzip
    gunzip-binary = ${gzip:location}/bin/gunzip

    # Directories
    wrapper = $${directory:bin}/logrotate
    conf = $${directory:etc}/logrotate.conf

    # in this folder will be defined tho files to save, and how to save them
    logrotate-entries = $${directory:logrotate-entries}

    # in this folder the logs will be compressed then saved
    backup = $${directory:logrotate-backup}
    state-file = $${directory:srv}/logrotate.status

    [logrotate-entry]
    <= logrotate
    recipe = slapos.cookbook:logrotate.d
    name = helloworld-logrotate
    log = $${directory:log}/log.log
    frequency = daily
    rotatep-num = 30
    sharedscripts = true
    notifempty = true
    create = true

# Rebuild Software

Once all changes have been made, save the files and rebuild the software clicking the green button. It should finish without problems.

# Verify Logs Are Running

Once the rebuild finished, head to Editor and check whether the instance includes the required log files.

# Update MD5 Sum

Once you finished all modifications, don't forget to uncomment and update the MD5 Sum. On your active file, click the Menu button in the subheader and select Get or Update MD5 sum. The MD5 sum will be displayed on the bottom of the screen. Paste it into your `software.cfg` profile and rebuild again.

# Add Promise

Promise is an executable doing some arbitrary work, then exiting with exit code 0 ("it works") or greater ("it doesn't work"). They are run by slapgrid to know whether an instance is working or not.

This section will add a simple promise to the helloworld software profile.

# Promise components

- Anonmaly
- Sense
- Test

Writing a promise consists of defining a class called RunPromise which inherits from the GenericPromise class and defines three methods:

- anomaly()
  - returns AnomalyResult object describing the promise state. The anomaly method is called by slapgrid when the partition is correctly processed to check if the partition has no anomaly. If AnomalyResult.hasFailed() is True, bang is called if another promise of the same instance didn't call bang.
- sense()
  - runs the promise with the given frequency, collects data for the promise whenever is makes sense and appends to a log file.
- test()
  - checks TestResult object describing the actual promise state. Test method is called when buildout process a partition, a partition is marked as correctly processed if there is no buildout failures and all promises test() pass.

# Check Port Listening Promise

The most simple example of promise is "check_if_port_listening" (https://lab.nexedi.com/nexedi/slapos/blob/master/slapos/recipe/check_port_listening.py) trying to open a socket to an ip/port. If it works, the promise exits with exit code 0, and slapgrid knows that the instance is working. If the socket can't be created, it exits with a different exit code, and slapgrid reports it to the SlapOS Master.

# Add Existing Promise to Software Release

In your instance profile, add the following code:

    # port listening promise
    [frontend-promise]
    recipe = slapos.cookbook:check_url_available
    path = ${directory:promise}/frontend_promise
    url = ${instance-parameter:url}
    dash_path = ${dash:location}/bin/dash
    curl_path = ${curl:location}/bin/curl

Don't forget to add frontend-promise to the `parts` section.

This will call the promise `check_url_available` previously shown and generate an executable in `$${directory:promises}/frontend_promise` that when run, will try to connect to the url provided. We also need to specify the path for dash and curl executables. Save the file, update the MD5 sum and rebuild.

# Thank You

- Nexedi SA
- 147 Rue du Ballon
- 59110 La Madeleine
- France

- nexedi.com/contact