Introduce CI documentation for services and languages

doc/README.md

@@ -24,9 +24,21 @@
 - [Using Docker Images](ci/docker/using_docker_images.md)
 - [Using Docker Build](ci/docker/using_docker_build.md)
 - [Using Variables](ci/variables/README.md)
+- [Using SSH keys](ci/ssh_keys/README.md)
 - [User permissions](ci/permissions/README.md)
 - [API](ci/api/README.md)
 
+### CI Languages
+
++ [Testing PHP](ci/languages/php.md)
+
+## CI Services
+
++ [Using MySQL](ci/services/mysql.md)
++ [Using PostgreSQL](ci/services/postgres.md)
++ [Using Redis](ci/services/redis.md)
++ [Using Other Services](ci/docker/using_docker_images.html#how-to-use-other-images-as-services)
+
 ### CI Examples
 
 - [Test and deploy Ruby applications to Heroku](ci/examples/test-and-deploy-ruby-application-to-heroku.md)

doc/ci/README.md

@@ -9,6 +9,18 @@
 + [Using Docker Images](docker/using_docker_images.md)
 + [Using Docker Build](docker/using_docker_build.md)
 + [Using Variables](variables/README.md)
++ [Using SSH keys](ssh_keys/README.md)
+
+### Languages
+
++ [Testing PHP](languages/php.md)
+
+### Services
+
++ [Using MySQL](services/mysql.md)
++ [Using PostgreSQL](services/postgres.md)
++ [Using Redis](services/redis.md)
++ [Using Other Services](docker/using_docker_images.html#how-to-use-other-images-as-services)
 
 ### Examples
 

doc/ci/docker/using_docker_images.md

@@ -42,7 +42,44 @@ So, **to access your database service you have to connect to host: `mysql` inste
 
 ### How to use other images as services?
 You are not limited to have only database services. 
-You can hand modify `config.toml` to add any image as service found at [Docker Hub](https://registry.hub.docker.com/). 
+You can add the services to `.gitlab-ci.yml` or hand modify the `config.toml`.
+You can use any image as service found at [Docker Hub](https://registry.hub.docker.com/). 
+
+### Define image and services from `.gitlab-ci.yml`
+You can simply define image or list services that you want to use for the build time.
+```
+image: ruby:2.2
+services:
+  - postgres:9.3
+before_install:
+  - bundle install
+  
+test:
+  script:
+  - bundle exec rake spec
+```
+
+It's possible to define image and service per-job:
+```
+before_install:
+  - bundle install
+
+test:2.1:
+  image: ruby:2.1
+  services:
+  - postgres:9.3
+  script:
+  - bundle exec rake spec
+
+test:2.2:
+  image: ruby:2.2
+  services:
+  - postgres:9.4
+  script:
+  - bundle exec rake spec
+```
+
+### Define image and services in `config.toml`
 Look for `[runners.docker]` section:
 ```
 [runners.docker]
@@ -50,13 +87,16 @@ Look for `[runners.docker]` section:
   services = ["mysql:latest", "postgres:latest"]
 ```
 
+The image and services defined these way will be added to all builds run by that runner.
+
+### Accessing the services
 For example you need `wordpress` instance to test some API integration with `Wordpress`. 
-You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/). 
-This is image that have fully preconfigured `wordpress` and have `MySQL` server built-in:
+You can for example use this image: [tutum/wordpress](https://registry.hub.docker.com/u/tutum/wordpress/).
+
 ```
-[runners.docker]
-  image = "ruby:2.1"
-  services = ["mysql:latest", "postgres:latest", "tutum/wordpress:latest"]
+# .gitlab-ci.yml
+services:
+- tutum/wordpress:latest
 ```
 
 Next time when you run your application the `tutum/wordpress` will be started 
@@ -64,7 +104,7 @@ and you will have access to it from your build container under hostname: `tutum_
 
 Alias hostname for the service is made from the image name:
 1. Everything after `:` is stripped,
-2. '/' is replaced with `__`.
+2. '/' is replaced to `__`.
 
 ### Configuring services
 Many services accept environment variables, which allow you to easily change database names or set account names depending on the environment.
@@ -99,67 +139,6 @@ or README page for any other Docker image.
 
 **Note: All variables will passed to all service containers. It's not designed to distinguish which variable should go where.**
 
-### Overwrite image and services
-It's possible to overwrite `docker-image` and specify services from `.gitlab-ci.yml`.
-If you add to your YAML the `image` and the `services` these parameters
-be used instead of the ones that were specified during runner's registration.
-```
-image: ruby:2.2
-services:
-  - postgres:9.3
-before_install:
-  - bundle install
-  
-test:
-  script:
-  - bundle exec rake spec
-```
-
-It's possible to define image and service per-job:
-```
-before_install:
-  - bundle install
-
-test:2.1:
-  image: ruby:2.1
-  services:
-  - postgres:9.3
-  script:
-  - bundle exec rake spec
-
-test:2.2:
-  image: ruby:2.2
-  services:
-  - postgres:9.4
-  script:
-  - bundle exec rake spec
-```
-
-#### How to enable overwriting?
-To enable overwriting you have to **enable it first** (it's disabled by default for security reasons). 
-You can do that by hand modifying runner configuration: `config.toml`. 
-Please go to section where is `[runners.docker]` definition for your runner. 
-Add `allowed_images` and `allowed_services` to specify what images are allowed to be picked from `.gitlab-ci.yml`:
-```
-[runners.docker]
-  image = "ruby:2.1"
-  allowed_images = ["ruby:*", "python:*"]
-  allowed_services = ["mysql:*", "redis:*"]
-```
-This enables you to use in your `.gitlab-ci.yml` any image that matches above wildcards. 
-You will be able to pick only `ruby` and `python` images. 
-The same rule can be applied to limit services. 
-
-If you are courageous enough, you can make it fully open and accept everything:
-```
-[runners.docker]
-  image = "ruby:2.1"
-  allowed_images = ["*", "*/*"]
-  allowed_services = ["*", "*/*"]
-```
-
-**It the feature is not enabled, or image isn't allowed the error message will be put into the build log.**
-
 ### How Docker integration works
 1. Create any service container: `mysql`, `postgresql`, `mongodb`, `redis`.
 1. Create cache container to store all volumes as defined in `config.toml` and `Dockerfile` of build image (`ruby:2.1` as in above example).

doc/ci/languages/README.md

+### Languages
+
++ [Testing PHP](php.md)

doc/ci/languages/php.md

+## Testing PHP projects
+
+This guide covers basic of building PHP projects.
+
+Is it possible to test PHP apps on any system.
+However, it will require manual configuration.
+The simplest is to use Docker executor as described below.
+
+### PHP projects on Docker executor
+It's possible to official [PHP](https://hub.docker.com/_/php/) repositories on Docker Hub.
+They allow to test PHP projects against different versions of the runtime.
+However, they require additional configuration.
+
+To build PHP project you need to create valid `.gitlab-ci.yml` describing the build environment:
+1. First you need to specify PHP image as described here: http://doc.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-image. To your `.gitlab-ci.yml` add:
+
+		image: php:5.6
+
+2. The official images are great, but they are lacking a few useful tools for testing. We need to install them first in build environment. Create `ci/docker_install.sh` file with following content:
+
+		#!/bin/bash
+
+		# We need to install dependencies only for Docker
+		[[ ! -e /.dockerinit ]] && exit 0
+
+		set -xe
+
+		# Install git (the php image doesn't have it) which is required by composer
+		apt-get update -yqq
+		apt-get install git -yqq
+
+		# Install phpunit, the tool that we will use for testing
+		curl -o /usr/local/bin/phpunit https://phar.phpunit.de/phpunit.phar
+		chmod +x /usr/local/bin/phpunit
+
+		# Install mysql driver
+		# Here you can install any other extension that you need
+		docker-php-ext-install pdo_mysql
+
+3. From your `.gitlab-ci.yml` run the created script:
+
+		before_script:
+		- bash ci/docker_install.sh > /dev/null
+
+4. Now you can run your tests. Usually it will be `phpunit` with arguments:
+
+		test:app:
+		  script:
+		  - phpunit --configuration phpunit_myapp.xml --coverage-text
+
+5. Commit your files, and push them to GitLab to see if it works. With GitLab Runner 1.0 you can also test the changes locally. From your terminal execute:
+
+		# Check using docker executor
+		gitlab-runner exec docker test:app
+
+		# Check using shell executor
+		gitlab-runner exec shell test:app
+
+The final `.gitlab-ci.yml` should look similar to this:
+
+		# Select image from https://hub.docker.com/_/php/
+		image: php:5.6
+
+		before_script:
+		# Install dependencies
+		- ci/docker_install.sh > /dev/null
+
+		test:app:
+		  script:
+		  - phpunit --configuration phpunit_myapp.xml --coverage-text
+
+#### Test against different PHP versions in Docker builds
+
+You can also test against multiple version of PHP runtime:
+
+		before_script:
+		# Install dependencies
+		- ci/docker_install.sh > /dev/null
+
+		# We test PHP5.6
+		test:5.6:
+		  image: php:5.6
+		  script:
+		  - phpunit --configuration phpunit_myapp.xml --coverage-text
+
+		# We test PHP7.0
+		test:7.0:
+		  image: php:7.0
+		  script:
+		  - phpunit --configuration phpunit_myapp.xml --coverage-text
+
+#### Custom PHP configuration in Docker builds
+
+You can customise your PHP environment by putting your .ini file into `/usr/local/etc/php/conf.d/`:
+
+		before_script:
+		- cp my_php.ini /usr/local/etc/php/conf.d/test.ini
+
+### Test PHP projects using Shell
+
+Shell executor runs your builds in terminal session of your server. Thus in order to test your projects you need to have all dependencies installed as root.
+
+1. Install PHP dependencies:
+
+		sudo apt-get update -qy
+		sudo apt-get install phpunit php5-mysql -y
+
+	This will install the PHP version available for your distribution.
+
+2. Now you can run your tests. Usually it will be `phpunit` with arguments:
+
+		test:app:
+		  script:
+		  - phpunit --configuration phpunit_myapp.xml --coverage-text
+
+#### Test against different PHP versions in Shell builds
+
+The [phpenv](https://github.com/phpenv/phpenv) allows you to easily manage different PHP with they own configs.
+This is specially usefull when testing PHP project with Shell executor.
+
+Login as `gitlab-runner` user and follow [the installation guide](https://github.com/phpenv/phpenv#installation).
+
+Using phpenv also allows to easily configure PHP environment with: `phpenv config-add my_config.ini`.
+
+#### Install custom extensions
+
+Since we have pretty bare installation of our PHP environment you may need some extensions that are not present on your installation.
+
+To install additional extensions simply execute.:
+
+		pecl install <extension>
+
+	It's not advised to add this to the `.gitlab-ci.yml`.
+	You should execute this command once, only to setup the build environment.
+
+### Extend your tests
+
+#### Using atoum
+
+Instead of PHPUnit, you can use any other tool to run unit tests. For example [atoum](https://github.com/atoum/atoum):
+
+		before_script:
+		- wget http://downloads.atoum.org/nightly/mageekguy.atoum.phar
+
+		test:atoum:
+		  script:
+		  - php mageekguy.atoum.phar
+
+#### Using Composer
+
+Majority of the PHP projects use Composer for managing the packages.
+It's very simple to execute the Composer before running your tests.
+To your `.gitlab-ci.yml` add:
+
+		# The composer stores all downloaded packages in vendor/
+		# Remove it if you committed the vendor/ directory
+		cache:
+		  paths:
+		  - vendor/
+
+		before_script:
+		# Install composer dependencies
+		- curl -sS https://getcomposer.org/installer | php
+		- php composer.phar install
+
+### Access private packages / dependencies
+
+You need to configure [the SSH keys](../ssh_keys/README.md) in order to checkout the repositories.
+
+### Use databases or other services
+
+Please checkout the docs about configuring [the CI services](../services/README.md).
+
+### Example project
+
+You maybe interested in our [Example Project](https://gitlab.com/gitlab-examples/php) that runs on [GitLab.com](https://gitlab.com) using our publically available shared runners.
+
+Want to hack it? Simply fork it, commit and push changes. Within a few moments the changes will be picked and rebuilt by public runners.

doc/ci/services/README.md

+## GitLab CI Services
+
++ [Using MySQL](mysql.md)
++ [Using PostgreSQL](postgres.md)
++ [Using Redis](redis.md)
++ [Using Other Services](../docker/using_docker_images.html#how-to-use-other-images-as-services)

doc/ci/services/docker-services.md

+## GitLab CI Services
+
++ [Using MySQL](mysql.md)
++ [Using PostgreSQL](postgres.md)
++ [Using Redis](redis.md)

doc/ci/services/mysql.md

+## Using MySQL
+
+It's possible to use MySQL database test your apps during builds.
+
+### Use MySQL with Docker executor
+
+If you are using our Docker integration you basically have everything already.
+
+1. Add this to your `.gitlab-ci.yml`:
+
+		services:
+		- mysql
+
+		variables:
+		  # Configure mysql service (https://hub.docker.com/_/mysql/)
+		  MYSQL_DATABASE: hello_world_test
+		  MYSQL_ROOT_PASSWORD: mysql
+
+2. Configure your application to use the database:
+
+		Host: mysql
+		User: root
+		Password: mysql
+		Database: hello_world_test
+
+3. You can also use any other available on [DockerHub](https://hub.docker.com/_/mysql/). For example: `mysql:5.5`. 
+
+Example: https://gitlab.com/gitlab-examples/mysql/blob/master/.gitlab-ci.yml
+
+### Use MySQL with Shell executor
+
+It's possible to use MySQL on manually configured servers that are using GitLab Runner with Shell executor.
+
+1. First install the MySQL server:
+	
+		sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev
+
+		# Pick a MySQL root password (can be anything), type it and press enter
+		# Retype the MySQL root password and press enter
+
+2. Create an user:
+
+		mysql -u root -p
+
+		# Create a user which will be used by your apps
+		# do not type the 'mysql>', this is part of the prompt
+		# change $password in the command below to a real password you pick
+		mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password';
+
+		# Ensure you can use the InnoDB engine which is necessary to support long indexes
+		# If this fails, check your MySQL config files (e.g. `/etc/mysql/*.cnf`, `/etc/mysql/conf.d/*`) for the setting "innodb = off"
+		mysql> SET storage_engine=INNODB;
+
+		# Create the database
+		mysql> CREATE DATABASE IF NOT EXISTS `hello_world_test` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`;
+
+		# Grant necessary permissions on the database
+		mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `hello_world_test`.* TO 'runner'@'localhost';
+
+		# Quit the database session
+		mysql> \q
+
+3. Try to connect to database:
+
+		sudo -u gitlab-runner -H mysql -u runner -p -D hello_world_test
+
+4. Configure your application to use the database:
+
+		Host: localhost
+		User: runner
+		Password: $password
+		Database: hello_world_test

doc/ci/services/postgres.md

+## Using PostgreSQL
+
+It's possible to use PostgreSQL database test your apps during builds.
+
+### Use PostgreSQL with Docker executor
+
+If you are using our Docker integration you basically have everything already.
+
+1. Add this to your `.gitlab-ci.yml`:
+
+		services:
+		- postgres
+
+		variables:
+		  # Configure postgres service (https://hub.docker.com/_/postgres/)
+		  POSTGRES_DB: hello_world_test
+		  POSTGRES_USER: postgres
+		  POSTGRES_PASSWORD: ""
+
+2. Configure your application to use the database:
+
+		Host: postgres
+		User: postgres
+		Password: postgres
+		Database: hello_world_test
+
+3. You can also use any other available on [DockerHub](https://hub.docker.com/_/postgres/). For example: `postgres:9.3`.
+
+Example: https://gitlab.com/gitlab-examples/postgres/blob/master/.gitlab-ci.yml
+
+### Use PostgreSQL with Shell executor
+
+It's possible to use PostgreSQL on manually configured servers that are using GitLab Runner with Shell executor.
+
+1. First install the PostgreSQL server:
+
+		sudo apt-get install -y postgresql postgresql-client libpq-dev
+
+2. Create an user:
+
+		# Install the database packages
+		sudo apt-get install -y postgresql postgresql-client libpq-dev
+
+		# Login to PostgreSQL
+		sudo -u postgres psql -d template1
+
+		# Create a user for runner
+		# Do not type the 'template1=#', this is part of the prompt
+		template1=# CREATE USER runner CREATEDB;
+
+		# Create the database & grant all privileges on database
+		template1=# CREATE DATABASE hello_world_test OWNER runner;
+
+		# Quit the database session
+		template1=# \q
+
+3. Try to connect to database:
+
+		# Try connecting to the new database with the new user
+		sudo -u gitlab-runner -H psql -d hello_world_test
+
+		# Quit the database session
+		hello_world_test> \q
+
+4. Configure your application to use the database:
+
+		Host: localhost
+		User: runner
+		Password:
+		Database: hello_world_test

doc/ci/services/redis.md

+## Using Redis
+
+It's possible to use Redis database test your apps during builds.
+
+### Use Redis with Docker executor
+
+If you are using our Docker integration you basically have everything already.
+
+1. Add this to your `.gitlab-ci.yml`:
+
+		services:
+		- redis
+
+2. Configure your application to use the database:
+
+		Host: redis
+
+3. You can also use any other available on [DockerHub](https://hub.docker.com/_/redis/). For example: `redis:2.6`.
+
+Example: https://gitlab.com/gitlab-examples/redis/blob/master/.gitlab-ci.yml
+
+### Use Redis with Shell executor
+
+It's possible to use Redis on manually configured servers that are using GitLab Runner with Shell executor.
+
+1. First install the Redis server:
+
+		sudo apt-get install redis-server
+
+2. Try to connect to the server:
+
+		# Try connecting the the Redis server
+		sudo -u gitlab-runner -H redis-cli
+
+		# Quit the session
+		127.0.0.1:6379> quit
+
+4. Configure your application to use the database:
+
+		Host: localhost

doc/ci/ssh_keys/README.md

+# Using SSH keys
+
+GitLab currently doesn't have built-in support for SSH keys in build environment.
+
+The SSH keys can be useful when:
+1. You want to checkout internal submodules,
+2. You want to download private packages using your package manager (ie. bundler),
+3. You want to deploy your app (ex. to Heroku or own server),
+4. You want to execute ssh commands from build environment on remote server,
+5. You want to rsync files from your build to remote server.
+
+If anyone of the above holds true, then you most likely need SSH key.
+
+There are two possibilities to add SSH keys to build environment.
+
+## Inject keys in your build environment
+The most widely supported is to inject SSH key into your build environment by extending your .gitlab-ci.yml.
+This is the universal solution which works with any type of executor (docker, shell, etc.).
+
+### How it works?
+1. We create a new SSH private key with [ssh-keygen](http://linux.die.net/man/1/ssh-keygen).
+2. We add the private key as the Secure Variable to project.
+3. We run the [ssh-agent](http://linux.die.net/man/1/ssh-agent) during build to load the private key.
+
+The example [.gitlab-ci.yml](https://gitlab.com/gitlab-examples/ssh-private-key/blob/master/.gitlab-ci.yml) looks like this.
+
+### Make it work?
+1. First, go to terminal and generate a new SSH key:
+```bash
+$ ssh-keygen -t rsa -f my_key
+
+Generating public/private rsa key pair.
+Enter passphrase (empty for no passphrase): 
+Enter same passphrase again: 
+Your identification has been saved in my_key.
+Your public key has been saved in my_key.pub.
+The key fingerprint is:
+SHA256:tBJEfyJUGTMNmPCiPg4UHywHs67MxlM2iEBAlI/W+TY fingeprint
+The key's randomart image is:
++---[RSA 2048]----+
+|=*. .o++*=       |
+|..=  +o..o.      |
+|.+++o + + .      |
+|+o*=.. + +       |
+|o+.=. . S        |
+|*.o .E .         |
+|o*o . .          |
+|.o..             |
+|  .              |
++----[SHA256]-----+
+```
+
+2. Create a new **Secure Variable** in your project settings on GitLab and name it: `SSH_PRIVATE_KEY`.
+
+3. Copy the content of `my_key` and paste it as a **Value** of **SSH_PRIVATE_KEY**.
+
+4. Next you need to modify your `.gitlab-ci.yml` and at the top of the file add:
+```
+before_script:
+# install ssh-agent (it is required for Docker, change apt-get to yum if you use CentOS-based image)
+- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
+
+# run ssh-agent (in build environment)
+- eval $(ssh-agent -s)
+
+# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
+- ssh-add <(echo "$SSH_PRIVATE_KEY")
+
+# for Docker builds disable host key checking, by adding that you are suspectible to man-in-the-middle attack
+- mkdir -p ~/.ssh
+- '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config`
+```
+
+5. Add the public key from `my_key.pub` to services that you want to have an access from build.
+
+6. If your builds are run using `shell` executor, you may need to login to server and execute the `ssh <address-of-my-server>` to store the fingerprint of remote server.
+
+## SSH keys when using Shell executor
+If use `shell`, not `docker` it can be easier to have the SSH key.
+
+We can generate the SSH key for the machine that holds `gitlab-runner` and use that key for all projects that are run on this machine.
+
+1. First, login to server that runs your builds.
+
+2. From terminal login as `gitlab-runner` user and generate the SSH private key:
+```bash
+$ ssh-keygen -t rsa
+Generating public/private rsa key pair.
+Enter passphrase (empty for no passphrase): 
+Enter same passphrase again: 
+Your identification has been saved in ~/.ssh/id_rsa.
+Your public key has been saved in ~/.ssh/id_rsa.pub.
+The key fingerprint is:
+SHA256:tBJEfyJUGTMNmPCiPg4UHywHs67MxlM2iEBAlI/W+TY fingeprint
+The key's randomart image is:
++---[RSA 2048]----+
+|=*. .o++*=       |
+|..=  +o..o.      |
+|.+++o + + .      |
+|+o*=.. + +       |
+|o+.=. . S        |
+|*.o .E .         |
+|o*o . .          |
+|.o..             |
+|  .              |
++----[SHA256]-----+
+```
+
+3. Add the public key from `~/.ssh/id_rsa.pub` to services that you want to have an access from build.
+
+4. Try to login for the first time and accept fingerprint:
+```bash
+ssh <address-of-my-server
+```