1. 05 Nov, 2015 1 commit
  2. 04 Nov, 2015 11 commits
    • Kirill Smelkov's avatar
      X Connect nginx to unicorn · 34b5fee0
      Kirill Smelkov authored
      34b5fee0
    • Kirill Smelkov's avatar
      X Unicorn starts to start · b4f1ccb0
      Kirill Smelkov authored
      b4f1ccb0
    • Kirill Smelkov's avatar
      X Stub unicorn service · 845fb66b
      Kirill Smelkov authored
      845fb66b
    • Kirill Smelkov's avatar
      X Stub gitlab-workhorse service · 68ef76d1
      Kirill Smelkov authored
      68ef76d1
    • Kirill Smelkov's avatar
      . · 509818df
      Kirill Smelkov authored
      509818df
    • Kirill Smelkov's avatar
      Merge branch 'master' into x/gitlab · 1cd49f99
      Kirill Smelkov authored
      * master:
        slapos/recipe/redis: Add support for UNIX sockets
        slapos/recipe/redis/promise: Don't create connection pool explicitly
        redis: v↑  (2.8.23)
        ruby: v↑ ruby2.1  (2.1.7)
        component/postgresql: v↑ postgresql92  (9.2.14)
        component/postgresql: Factor out common build stuff to postgresql-common
        software/postgresql: Don't build postgresql twice
        version up: HAProxy 1.5.15
      1cd49f99
    • Kirill Smelkov's avatar
      slapos/recipe/redis: Add support for UNIX sockets · cbbfd405
      Kirill Smelkov authored
      It is well known that UNIX sockets are faster than TCP over loopback.
      
      E.g. on my machine according to lmbench[1] they have ~ 2 times
      lower latency and ~ 2-3 times more throughput compared to TCP over
      loopback:
      
          *Local* Communication latencies in microseconds - smaller is better
          ---------------------------------------------------------------------
          Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                                  ctxsw       UNIX         UDP         TCP conn
          --------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
          teco      Linux 4.2.0-1  13.8  29.2 26.8  45.0  47.9  48.5  55.5  45.
      
          *Local* Communication bandwidths in MB/s - bigger is better
          -----------------------------------------------------------------------------
          Host                OS  Pipe AF    TCP  File   Mmap  Bcopy  Bcopy  Mem   Mem
                                       UNIX      reread reread (libc) (hand) read write
          --------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
          teco      Linux 4.2.0-1 1084 4353 1493 2329.1 3720.7 1613.8 1109.2 3402 1404.
      
      The same ratio holds for our std shuttle servers.
      
      API to work with unix sockets is essentially the same as with TCP/UDP.
      Because of that it is easy to support both TCP and UNIX socket in one
      software, and this way a lot of software support unix sockets out of the
      box, including Redis.
      
      Because of lower latencies and higher throughput, for performance
      reasons, it makes sense to interconnect services on one machine via unix
      sockets and talk via TCP only to outside world.
      
      Here we add support for unix sockets to Redis recipe.
      
      [1] http://www.bitmover.com/lmbench/
      
      /reviewed-by @kazuhiko  (on !27)
      /cc @alain.takoudjou, @jerome, @vpelletier
      cbbfd405
    • Kirill Smelkov's avatar
      slapos/recipe/redis/promise: Don't create connection pool explicitly · 442866bc
      Kirill Smelkov authored
      Because redis.Redis(...) ctor creates connection pool on initialization
      and we can rely on it.
      
      Another reason: Redis ctor (in form of StrictRedis.__init__()) has logic
      how to process arguments and does selecting - either it is TCP (`host` and
      `port` args), or UNIX socket (`unix_socket_path` arg):
      
          https://lab.nexedi.com/nexedi/slapos/blob/95dbb5b2/slapos/recipe/redis/MyRedis2410.py#L560
      
      Since we are going to introduce unix socket support to redis recipe in
      the next patch, and don't want to duplicate StrictRedis.__init__() logic
      in promise code, let's refactor promise to delegate argument processing
      logic to Redis.
      
      /reviewed-by @kazuhiko  (on !27)
      /cc @alain.takoudjou
      442866bc
    • Kirill Smelkov's avatar
      redis: v↑ (2.8.23) · 9b3cfff4
      Kirill Smelkov authored
      - update Redis software to latest upstream in 2.8.* series (which now
        supports IPv6 out of the box);
      
      - update Redis instance template to the one from 2.8.23 and re-merge our
        templating changes to it (file/dir locations, port and binding, master
        password).
      
        The whole diff to pristine 2.8.23 redis conf is now this:
      
        diff --git a/.../redis-2.8.23/redis.conf b/slapos/recipe/redis/template/redis.conf.in
        index 870959f..2895539 100644
        --- a/.../redis-2.8.23/redis.conf
        +++ b/slapos/recipe/redis/template/redis.conf.in
        @@ -46 +46 @@ daemonize no
        -pidfile /var/run/redis.pid
        +pidfile %(pid_file)s
        @@ -50 +50 @@ pidfile /var/run/redis.pid
        -port 6379
        +port %(port)s
        @@ -69,0 +70 @@ tcp-backlog 511
        +bind %(ipv6)s
        @@ -108 +109 @@ loglevel notice
        -logfile ""
        +logfile %(log_file)s
        @@ -174 +175 @@ rdbcompression yes
        -# hit to pay (around 10%) when saving and loading RDB files, so you can disable it
        +# hit to pay (around 10%%) when saving and loading RDB files, so you can disable it
        @@ -192 +193 @@ dbfilename dump.rdb
        -dir ./
        +dir %(server_dir)s
        @@ -217 +218 @@ dir ./
        -# masterauth <master-password>
        +%(master_passwd)s
      
      NOTE There are test failures for almost all Redis versions when machine
      have not small amount of CPUs:
      
          https://github.com/antirez/redis/issues/2715#issuecomment-151608948
      
      Because the failure is in replication test, and so far we do not use
      replication, and there is no feedback from upstream author to handle
      this (for 7 days for my detailed report, and for ~ 3 months for this
      issue in general), we can just disable replication test as a temporary
      solution.  ( to handle remote patches with md5 hash easily the building
      recipe is changed to slapos.recipe.cmmi )
      
      NOTE Redis updated to 2.8 version because GitLab uses this series.
      If/when we need more recent one we can add [redis30] in addition to
      [redis28].
      
      /reviewed-by @kazuhiko  (on !27 and on !26)
      /cc @alain.takoudjou, @jerome
      9b3cfff4
    • Kirill Smelkov's avatar
      ruby: v↑ ruby2.1 (2.1.7) · e393ada8
      Kirill Smelkov authored
      Ruby 2.1.7 contain security and other bugfixes:
      
          https://www.ruby-lang.org/en/news/2015/08/18/ruby-2-1-7-released/
      
      /reviewed-by @kazuhiko  (on nexedi/slapos!28)
      e393ada8
    • Kirill Smelkov's avatar
      X Sync with redis-unixsock · 1b480009
      Kirill Smelkov authored
      1b480009
  3. 03 Nov, 2015 10 commits
  4. 02 Nov, 2015 4 commits
  5. 30 Oct, 2015 7 commits
    • Kirill Smelkov's avatar
      helloworld & helloweb: Go version · 911a8d8c
      Kirill Smelkov authored
      We added support for Go in the previous patch. Let's now illustrate how
      to use the toolchain and build simple service based on it.
      
      /cc @gabriel, @Camata
      911a8d8c
    • Kirill Smelkov's avatar
      Go component · e0d4c7af
      Kirill Smelkov authored
      Add support for Go language in the form of providing component for Go
      toolchain and stdlib.
      
      Current Go stable series is 1.5.* but to bootstrap it one need to have
      Go 1.4.* available, which itself needs C compiler to bootstrap.
      
      We need Go support for upcoming GitLab SR, where one server is written
      in this language:
      
          https://gitlab.com/gitlab-org/gitlab-workhorse
      
      /cc @gabriel, @Camata
      e0d4c7af
    • Kirill Smelkov's avatar
      helloworld & helloweb: Ruby version · dbdaa3a7
      Kirill Smelkov authored
      To illustrate how to build Ruby software, let's add helloweb-ruby
      component and integrate it into helloworld SR as a service.
      
      helloweb-ruby installation is a bit tricky, because we do not have it
      initially as a released gem, and have to build it with Bundler by hand.
      
      Changes for integration into helloworld services infrastructure are
      small, thanks for the "kind" rework we did in previous patch.
      
      /cc @kazuhiko
      dbdaa3a7
    • Kirill Smelkov's avatar
      helloworld & helloweb: Prepare to show different kinds of helloweb to the world · acc5f427
      Kirill Smelkov authored
      Prepare to have several helloweb programs each written in its own
      language, and installed into bin/ as helloworld-<language> and exposed
      as separate service to the web on its own port.
      
      For component/helloweb we just rename installed script and section.
      
      For software/helloworld we split helloweb & helloweb-promise section
      into base and generate requested helloweb-<kinds> & friends via jinja2
      programming; the same for exposing url for each kind.
      
      So far it is only preparatory changes for new kinds - i.e. instance code
      now supports it, but the only kind so far is still python.
      
      /cc @jerome
      acc5f427
    • Kirill Smelkov's avatar
      helloworld: Convert instance.cfg.in to jinja2 · 6506bb67
      Kirill Smelkov authored
      Jinja2 is currently the preferred way to do instance templating, because
      it has control structures.
      
      We'll need control structures in the following patches, but even without
      it, it is better to show "how to" do instances the preferred way.
      
      /cc @jerome
      6506bb67
    • Kirill Smelkov's avatar
      helloworld: Move helloweb program to component/helloweb · 6fcf28cf
      Kirill Smelkov authored
      This
      
      - better illustrates what components do (provide building recipes for
        programs) and what software do (gather components and provide recipes
        for instances).
      
      - illustrates how to build the software properly - usually we build
        eggs or scripts with zc.recipe.egg:* , not generate python programs via
        jinja2 templating. Because when installing via egg, it is possible to
        import other installed eggs relatively straghtforward, and for jinja2
        way it is hard to do.
      
      The helloweb program itself is moved into newly introduced helloweb.git
      repository and, as almost all other software, becomes separate from
      slapos.git .
      
      /cc @jerome
      6fcf28cf
    • Kirill Smelkov's avatar
      helloworld: Unify naming for internal "hello web" service · 4a4e3e32
      Kirill Smelkov authored
      Since 7cd2f953 (helloworld: Turn it into real web-service) helloworld SR
      implements real web-service which talks to outside world with the help
      of shipped-with-it "hello web" program.
      
      Buildout sections which create helloweb service and promise were named
      based on helloworld prefix though, which is not very consistent. Thus the
      change for such sections to the same name as the "hello web" program uses.
      
      Also for style consistency the "hello web" program is renamed from
      hello-web to helloweb, as this naming style is usually used for
      helloworld.{c,py.rb,go,etc}
      
      So the final naming is:
      
          helloweb
      
      for both program and related sections.
      4a4e3e32
  6. 29 Oct, 2015 2 commits
  7. 28 Oct, 2015 5 commits
    • Kirill Smelkov's avatar
      Merge branch 'next' into x/gitlab · 1c765a05
      Kirill Smelkov authored
      * next:
        X redis: v↑  (2.8.23)
        component/postgresql: v↑ postgresql92  (9.2.14)
        component/postgresql: Factor out common build stuff to postgresql-common
        software/postgresql: Don't build postgresql twice
      1c765a05
    • Kirill Smelkov's avatar
      X Sync redis-vup · 3eec4a11
      Kirill Smelkov authored
      3eec4a11
    • Kirill Smelkov's avatar
      Merge branch 'redis-vup' into next · e3367084
      Kirill Smelkov authored
      * redis-vup:
        X redis: v↑  (2.8.23)
      e3367084
    • Kirill Smelkov's avatar
      Merge branch 'pg-vup' into next · 6ca54762
      Kirill Smelkov authored
      * pg-vup:
        component/postgresql: v↑ postgresql92  (9.2.14)
        component/postgresql: Factor out common build stuff to postgresql-common
        software/postgresql: Don't build postgresql twice
      6ca54762
    • Kirill Smelkov's avatar
      X redis: v↑ (2.8.23) · 2d5c2a5e
      Kirill Smelkov authored
      - update Redis software to latest upstream in 2.8.* series (which now
        supports IPv6 out of the box);
      
      - update Redis instance template to the one from 2.8.23 and re-merge our
        templating changes to it (file/dir locations, port and binding, master
        password).
      
        The whole diff to pristine 2.8.23 redis conf is now this:
      
        diff --git a/.../redis-2.8.23/redis.conf b/slapos/recipe/redis/template/redis.conf.in
        index 870959f..2895539 100644
        --- a/.../redis-2.8.23/redis.conf
        +++ b/slapos/recipe/redis/template/redis.conf.in
        @@ -46 +46 @@ daemonize no
        -pidfile /var/run/redis.pid
        +pidfile %(pid_file)s
        @@ -50 +50 @@ pidfile /var/run/redis.pid
        -port 6379
        +port %(port)s
        @@ -69,0 +70 @@ tcp-backlog 511
        +bind %(ipv6)s
        @@ -108 +109 @@ loglevel notice
        -logfile ""
        +logfile %(log_file)s
        @@ -174 +175 @@ rdbcompression yes
        -# hit to pay (around 10%) when saving and loading RDB files, so you can disable it
        +# hit to pay (around 10%%) when saving and loading RDB files, so you can disable it
        @@ -192 +193 @@ dbfilename dump.rdb
        -dir ./
        +dir %(server_dir)s
        @@ -217 +218 @@ dir ./
        -# masterauth <master-password>
        +%(master_passwd)s
      
      XXX tests failure https://github.com/antirez/redis/issues/2715
      
      XXX 2.8 because GitLab uses this series
      
      /cc @alain.takoudjou
      2d5c2a5e