wendelin.core: Add part/recipe how to build it from git repository

There is now

    wendelin.core       and
    wendelin.core-dev

parts.

wendelin.core just installs released egg from pypi, and
wendelin.core-dev installs it from latest git version.

A reminder how to use wendelin.core-dev instead of wendelin.core:

if your buildout.cfg

    [buildout]
        # directly or indirectly
        extends = ..../wendelin.core/buildout.cfg

then at tails, do

    [buildout]
        parts -= wendelin.core
        parts += wendelin.core-dev

This way wendelin.core will be fetched from git repository.

component/wendelin.core/buildout.cfg

@@ -2,6 +2,31 @@
 parts =
   wendelin.core
 
+
+# wendelin.core installed from released egg from pypi
 [wendelin.core]
 recipe = zc.recipe.egg:custom
 egg = wendelin.core
+
+
+# wendelin.core installed from latest git version
+[wendelin.core-dev]
+recipe = zc.recipe.egg:develop
+setup = ${wendelin.core-repository-submoduleinit:location}
+
+
+[wendelin.core-repository]
+recipe  = slapos.recipe.build:gitclone
+repository = https://lab.nexedi.cn/nexedi/wendelin.core.git
+# dir is pretty name as top-level -dev recipe
+location = ${buildout:parts-directory}/wendelin.core-dev
+
+
+# TODO add `git clone --recursive` to slapos.recipe.build:gitclone
+#      and this way merge this -submoduleinit into -repository part
+[wendelin.core-repository-submoduleinit]
+recipe = plone.recipe.command
+command = cd "${wendelin.core-repository:location}" && git submodule update --init
+stop-on-error = true
+# propagate location of main repo
+location= ${wendelin.core-repository:location}

[Jérome Perrin]

@kirr, we usually extends component/git/buildout.cfg and use ${git:location}/bin/git instead of git, in case git is not always in the path, especially in webrunner. Also, you may want to set develop=true to slapos.recipe.build:gitclone to prevent the recipe from rm'ing the working copy. (be careful, it will rm the working copy just after setting develop=true)

[Kirill Smelkov]

@jerome, thanks for feedback. I've changed the recipe to use ${git:location}/bin/git where appropriate, but in turn I have two questions:

1. The build time for wendelin.core-dev raised up tremendeously, becuase before we can build it, we need to build git (and zlib, xz-utils, gdbm, perl, openssl, gettext, etc ... - as its dependencies). Is there maybe something that could be done to shorten modify / try cycle? (even with MAKEFLAGS=-j2, for wendelin.core-dev time ./bin/buildout is now ~ 30 minutes)

2. In the process of investingating this topic, I've discovered there is a mr.developer buildout extension[1] which allows to "activate" parts for development, i.e. in it's speak we only setup parts, without additional -dev parts, and then for each part we only provide where appropriate sources live, e.g. this way:

[buildout]
parts = wendelin.core

[wendelin.core]
recipe = zc.recipe.egg:custom
egg = wendelin.core

[sources]
wendelin.core = git https://lab.nexedi.cn/nexedi/wendelin.core.git

then after you have built the buildout, there is a bin/develop command and you can checkout parts, e.g. ./bin/develop co wendelin.core and it will place wendelin.core git checkout into src/ and use it. Also there is a way to just build afresh with only git checkouts, e.g. if one specifies buildout:auto-checkout = wendelin.core.

I'm also asking, because mr.developer is set as a core dependency of stack/slapos.cfg, so it looks like the idea is we should use it. Then why define -dev or -develop parts at all?

kirr@teco:~/src/wendelin/slapos/slapos$ git grep -- -develop |wc -l
14

Thanks beforehand, Kirill

P.S. I did not change to develop=true before we discuss mr.developer

[1] https://pypi.python.org/pypi/mr.developer
[2] https://lab.nexedi.cn/nexedi/slapos/blob/master/stack/slapos.cfg#L18

[Jérome Perrin]

1. Yes. Developer time lost waiting for compilation is a problem in SlapOS. We discussed symlinking or caching individuals parts, but this seem hard to implement.

2. I have no idea why we have mr.developer here. I already asked Vincent and others, but nobody knows. One problem with mr.developer is that it only supports python eggs, and ERP5 is not distributed as an egg. But I don't the early history of design decisions in SlapOS, maybe this can be found by asking on vifib mailing list or digging in the repository. I think I introduced the use zc.recipe.egg:develop, before that, most developers where cloning a git repo in part and using this as:

  [buildout]
  develop = ${this_git_checkout:location}

together with a hackish part that makes running buildout fail when ${this_git_checkout:location} is not present (it was using grep command), because the paths in develop has to be present when buildout starts. I thought this way of running the buildout twice was not acceptable, so I switched to zc.recipe.egg:develop that does not need to run twice and display scary traceback to the developer

[Jérome Perrin]

Since we were talking about it, there is also repository loss with zc.recipe.egg:custom as we often do. Reported upstream here: https://github.com/buildout/buildout/issues/159

[Kirill Smelkov]

@jerome, thanks for mentioned it.

As to your original develop = True note, after using slapos for a while, I made the following conclusion:

1. it is logical for slapos/buildout to manage part's source: to update it, delete when part goes away, etc, because buildout logic is that it manages parts.
   So to me it is logical to have develop = False by default - buildout just clones git repo at appropriate revision, and if/when revision changes just updates it appropriately.

2. it is handy to have develop = True for developers.

And what is more important is:

3. developers should be well aware of buildout/slapos logic to manage parts completly.
4. developers should be well aware to use develop = True for developing.

and even if there is not develop = True, before updating slapos.recipe.build:gitclone can and should do

git stash create

so that worktree changes, if any, are not thrown away, but are leaved there in a stash, so that people can at lease recover their work after "unexpected" update.

Does it sounds ok?

[Jérome Perrin]

In the past, there have been several bugs leading to repository deletion. Most of the time it came from the fact that buildout removes installed things on update (see buildout_installed in .installed.cfg). In the case of this bug, the source code directory is removed eventhough it is outside of parts, so I agree this is rather major. Even if we implement something like you suggest in the gitclone recipe, buildout will remove the directory before the recipe code is executed.

That said, I tend to agree with your suggestion, but since we only update SR inplace during development (for production, we install another SR and switch instances to the new SR, to prevent downtime), after some discussion with Cédric DSM we concluded that it was OK to assume that the developer will fix if something goes wrong after he clicks on build button, because what goes wrong is due to the changes he made to the repository.