Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
slapos
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Lukas Niegsch
slapos
Commits
bd1049cd
Commit
bd1049cd
authored
8 years ago
by
Rafael Monnerat
Browse files
Options
Browse Files
Download
Plain Diff
Update Release Candidate
parents
a69c0bc9
31a69ae3
Changes
30
Expand all
Hide whitespace changes
Inline
Side-by-side
Showing
30 changed files
with
1064 additions
and
296 deletions
+1064
-296
component/curl/buildout.cfg
component/curl/buildout.cfg
+2
-2
component/libexpat/CVE-2015-1283-refix.patch
component/libexpat/CVE-2015-1283-refix.patch
+37
-0
component/libexpat/CVE-2015-1283.patch
component/libexpat/CVE-2015-1283.patch
+87
-0
component/libexpat/CVE-2016-0718-v2-2-1.patch
component/libexpat/CVE-2016-0718-v2-2-1.patch
+755
-0
component/libexpat/buildout.cfg
component/libexpat/buildout.cfg
+10
-0
setup.py
setup.py
+1
-1
slapos/recipe/librecipe/genericslap.py
slapos/recipe/librecipe/genericslap.py
+12
-3
slapos/recipe/wrapper.py
slapos/recipe/wrapper.py
+8
-1
software/agent/software.cfg
software/agent/software.cfg
+1
-1
software/apache-frontend/common.cfg
software/apache-frontend/common.cfg
+4
-4
software/apache-frontend/instance-apache-frontend.cfg
software/apache-frontend/instance-apache-frontend.cfg
+12
-51
software/apache-frontend/instance-apache-replicate.cfg.in
software/apache-frontend/instance-apache-replicate.cfg.in
+3
-0
software/apache-frontend/instance.cfg
software/apache-frontend/instance.cfg
+1
-3
software/apache-frontend/software.cfg
software/apache-frontend/software.cfg
+1
-1
software/apache-frontend/templates/apache-custom-slave-list.cfg.in
...apache-frontend/templates/apache-custom-slave-list.cfg.in
+2
-0
software/apache-frontend/templates/apache-default-slave-list.cfg.in
...pache-frontend/templates/apache-default-slave-list.cfg.in
+0
-143
software/kvm/development.cfg
software/kvm/development.cfg
+1
-2
software/kvm/software.cfg
software/kvm/software.cfg
+1
-1
software/neoppod/software-common.cfg
software/neoppod/software-common.cfg
+1
-1
software/re6stnet/software.cfg
software/re6stnet/software.cfg
+1
-1
software/slaprunner/development.cfg
software/slaprunner/development.cfg
+4
-4
software/slaprunner/software.cfg
software/slaprunner/software.cfg
+1
-1
stack/erp5/buildout.cfg
stack/erp5/buildout.cfg
+3
-3
stack/monitor/buildout.cfg
stack/monitor/buildout.cfg
+5
-5
stack/monitor/instance-monitor.cfg.jinja2.in
stack/monitor/instance-monitor.cfg.jinja2.in
+2
-1
stack/monitor/scripts/collect.py
stack/monitor/scripts/collect.py
+47
-25
stack/monitor/scripts/globalstate.py
stack/monitor/scripts/globalstate.py
+1
-0
stack/monitor/scripts/monitor.py
stack/monitor/scripts/monitor.py
+16
-17
stack/monitor/scripts/run-promise.py
stack/monitor/scripts/run-promise.py
+41
-21
stack/slapos.cfg
stack/slapos.cfg
+4
-4
No files found.
component/curl/buildout.cfg
View file @
bd1049cd
...
@@ -12,8 +12,8 @@ parts =
...
@@ -12,8 +12,8 @@ parts =
[curl]
[curl]
recipe = slapos.recipe.cmmi
recipe = slapos.recipe.cmmi
url = http://curl.haxx.se/download/curl-7.4
7
.0.tar.bz2
url = http://curl.haxx.se/download/curl-7.4
9
.0.tar.bz2
md5sum =
85c58a00412476993050cb242a3f365d
md5sum =
7416aaff4a9210b43edda7615ffa4169
configure-options =
configure-options =
--disable-static
--disable-static
--disable-ldap
--disable-ldap
...
...
This diff is collapsed.
Click to expand it.
component/libexpat/CVE-2015-1283-refix.patch
0 → 100644
View file @
bd1049cd
From 29a11774d8ebbafe8418b4a5ffb4cc1160b194a1 Mon Sep 17 00:00:00 2001
From: Pascal Cuoq <cuoq@trust-in-soft.com>
Date: Sun, 15 May 2016 09:05:46 +0200
Subject: [PATCH] Avoid relying on undefined behavior in CVE-2015-1283 fix. It
does not really work: https://godbolt.org/g/Zl8gdF
---
expat/lib/xmlparse.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/lib/xmlparse.c b/lib/xmlparse.c
index 13e080d..cdb12ef 100644
--- a/lib/xmlparse.c
+++ b/lib/xmlparse.c
@@ -1695,7 +1695,8 @@
XML_GetBuffer(XML_Parser parser, int len
}
if (len > bufferLim - bufferEnd) {
- int neededSize = len + (int)(bufferEnd - bufferPtr);
+ /* Do not invoke signed arithmetic overflow: */
+ int neededSize = (int) ((unsigned)len + (unsigned)(bufferEnd - bufferPtr));
/* BEGIN MOZILLA CHANGE (sanity check neededSize) */
if (neededSize < 0) {
errorCode = XML_ERROR_NO_MEMORY;
@@ -1729,7 +1730,8 @@
XML_GetBuffer(XML_Parser parser, int len
if (bufferSize == 0)
bufferSize = INIT_BUFFER_SIZE;
do {
- bufferSize *= 2;
+ /* Do not invoke signed arithmetic overflow: */
+ bufferSize = (int) (2U * (unsigned) bufferSize);
/* BEGIN MOZILLA CHANGE (prevent infinite loop on overflow) */
} while (bufferSize < neededSize && bufferSize > 0);
/* END MOZILLA CHANGE */
--
2.8.2
This diff is collapsed.
Click to expand it.
component/libexpat/CVE-2015-1283.patch
0 → 100644
View file @
bd1049cd
Description: fix multiple integer overflows in the XML_GetBuffer function
Multiple integer overflows in the XML_GetBuffer function in Expat through
2.1.0, as used in Google Chrome before 44.0.2403.89 and other products,
allow remote attackers to cause a denial of service (heap-based buffer
overflow) or possibly have unspecified other impact via crafted XML data,
a related issue to CVE-2015-2716.
Origin: Mozilla, https://hg.mozilla.org/releases/mozilla-esr31/rev/2f3e78643f5c
Author: Eric Rahm <erahm@mozilla.com>
Forwarded: not-needed
Last-Update: 2015-07-24
diff --git a/lib/xmlparse.c b/lib/xmlparse.c
--- a/lib/xmlparse.c
+++ b/lib/xmlparse.c
@@ -1673,29 +1673,40 @@
XML_ParseBuffer(XML_Parser parser, int l
XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position);
positionPtr = bufferPtr;
return result;
}
void * XMLCALL
XML_GetBuffer(XML_Parser parser, int len)
{
+/* BEGIN MOZILLA CHANGE (sanity check len) */
+ if (len < 0) {
+ errorCode = XML_ERROR_NO_MEMORY;
+ return NULL;
+ }
+/* END MOZILLA CHANGE */
switch (ps_parsing) {
case XML_SUSPENDED:
errorCode = XML_ERROR_SUSPENDED;
return NULL;
case XML_FINISHED:
errorCode = XML_ERROR_FINISHED;
return NULL;
default: ;
}
if (len > bufferLim - bufferEnd) {
- /* FIXME avoid integer overflow */
int neededSize = len + (int)(bufferEnd - bufferPtr);
+/* BEGIN MOZILLA CHANGE (sanity check neededSize) */
+ if (neededSize < 0) {
+ errorCode = XML_ERROR_NO_MEMORY;
+ return NULL;
+ }
+/* END MOZILLA CHANGE */
#ifdef XML_CONTEXT_BYTES
int keep = (int)(bufferPtr - buffer);
if (keep > XML_CONTEXT_BYTES)
keep = XML_CONTEXT_BYTES;
neededSize += keep;
#endif /* defined XML_CONTEXT_BYTES */
if (neededSize <= bufferLim - buffer) {
@@ -1714,17 +1725,25 @@
XML_GetBuffer(XML_Parser parser, int len
}
else {
char *newBuf;
int bufferSize = (int)(bufferLim - bufferPtr);
if (bufferSize == 0)
bufferSize = INIT_BUFFER_SIZE;
do {
bufferSize *= 2;
- } while (bufferSize < neededSize);
+/* BEGIN MOZILLA CHANGE (prevent infinite loop on overflow) */
+ } while (bufferSize < neededSize && bufferSize > 0);
+/* END MOZILLA CHANGE */
+/* BEGIN MOZILLA CHANGE (sanity check bufferSize) */
+ if (bufferSize <= 0) {
+ errorCode = XML_ERROR_NO_MEMORY;
+ return NULL;
+ }
+/* END MOZILLA CHANGE */
newBuf = (char *)MALLOC(bufferSize);
if (newBuf == 0) {
errorCode = XML_ERROR_NO_MEMORY;
return NULL;
}
bufferLim = newBuf + bufferSize;
#ifdef XML_CONTEXT_BYTES
if (bufferPtr) {
This diff is collapsed.
Click to expand it.
component/libexpat/CVE-2016-0718-v2-2-1.patch
0 → 100644
View file @
bd1049cd
This diff is collapsed.
Click to expand it.
component/libexpat/buildout.cfg
View file @
bd1049cd
...
@@ -2,9 +2,19 @@
...
@@ -2,9 +2,19 @@
parts =
parts =
libexpat
libexpat
extends =
../patch/buildout.cfg
[libexpat]
[libexpat]
recipe = slapos.recipe.cmmi
recipe = slapos.recipe.cmmi
url = http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz
url = http://downloads.sourceforge.net/project/expat/expat/2.1.0/expat-2.1.0.tar.gz
md5sum = dd7dab7a5fea97d2a6a43f511449b7cd
md5sum = dd7dab7a5fea97d2a6a43f511449b7cd
configure-options =
configure-options =
--disable-static
--disable-static
patch-options = -p1
patches =
${:_profile_base_location_}/CVE-2015-1283.patch#44b31d7377035591f37ad94a31a8042b
${:_profile_base_location_}/CVE-2015-1283-refix.patch#47382fe30c9a49724c626cef793ef382
${:_profile_base_location_}/CVE-2016-0718-v2-2-1.patch#4ee57e7a052ada99f2e11fa21a229727
environment =
PATH=${patch:location}/bin:%(PATH)s
This diff is collapsed.
Click to expand it.
setup.py
View file @
bd1049cd
...
@@ -28,7 +28,7 @@ from setuptools import setup, find_packages
...
@@ -28,7 +28,7 @@ from setuptools import setup, find_packages
import
glob
import
glob
import
os
import
os
version
=
'1.0.30'
version
=
'1.0.3
1.dev
0'
name
=
'slapos.cookbook'
name
=
'slapos.cookbook'
long_description
=
open
(
"README.txt"
).
read
()
+
"
\
n
"
+
\
long_description
=
open
(
"README.txt"
).
read
()
+
"
\
n
"
+
\
open
(
"CHANGES.txt"
).
read
()
+
"
\
n
"
open
(
"CHANGES.txt"
).
read
()
+
"
\
n
"
...
...
This diff is collapsed.
Click to expand it.
slapos/recipe/librecipe/genericslap.py
View file @
bd1049cd
...
@@ -29,6 +29,8 @@ import time
...
@@ -29,6 +29,8 @@ import time
from
generic
import
GenericBaseRecipe
from
generic
import
GenericBaseRecipe
CONNECTION_CACHE
=
{}
class
GenericSlapRecipe
(
GenericBaseRecipe
):
class
GenericSlapRecipe
(
GenericBaseRecipe
):
"""Base class for all slap.recipe.* needing SLAP informations like instance
"""Base class for all slap.recipe.* needing SLAP informations like instance
parameters.
parameters.
...
@@ -50,11 +52,18 @@ class GenericSlapRecipe(GenericBaseRecipe):
...
@@ -50,11 +52,18 @@ class GenericSlapRecipe(GenericBaseRecipe):
self
.
cert_file
=
slap_connection
.
get
(
'cert-file'
)
self
.
cert_file
=
slap_connection
.
get
(
'cert-file'
)
def
install
(
self
):
def
install
(
self
):
cache_key
=
"%s_%s"
%
(
self
.
computer_id
,
self
.
computer_partition_id
)
self
.
computer_partition
=
CONNECTION_CACHE
.
get
(
cache_key
,
None
)
self
.
slap
.
initializeConnection
(
self
.
server_url
,
self
.
key_file
,
self
.
slap
.
initializeConnection
(
self
.
server_url
,
self
.
key_file
,
self
.
cert_file
)
self
.
cert_file
)
self
.
computer_partition
=
self
.
slap
.
registerComputerPartition
(
if
self
.
computer_partition
is
None
:
self
.
computer_id
,
self
.
computer_partition
=
self
.
slap
.
registerComputerPartition
(
self
.
computer_partition_id
)
self
.
computer_id
,
self
.
computer_partition_id
)
CONNECTION_CACHE
[
cache_key
]
=
self
.
computer_partition
self
.
request
=
self
.
computer_partition
.
request
self
.
request
=
self
.
computer_partition
.
request
self
.
setConnectionDict
=
self
.
computer_partition
.
setConnectionDict
self
.
setConnectionDict
=
self
.
computer_partition
.
setConnectionDict
self
.
parameter_dict
=
self
.
computer_partition
.
getInstanceParameterDict
()
self
.
parameter_dict
=
self
.
computer_partition
.
getInstanceParameterDict
()
...
...
This diff is collapsed.
Click to expand it.
slapos/recipe/wrapper.py
View file @
bd1049cd
...
@@ -26,6 +26,7 @@
...
@@ -26,6 +26,7 @@
##############################################################################
##############################################################################
import
shlex
import
shlex
import
os
from
slapos.recipe.librecipe
import
GenericBaseRecipe
from
slapos.recipe.librecipe
import
GenericBaseRecipe
...
@@ -57,8 +58,14 @@ class Recipe(GenericBaseRecipe):
...
@@ -57,8 +58,14 @@ class Recipe(GenericBaseRecipe):
# We create a python script and a wrapper around the python
# We create a python script and a wrapper around the python
# script because the python script might have a too long #! line
# script because the python script might have a too long #! line
if
os
.
path
.
exists
(
os
.
path
.
join
(
self
.
buildout
[
'buildout'
][
'directory'
],
"bin"
)):
base_script_path
=
os
.
path
.
join
(
self
.
buildout
[
'buildout'
][
'directory'
],
"bin/"
+
wrapper_path
.
split
(
"/"
)[
-
1
])
else
:
base_script_path
=
os
.
path
.
join
(
self
.
buildout
[
'buildout'
][
'directory'
],
wrapper_path
.
split
(
"/"
)[
-
1
])
python_script
=
self
.
createPythonScript
(
python_script
=
self
.
createPythonScript
(
wrapper_path
+
'.py'
,
base_script_path
+
'.py'
,
'slapos.recipe.librecipe.execute.generic_exec'
,
'slapos.recipe.librecipe.execute.generic_exec'
,
(
command_line
,
wait_files
,
environment
,),
)
(
command_line
,
wait_files
,
environment
,),
)
return
[
python_script
,
self
.
createWrapper
(
return
[
python_script
,
self
.
createWrapper
(
...
...
This diff is collapsed.
Click to expand it.
software/agent/software.cfg
View file @
bd1049cd
...
@@ -65,7 +65,7 @@ feedparser = 5.2.1
...
@@ -65,7 +65,7 @@ feedparser = 5.2.1
# Required by:
# Required by:
# slapos.toolbox==0.55
# slapos.toolbox==0.55
lockfile = 0.1
0
.2
lockfile = 0.1
2
.2
# Required by:
# Required by:
# slapos.toolbox==0.55
# slapos.toolbox==0.55
...
...
This diff is collapsed.
Click to expand it.
software/apache-frontend/common.cfg
View file @
bd1049cd
...
@@ -60,27 +60,27 @@ command =
...
@@ -60,27 +60,27 @@ command =
[template]
[template]
recipe = slapos.recipe.template
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance.cfg
url = ${:_profile_base_location_}/instance.cfg
md5sum =
51752d0e4eae8c761750117c44983594
md5sum =
f686f765e55d1dce2e55a400f0714b3e
output = ${buildout:directory}/template.cfg
output = ${buildout:directory}/template.cfg
mode = 0644
mode = 0644
[template-apache-frontend]
[template-apache-frontend]
recipe = slapos.recipe.template
recipe = slapos.recipe.template
url = ${:_profile_base_location_}/instance-apache-frontend.cfg
url = ${:_profile_base_location_}/instance-apache-frontend.cfg
md5sum =
07b6f864b46cb2f7b4b8aab71da95d13
md5sum =
cc64151e4fe953f15f7ea8cf20718d84
output = ${buildout:directory}/template-apache-frontend.cfg
output = ${buildout:directory}/template-apache-frontend.cfg
mode = 0644
mode = 0644
[template-apache-replicate]
[template-apache-replicate]
recipe = slapos.recipe.build:download
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/instance-apache-replicate.cfg.in
url = ${:_profile_base_location_}/instance-apache-replicate.cfg.in
md5sum =
a05bc0f925902ddf0b4d12146b10c9e1
md5sum =
9b17c835bcd927269cf510bf612f5985
mode = 0644
mode = 0644
[template-slave-list]
[template-slave-list]
recipe = slapos.recipe.build:download
recipe = slapos.recipe.build:download
url = ${:_profile_base_location_}/templates/apache-custom-slave-list.cfg.in
url = ${:_profile_base_location_}/templates/apache-custom-slave-list.cfg.in
md5sum =
7f010ef92fdea8fb4de9bc93c76604ba
md5sum =
8b278b34a4fb063ba94e10186f725fcb
mode = 640
mode = 640
[template-slave-configuration]
[template-slave-configuration]
...
...
This diff is collapsed.
Click to expand it.
software/apache-frontend/instance-apache-frontend.cfg
View file @
bd1049cd
...
@@ -73,9 +73,8 @@ ca-dir = $${:srv}/ssl
...
@@ -73,9 +73,8 @@ ca-dir = $${:srv}/ssl
[switch-apache-softwaretype]
[switch-apache-softwaretype]
recipe = slapos.cookbook:softwaretype
recipe = slapos.cookbook:softwaretype
single-default = $${dynamic-
default
-template-slave-list:rendered}
single-default = $${dynamic-
custom-personal
-template-slave-list:rendered}
single-custom-personal = $${dynamic-custom-personal-template-slave-list:rendered}
single-custom-personal = $${dynamic-custom-personal-template-slave-list:rendered}
single-custom-group = $${dynamic-custom-group-template-slave-list:rendered}
[instance-parameter]
[instance-parameter]
# Fetches parameters defined in SlapOS Master for this instance.
# Fetches parameters defined in SlapOS Master for this instance.
...
@@ -164,53 +163,6 @@ extra-context =
...
@@ -164,53 +163,6 @@ extra-context =
section apache_configuration apache-configuration
section apache_configuration apache-configuration
key monitor_base_url monitor-instance-parameter:monitor-base-url
key monitor_base_url monitor-instance-parameter:monitor-base-url
[dynamic-custom-group-template-slave-list]
< = jinja2-template-base
template = ${template-custom-slave-list:target}
filename = custom-group-instance-slave-list.cfg
extensions = jinja2.ext.do
extra-context =
key apache_configuration_directory apache-directory:slave-configuration
key domain instance-parameter:configuration.domain
key http_port instance-parameter:configuration.plain_http_port
key https_port instance-parameter:configuration.port
key public_ipv4 instance-parameter:configuration.public-ipv4
key slave_instance_list instance-parameter:slave-instance-list
key extra_slave_instance_list instance-parameter:configuration.extra_slave_instance_list
key rewrite_cached_configuration apache-configuration:cached-rewrite-file
key custom_ssl_directory apache-directory:vh-ssl
key template_slave_configuration dynamic-virtualhost-template-slave:rendered
key apache_log_directory apache-directory:slave-log
key local_ipv4 instance-parameter:ipv4-random
key cache_port apache-configuration:cache-port
raw empty_template ${template-empty:target}
raw template_rewrite_cached ${template-rewrite-cached:target}
raw software_type single-custom-group
[dynamic-default-template-slave-list]
< = jinja2-template-base
template = ${template-custom-slave-list:target}
filename = default-instance-slave-list.cfg
extensions = jinja2.ext.do
extra-context =
key apache_configuration_directory apache-directory:slave-configuration
key domain instance-parameter:configuration.domain
key http_port instance-parameter:configuration.plain_http_port
key https_port instance-parameter:configuration.port
key public_ipv4 instance-parameter:configuration.public-ipv4
key slave_instance_list instance-parameter:slave-instance-list
key extra_slave_instance_list instance-parameter:configuration.extra_slave_instance_list
key rewrite_cached_configuration apache-configuration:cached-rewrite-file
key custom_ssl_directory apache-directory:vh-ssl
key apache_log_directory apache-directory:slave-log
key local_ipv4 instance-parameter:ipv4-random
key cache_port apache-configuration:cache-port
raw template_slave_configuration ${template-default-slave-virtualhost:target}
raw empty_template ${template-empty:target}
raw template_rewrite_cached ${template-rewrite-cached:target}
raw software_type single-default
# XXXX Hack to allow two software types
[dynamic-virtualhost-template-slave]
[dynamic-virtualhost-template-slave]
<= jinja2-template-base
<= jinja2-template-base
template = ${template-slave-configuration:target}
template = ${template-slave-configuration:target}
...
@@ -608,12 +560,21 @@ apache-certificate =
...
@@ -608,12 +560,21 @@ apache-certificate =
open-port = 80 443
open-port = 80 443
extra_slave_instance_list =
extra_slave_instance_list =
frontend-name =
frontend-name =
monitor-cors-domains =
monitor-httpd-port = 8072
monitor-username = $${monitor-htpasswd:username}
monitor-password = $${monitor-htpasswd:passwd}
#######
#######
# Monitoring sections
# Monitoring sections
#
#
[monitor-instance-parameter]
monitor-httpd-port = $${slap-parameter:monitor-httpd-port}
cors-domains = $${slap-parameter:monitor-cors-domains}
username = $${slap-parameter:monitor-username}
password = $${slap-parameter:monitor-password}
[monitor-conf-parameters]
[monitor-conf-parameters]
private-path-list +=
private-path-list +=
$${directory:logrotate-backup}
$${directory:logrotate-backup}
...
@@ -622,7 +583,7 @@ private-path-list +=
...
@@ -622,7 +583,7 @@ private-path-list +=
[monitor-ats-cache-stats-wrapper]
[monitor-ats-cache-stats-wrapper]
< = jinja2-template-base
< = jinja2-template-base
template = ${template-wrapper:output}
template = ${template-wrapper:output}
rendered = $${monitor-directory:report}/ats-cache-stats
rendered = $${monitor-directory:report
s
}/ats-cache-stats
mode = 0700
mode = 0700
command = export TS_ROOT=$${buildout:directory} && echo "<pre>$(${trafficserver:location}/bin/traffic_shell $${monitor-ats-cache-stats-config:rendered})</pre>"
command = export TS_ROOT=$${buildout:directory} && echo "<pre>$(${trafficserver:location}/bin/traffic_shell $${monitor-ats-cache-stats-config:rendered})</pre>"
extra-context =
extra-context =
...
...
This diff is collapsed.
Click to expand it.
software/apache-frontend/instance-apache-replicate.cfg.in
View file @
bd1049cd
...
@@ -124,6 +124,9 @@ monitor-base-url = ${monitor-conf-parameters:base-url}
...
@@ -124,6 +124,9 @@ monitor-base-url = ${monitor-conf-parameters:base-url}
monitor-url = ${:monitor-base-url}/public/feeds
monitor-url = ${:monitor-base-url}/public/feeds
monitor-user = ${monitor-instance-parameter:username}
monitor-user = ${monitor-instance-parameter:username}
monitor-password = ${monitor-instance-parameter:password}
monitor-password = ${monitor-instance-parameter:password}
{% set monitor_interface_url = slapparameter_dict.get('monitor-interface-url', 'https://monitor.app.officejs.com') -%}
monitor-setup-url = {{ monitor_interface_url }}/#page=settings_configurator&url=${:monitor-url}
#----------------------------
#----------------------------
#--
#--
...
...
This diff is collapsed.
Click to expand it.
software/apache-frontend/instance.cfg
View file @
bd1049cd
...
@@ -33,10 +33,8 @@ recipe = slapos.cookbook:softwaretype
...
@@ -33,10 +33,8 @@ recipe = slapos.cookbook:softwaretype
default = $${dynamic-template-apache-replicate:rendered}
default = $${dynamic-template-apache-replicate:rendered}
RootSoftwareInstance = $${dynamic-template-apache-replicate:rendered}
RootSoftwareInstance = $${dynamic-template-apache-replicate:rendered}
custom-personal = $${dynamic-template-apache-replicate:rendered}
custom-personal = $${dynamic-template-apache-replicate:rendered}
custom-group = $${dynamic-template-apache-replicate:rendered}
single-default = ${template-apache-frontend:output}
single-default = ${template-apache-frontend:output}
single-custom-personal = ${template-apache-frontend:output}
single-custom-personal = ${template-apache-frontend:output}
single-custom-group = ${template-apache-frontend:output}
replicate = $${dynamic-template-apache-replicate:rendered}
replicate = $${dynamic-template-apache-replicate:rendered}
[dynamic-template-apache-replicate]
[dynamic-template-apache-replicate]
...
@@ -47,6 +45,6 @@ extensions = jinja2.ext.do
...
@@ -47,6 +45,6 @@ extensions = jinja2.ext.do
extra-context =
extra-context =
raw template_publish_slave_information ${template-replicate-publish-slave-information:target}
raw template_publish_slave_information ${template-replicate-publish-slave-information:target}
# Must match the key id in [switch-softwaretype] which uses this section.
# Must match the key id in [switch-softwaretype] which uses this section.
raw software_type RootSoftwareInstance-default-custom-personal-
custom-group-
replicate
raw software_type RootSoftwareInstance-default-custom-personal-replicate
raw template_monitor ${monitor2-template:rendered}
raw template_monitor ${monitor2-template:rendered}
This diff is collapsed.
Click to expand it.
software/apache-frontend/software.cfg
View file @
bd1049cd
...
@@ -28,7 +28,7 @@ feedparser = 5.2.1
...
@@ -28,7 +28,7 @@ feedparser = 5.2.1
# Required by:
# Required by:
# slapos.toolbox==0.55
# slapos.toolbox==0.55
lockfile = 0.1
0
.2
lockfile = 0.1
2
.2
# Required by:
# Required by:
# slapos.toolbox==0.55
# slapos.toolbox==0.55
...
...
This diff is collapsed.
Click to expand it.
software/apache-frontend/templates/apache-custom-slave-list.cfg.in
View file @
bd1049cd
...
@@ -132,6 +132,8 @@ crl = {{ custom_ssl_directory }}/crl/
...
@@ -132,6 +132,8 @@ crl = {{ custom_ssl_directory }}/crl/
{% set cert_file = '/'.join([custom_ssl_directory, cert_title.replace('-','.')]) -%}
{% set cert_file = '/'.join([custom_ssl_directory, cert_title.replace('-','.')]) -%}
{% set key_file = '/'.join([custom_ssl_directory, key_title.replace('-','.')]) -%}
{% set key_file = '/'.join([custom_ssl_directory, key_title.replace('-','.')]) -%}
{% do part_list.append(cert_title) -%}
{% do part_list.append(cert_title) -%}
{% do slave_parameter_dict.__setitem__("ssl_crt", cert_file) -%}
{% do slave_parameter_dict.__setitem__("ssl_key", key_file) -%}
{% do slave_instance.__setitem__('path_to_ssl_crt', cert_file) -%}
{% do slave_instance.__setitem__('path_to_ssl_crt', cert_file) -%}
{% do slave_instance.__setitem__('path_to_ssl_key', key_file) -%}
{% do slave_instance.__setitem__('path_to_ssl_key', key_file) -%}
...
...
This diff is collapsed.
Click to expand it.
software/apache-frontend/templates/apache-default-slave-list.cfg.in
deleted
100644 → 0
View file @
a69c0bc9
{% if slap_software_type in software_type -%}
{% set cached_server_dict = {} -%}
{% set part_list = [] -%}
{% set cache_access = "http://%s:%s" % (local_ipv4, cache_port) -%}
{% set TRUE_VALUES = ['y', 'yes', '1', 'true'] -%}
{% set generic_instance_parameter_dict = {'cache_access': cache_access,} -%}
{% if extra_slave_instance_list -%}
{% set slave_instance_information_list = []-%}
{% set slave_instance_list = slave_instance_list + json_module.loads(extra_slave_instance_list) -%}
{% endif -%}
[jinja2-template-base]
recipe = slapos.recipe.template:jinja2
rendered = {{ apache_configuration_directory }}/${:filename}
extra-context =
context =
key eggs_directory buildout:eggs-directory
key develop_eggs_directory buildout:develop-eggs-directory
${:extra-context}
# Go throught slave list to set their configuration
{% for slave_instance in slave_instance_list -%}
{% set slave_reference = slave_instance.get('slave_reference') -%}
{% set slave_section_title = 'dynamic-template-slave-instance-%s' % slave_reference -%}
{% set slave_parameter_dict = generic_instance_parameter_dict.copy() -%}
# Set slave domain if none was defined
{% if slave_instance.get('domain', None) == None -%}
# Backward compatibility
{% if slave_instance.get('custom_domain', None) != None -%}
{% do slave_instance.__setitem__('domain', slave_instance.get('custom_domain') )-%}
{% else -%}
{% do slave_instance.__setitem__('domain', "%s.%s" % (slave_instance.get('slave_reference').replace("-", "").lower(), domain)) -%}
{% endif -%}
{% endif -%}
# Set personal log, two per slave
{% set access_log = '/'.join([apache_log_directory, '%s_access_log' % slave_reference]) -%}
{% set error_log = '/'.join([apache_log_directory, '%s_error_log' % slave_reference]) -%}
# The slave use cache
# Next line is forbidden and people who copy it will be hanged short
{% set enable_cache = ('' ~ slave_instance.get('enable_cache', '')).lower() in TRUE_VALUES -%}
{% if enable_cache -%}
{% do cached_server_dict.__setitem__(slave_instance.get('domain'), slave_instance.get('url')) -%}
{% do slave_instance.__setitem__('url', cache_access) -%}
{% endif -%}
{% do part_list.append(slave_section_title) -%}
# Set up slave configuration file
[{{ slave_section_title }}]
< = jinja2-template-base
template = {{ template_slave_configuration }}
filename = {{ '%s.conf' % slave_reference }}
extensions = jinja2.ext.do
extra-context =
section slave_parameter {{ 'slave-instance-%s-configuration' % slave_reference }}
raw domain {{ domain }}
raw https_port {{ https_port }}
raw http_port {{ http_port }}
raw access_log {{ access_log }}
raw error_log {{ error_log }}
{{ '\n' }}
# Set ssl certificates for each slave
{% for cert_name in ('ssl_key', 'ssl_crt', 'ssl_ca_crt', 'ssl_csr')-%}
{% if cert_name in slave_instance -%}
{% set cert_title = '%s-%s' % (slave_reference, cert_name.replace('ssl_', '')) -%}
{% set cert_file = '/'.join([custom_ssl_directory, cert_title.replace('-','.')]) -%}
{% do part_list.append(cert_title) -%}
{% do slave_instance.__setitem__('path_to_' ~ cert_name, cert_file) -%}
# Store certificates on fs
[{{ cert_title }}]
< = jinja2-template-base
template = {{ empty_template }}
rendered = {{ cert_file }}
extra-context =
key content {{ cert_title + '-config:value' }}
# Store certificate in config
[{{ cert_title + '-config' }}]
value = {{ dumps(slave_instance.get(cert_name)) }}
{% endif -%}
{% endfor -%}
# Set apache configuration for slave
[{{ ('slave-instance-%s-configuration' % slave_reference) }}]
{% for key, value in slave_instance.iteritems() -%}
{{ key }} = {{ dumps(value) }}
{% endfor %}
# Publish slave information
{% if not extra_slave_instance_list -%}
{% set publish_section_title = 'publish-%s-connection-information' % slave_instance.get('slave_reference') -%}
{% do part_list.append(publish_section_title) -%}
[{{ publish_section_title }}]
recipe = slapos.cookbook:publish
-slave-reference = {{ slave_instance.get('slave_reference') }}
public-ipv4 = {{ public_ipv4 }}
domain = {{ slave_instance.get('domain') }}
url = http://{{ slave_instance.get('domain') }}
# Backward compatibility
site_url = ${:url}
{% else -%}
{% do slave_instance_information_list.append({'slave-reference':slave_instance.get('slave_reference'), 'public-ipv4':public_ipv4, 'domain':slave_instance.get('domain'), 'url':"http://%s" % slave_instance.get('domain'), 'site_url':"http://%s" % slave_instance.get('domain')}) -%}
{% endif -%}
{% endfor -%}
# Publish information for the instance
{% set publish_section_title = 'publish-apache-information' -%}
{% do part_list.append(publish_section_title) -%}
[{{ publish_section_title }}]
recipe = slapos.cookbook:publish
public-ipv4 = {{ public_ipv4 }}
private-ipv4 = {{ local_ipv4 }}
domain = {{ domain }}
{% if extra_slave_instance_list -%}
slave-instance-information-list = {{ json_module.dumps(slave_instance_information_list) }}
{% endif -%}
{% do part_list.append('cached-rewrite-rules') -%}
# Set rewrite rules for second apache
[cached-rewrite-rules]
< = jinja2-template-base
template = {{ template_rewrite_cached }}
rendered = {{ rewrite_cached_configuration }}
extra-context =
import json_module json
key server_dict rewrite-rules:rules
# Store Rewrite rules for second apache
[rewrite-rules]
rules = {{ dumps(cached_server_dict) }}
# Add parts generated by template
[buildout]
parts +=
{% for part in part_list -%}
{{ ' %s' % part }}
{% endfor -%}
eggs-directory = {{ eggs_directory }}
develop-eggs-directory = {{ develop_eggs_directory }}
offline = true
cache-access = {{ cache_access }}
{% endif -%}
This diff is collapsed.
Click to expand it.
software/kvm/development.cfg
View file @
bd1049cd
...
@@ -55,5 +55,4 @@ slapos.cookbook =
...
@@ -55,5 +55,4 @@ slapos.cookbook =
slapos.core =
slapos.core =
slapos.toolbox =
slapos.toolbox =
erp5.util =
erp5.util =
# XXX Fix lockfile 0.11.0 requirement (pbr!=0.7,<1.0,>=0.6)
lockfile =
lockfile = 0.10.2
\ No newline at end of file
This diff is collapsed.
Click to expand it.
software/kvm/software.cfg
View file @
bd1049cd
...
@@ -36,7 +36,7 @@ feedparser = 5.2.0.post1
...
@@ -36,7 +36,7 @@ feedparser = 5.2.0.post1
# Required by:
# Required by:
# slapos.toolbox==0.48
# slapos.toolbox==0.48
lockfile = 0.1
0
.2
lockfile = 0.1
2
.2
# Required by:
# Required by:
# websockify==0.5.1
# websockify==0.5.1
...
...
This diff is collapsed.
Click to expand it.
software/neoppod/software-common.cfg
View file @
bd1049cd
...
@@ -123,7 +123,7 @@ ecdsa = 0.13
...
@@ -123,7 +123,7 @@ ecdsa = 0.13
feedparser = 5.2.1
feedparser = 5.2.1
GitPython = 2.0.2
GitPython = 2.0.2
gitdb = 0.6.4
gitdb = 0.6.4
lockfile = 0.1
0
.2
lockfile = 0.1
2
.2
mysqlclient = 1.3.7
mysqlclient = 1.3.7
paramiko = 2.0.0
paramiko = 2.0.0
pycrypto = 2.6.1
pycrypto = 2.6.1
...
...
This diff is collapsed.
Click to expand it.
software/re6stnet/software.cfg
View file @
bd1049cd
...
@@ -133,7 +133,7 @@ feedparser = 5.1.3
...
@@ -133,7 +133,7 @@ feedparser = 5.1.3
# Required by:
# Required by:
# slapos.toolbox==0.55
# slapos.toolbox==0.55
lockfile = 0.1
0
.2
lockfile = 0.1
2
.2
# Required by:
# Required by:
# re6stnet===0-413.gbec6b3c.dirty
# re6stnet===0-413.gbec6b3c.dirty
...
...
This diff is collapsed.
Click to expand it.
software/slaprunner/development.cfg
View file @
bd1049cd
...
@@ -11,13 +11,14 @@ parts =
...
@@ -11,13 +11,14 @@ parts =
slapos.toolbox-dev
slapos.toolbox-dev
slapos.cookbook-dev
slapos.cookbook-dev
slapos.core-dev
slapos.core-dev
erp5.util-dev
slapos-cookbook
slapos-cookbook
${:common-parts}
${:common-parts}
[slapos.toolbox-repository]
[slapos.toolbox-repository]
recipe = slapos.recipe.build:gitclone
recipe = slapos.recipe.build:gitclone
repository = https://lab.nexedi.com/nexedi/slapos.toolbox.git
repository = https://lab.nexedi.com/nexedi/slapos.toolbox.git
branch =
slaprunner-paas
branch =
master
git-executable = ${git:location}/bin/git
git-executable = ${git:location}/bin/git
develop = true
develop = true
...
@@ -29,7 +30,7 @@ setup = ${slapos.toolbox-repository:location}
...
@@ -29,7 +30,7 @@ setup = ${slapos.toolbox-repository:location}
[slapos.cookbook-repository]
[slapos.cookbook-repository]
recipe = slapos.recipe.build:gitclone
recipe = slapos.recipe.build:gitclone
repository = https://lab.nexedi.com/nexedi/slapos.git
repository = https://lab.nexedi.com/nexedi/slapos.git
branch =
slaprunner-paas
branch =
master
git-executable = ${git:location}/bin/git
git-executable = ${git:location}/bin/git
develop = true
develop = true
...
@@ -67,5 +68,4 @@ setup = ${slapos.core-repository:location}
...
@@ -67,5 +68,4 @@ setup = ${slapos.core-repository:location}
slapos.cookbook =
slapos.cookbook =
slapos.core =
slapos.core =
slapos.toolbox =
slapos.toolbox =
# XXX Fix lockfile 0.11.0 requirement (pbr!=0.7,<1.0,>=0.6)
lockfile =
lockfile = 0.10.2
This diff is collapsed.
Click to expand it.
software/slaprunner/software.cfg
View file @
bd1049cd
...
@@ -36,7 +36,7 @@ feedparser = 5.2.1
...
@@ -36,7 +36,7 @@ feedparser = 5.2.1
# Required by:
# Required by:
# slapos.toolbox==0.55
# slapos.toolbox==0.55
lockfile = 0.1
0
.2
lockfile = 0.1
2
.2
# Required by:
# Required by:
# slapos.toolbox==0.55
# slapos.toolbox==0.55
...
...
This diff is collapsed.
Click to expand it.
stack/erp5/buildout.cfg
View file @
bd1049cd
...
@@ -679,7 +679,7 @@ Products.LongRequestLogger = 2.0.0
...
@@ -679,7 +679,7 @@ Products.LongRequestLogger = 2.0.0
Products.MimetypesRegistry = 2.0.8
Products.MimetypesRegistry = 2.0.8
Products.PluginRegistry = 1.4
Products.PluginRegistry = 1.4
Products.TIDStorage = 5.4.9
Products.TIDStorage = 5.4.9
PyPDF2 = 1.2
5.1
PyPDF2 = 1.2
6.0
PyStemmer = 1.3.0
PyStemmer = 1.3.0
PyXML = 0.8.5
PyXML = 0.8.5
Pympler = 0.4.3
Pympler = 0.4.3
...
@@ -704,7 +704,7 @@ interval = 1.0.0
...
@@ -704,7 +704,7 @@ interval = 1.0.0
ipdb = 0.10.0
ipdb = 0.10.0
ipykernel = 4.3.1
ipykernel = 4.3.1
ipython = 4.2.0
ipython = 4.2.0
ipywidgets = 5.1.
3
ipywidgets = 5.1.
4
logilab-common = 1.2.0
logilab-common = 1.2.0
matplotlib = 1.5.1
matplotlib = 1.5.1
mistune = 0.7.2
mistune = 0.7.2
...
@@ -723,7 +723,7 @@ pylint = 1.4.4
...
@@ -723,7 +723,7 @@ pylint = 1.4.4
python-memcached = 1.57
python-memcached = 1.57
pytracemalloc = 1.2
pytracemalloc = 1.2
pyzmq = 15.2.0
pyzmq = 15.2.0
qrcode = 5.
2.2
qrcode = 5.
3
restkit = 4.2.2
restkit = 4.2.2
rtjp-eventlet = 0.3.2
rtjp-eventlet = 0.3.2
scikit-learn = 0.17.1
scikit-learn = 0.17.1
...
...
This diff is collapsed.
Click to expand it.
stack/monitor/buildout.cfg
View file @
bd1049cd
...
@@ -100,7 +100,7 @@ recipe = slapos.recipe.template:jinja2
...
@@ -100,7 +100,7 @@ recipe = slapos.recipe.template:jinja2
filename = template-monitor.cfg
filename = template-monitor.cfg
template = ${:_profile_base_location_}/instance-monitor.cfg.jinja2.in
template = ${:_profile_base_location_}/instance-monitor.cfg.jinja2.in
rendered = ${buildout:directory}/template-monitor.cfg
rendered = ${buildout:directory}/template-monitor.cfg
md5sum =
b6da7709da533c5c8280e5e103809c58
md5sum =
354da434447697b8b4f503a62599cf32
context =
context =
key apache_location apache:location
key apache_location apache:location
key gzip_location gzip:location
key gzip_location gzip:location
...
@@ -131,13 +131,13 @@ depends =
...
@@ -131,13 +131,13 @@ depends =
[monitor2-bin]
[monitor2-bin]
<= monitor-template-script
<= monitor-template-script
filename = monitor.py
filename = monitor.py
md5sum =
6db3d9fcfff32c556f2bedee804ffb47
md5sum =
3166f5d110fda63aade1ca486dfb7996
[run-promise-py]
[run-promise-py]
recipe = slapos.recipe.template:jinja2
recipe = slapos.recipe.template:jinja2
template = ${:_profile_base_location_}/scripts/run-promise.py
template = ${:_profile_base_location_}/scripts/run-promise.py
rendered = ${buildout:parts-directory}/monitor-scripts/run-promise.py
rendered = ${buildout:parts-directory}/monitor-scripts/run-promise.py
md5sum =
5694920c5789f4d953cb5d41adfa7f8b
md5sum =
2570b11cd1f718f240a9c7a05ffa861d
mode = 0755
mode = 0755
context =
context =
raw python ${buildout:directory}/bin/${extra-eggs:interpreter}
raw python ${buildout:directory}/bin/${extra-eggs:interpreter}
...
@@ -155,12 +155,12 @@ md5sum = f297779d0881f4bd48081506efb492a4
...
@@ -155,12 +155,12 @@ md5sum = f297779d0881f4bd48081506efb492a4
[monitor-globalstate]
[monitor-globalstate]
<= monitor-template-script
<= monitor-template-script
filename = globalstate.py
filename = globalstate.py
md5sum =
384a1148cb3da9cf353a108fe70709c5
md5sum =
1ebead4480d62d5e1d2612c2d9f4e1d8
[monitor-collect]
[monitor-collect]
<= monitor-template-script
<= monitor-template-script
filename = collect.py
filename = collect.py
md5sum =
cc65aebd4c35b3172a7ca83abde761bc
md5sum =
78fbcb56761315bde354fe7914d3c54f
[monitor-document-edit]
[monitor-document-edit]
<= monitor-template-script
<= monitor-template-script
...
...
This diff is collapsed.
Click to expand it.
stack/monitor/instance-monitor.cfg.jinja2.in
View file @
bd1049cd
...
@@ -221,7 +221,8 @@ update-command = ${:command}
...
@@ -221,7 +221,8 @@ update-command = ${:command}
recipe = plone.recipe.command
recipe = plone.recipe.command
stop-on-error = true
stop-on-error = true
htpasswd-path = ${monitor-directory:etc}/monitor-htpasswd
htpasswd-path = ${monitor-directory:etc}/monitor-htpasswd
command = {{ apache_location }}/bin/htpasswd -cb ${:htpasswd-path} ${:user} ${:password}
command = if [ ! -f "${:htpasswd-path}" ]; then {{ apache_location }}/bin/htpasswd -cb ${:htpasswd-path} ${:user} ${:password}; fi
update-command = ${:command}
user = ${monitor-instance-parameter:username}
user = ${monitor-instance-parameter:username}
password = ${monitor-instance-parameter:password}
password = ${monitor-instance-parameter:password}
...
...
This diff is collapsed.
Click to expand it.
stack/monitor/scripts/collect.py
View file @
bd1049cd
...
@@ -262,51 +262,73 @@ def appendToJsonFile(file_path, content, stepback=2):
...
@@ -262,51 +262,73 @@ def appendToJsonFile(file_path, content, stepback=2):
jfile
.
seek
(
position
)
jfile
.
seek
(
position
)
jfile
.
write
(
'%s}'
%
',"{}"]'
.
format
(
content
))
jfile
.
write
(
'%s}'
%
',"{}"]'
.
format
(
content
))
def
initProcessDataFile
(
file_path
):
with
open
(
process_file
,
'w'
)
as
fprocess
:
data_dict
=
{
"date"
:
time
.
time
(),
"data"
:
[
"date, total process, CPU percent, CPU time, CPU threads"
]
}
fprocess
.
write
(
json
.
dumps
(
data_dict
))
def
initMemoryDataFile
(
file_path
):
with
open
(
mem_file
,
'w'
)
as
fmem
:
data_dict
=
{
"date"
:
time
.
time
(),
"data"
:
[
"date, memory used percent, memory used"
]
}
fmem
.
write
(
json
.
dumps
(
data_dict
))
def
initIODataFile
(
file_path
):
with
open
(
io_file
,
'w'
)
as
fio
:
data_dict
=
{
"date"
:
time
.
time
(),
"data"
:
[
"date, io rw counter, io cycles counter, disk used"
]
}
fio
.
write
(
json
.
dumps
(
data_dict
))
if
__name__
==
"__main__"
:
if
__name__
==
"__main__"
:
parser
=
parseArguments
()
parser
=
parseArguments
()
if
not
os
.
path
.
exists
(
parser
.
output_folder
)
and
os
.
path
.
isdir
(
parser
.
output_folder
):
if
not
os
.
path
.
exists
(
parser
.
output_folder
)
and
os
.
path
.
isdir
(
parser
.
output_folder
):
raise
Exception
(
"Invalid ouput folder: %s"
%
parser
.
output_folder
)
raise
Exception
(
"Invalid ouput folder: %s"
%
parser
.
output_folder
)
collector
=
RessourceCollect
(
parser
.
collector_db
)
date_scope
=
datetime
.
now
().
strftime
(
'%Y-%m-%d'
)
stat_info
=
os
.
stat
(
parser
.
output_folder
)
partition_user
=
pwd
.
getpwuid
(
stat_info
.
st_uid
)[
0
]
# Consumption global status
# Consumption global status
process_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource_process.data.json'
)
process_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource_process.data.json'
)
mem_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource_memory.data.json'
)
mem_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource_memory.data.json'
)
io_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource_io.data.json'
)
io_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource_io.data.json'
)
resource_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_process_resource.status.json'
)
status_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource.status.json'
)
if
not
os
.
path
.
exists
(
parser
.
collector_db
):
print
"Collector database not found..."
initProcessDataFile
(
process_file
)
initMemoryDataFile
(
mem_file
)
initIODataFile
(
io_file
)
with
open
(
status_file
,
"w"
)
as
status_file
:
status_file
.
write
(
'{"cpu_time": 0, "cpu_percent": 0, "memory_rss": 0, "memory_percent": 0, "io_rw_counter": 0, "date": "", "total_process": 0, "disk_used": 0, "io_cycles_counter": 0, "cpu_num_threads": 0}'
)
with
open
(
resource_file
,
"w"
)
as
resource_file
:
resource_file
.
write
(
'[]'
)
exit
(
1
)
collector
=
RessourceCollect
(
parser
.
collector_db
)
date_scope
=
datetime
.
now
().
strftime
(
'%Y-%m-%d'
)
stat_info
=
os
.
stat
(
parser
.
output_folder
)
partition_user
=
pwd
.
getpwuid
(
stat_info
.
st_uid
)[
0
]
process_result
,
memory_result
,
io_result
=
collector
.
getPartitionComsumptionStatus
(
partition_user
)
process_result
,
memory_result
,
io_result
=
collector
.
getPartitionComsumptionStatus
(
partition_user
)
resource_file
=
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_process_resource.status.json'
)
label_list
=
[
'date'
,
'total_process'
,
'cpu_percent'
,
'cpu_time'
,
'cpu_num_threads'
,
label_list
=
[
'date'
,
'total_process'
,
'cpu_percent'
,
'cpu_time'
,
'cpu_num_threads'
,
'memory_percent'
,
'memory_rss'
,
'io_rw_counter'
,
'io_cycles_counter'
,
'memory_percent'
,
'memory_rss'
,
'io_rw_counter'
,
'io_cycles_counter'
,
'disk_used'
]
'disk_used'
]
resource_status_dict
=
{}
resource_status_dict
=
{}
if
not
os
.
path
.
exists
(
process_file
):
if
not
os
.
path
.
exists
(
process_file
):
with
open
(
process_file
,
'w'
)
as
fprocess
:
initProcessDataFile
(
process_file
)
data_dict
=
{
"date"
:
time
.
time
(),
"data"
:
[
"date, total process, CPU percent, CPU time, CPU threads"
]
}
fprocess
.
write
(
json
.
dumps
(
data_dict
))
if
not
os
.
path
.
exists
(
mem_file
):
if
not
os
.
path
.
exists
(
mem_file
):
with
open
(
mem_file
,
'w'
)
as
fmem
:
initMemoryDataFile
(
mem_file
)
data_dict
=
{
"date"
:
time
.
time
(),
"data"
:
[
"date, memory used percent, memory used"
]
}
fmem
.
write
(
json
.
dumps
(
data_dict
))
if
not
os
.
path
.
exists
(
io_file
):
if
not
os
.
path
.
exists
(
io_file
):
with
open
(
io_file
,
'w'
)
as
fio
:
initIODataFile
(
io_file
)
data_dict
=
{
"date"
:
time
.
time
(),
"data"
:
[
"date, io rw counter, io cycles counter, disk used"
]
}
fio
.
write
(
json
.
dumps
(
data_dict
))
if
process_result
and
process_result
[
'total_process'
]
!=
0.0
:
if
process_result
and
process_result
[
'total_process'
]
!=
0.0
:
appendToJsonFile
(
process_file
,
", "
.
join
(
appendToJsonFile
(
process_file
,
", "
.
join
(
...
@@ -324,7 +346,7 @@ if __name__ == "__main__":
...
@@ -324,7 +346,7 @@ if __name__ == "__main__":
)
)
resource_status_dict
.
update
(
io_result
)
resource_status_dict
.
update
(
io_result
)
with
open
(
os
.
path
.
join
(
parser
.
output_folder
,
'monitor_resource.status.json'
)
,
'w'
)
as
fp
:
with
open
(
status_file
,
'w'
)
as
fp
:
fp
.
write
(
json
.
dumps
(
resource_status_dict
))
fp
.
write
(
json
.
dumps
(
resource_status_dict
))
# Consumption Ressource
# Consumption Ressource
...
...
This diff is collapsed.
Click to expand it.
stack/monitor/scripts/globalstate.py
View file @
bd1049cd
...
@@ -104,6 +104,7 @@ def main(args_list):
...
@@ -104,6 +104,7 @@ def main(args_list):
'success'
:
success
,
'success'
:
success
,
'warning'
:
warning
,
'warning'
:
warning
,
},
},
type
=
'global'
,
date
=
latest_date
,
date
=
latest_date
,
_links
=
{
"rss_url"
:
{
"href"
:
"%s/public/feed"
%
base_url
},
_links
=
{
"rss_url"
:
{
"href"
:
"%s/public/feed"
%
base_url
},
"public_url"
:
{
"href"
:
"%s/share/jio_public/"
%
base_url
},
"public_url"
:
{
"href"
:
"%s/share/jio_public/"
%
base_url
},
...
...
This diff is collapsed.
Click to expand it.
stack/monitor/scripts/monitor.py
View file @
bd1049cd
...
@@ -97,7 +97,6 @@ class Monitoring(object):
...
@@ -97,7 +97,6 @@ class Monitoring(object):
self
.
report_script_folder
=
config
.
get
(
"monitor"
,
"report-folder"
)
self
.
report_script_folder
=
config
.
get
(
"monitor"
,
"report-folder"
)
self
.
webdav_url
=
'%s/share'
%
config
.
get
(
"monitor"
,
"base-url"
)
self
.
webdav_url
=
'%s/share'
%
config
.
get
(
"monitor"
,
"base-url"
)
self
.
public_url
=
'%s/public'
%
config
.
get
(
"monitor"
,
"base-url"
)
self
.
public_url
=
'%s/public'
%
config
.
get
(
"monitor"
,
"base-url"
)
self
.
status_history_folder
=
os
.
path
.
join
(
self
.
public_folder
,
'history'
)
self
.
python
=
config
.
get
(
"monitor"
,
"python"
)
or
"python"
self
.
python
=
config
.
get
(
"monitor"
,
"python"
)
or
"python"
self
.
public_path_list
=
config
.
get
(
"monitor"
,
"public-path-list"
).
split
()
self
.
public_path_list
=
config
.
get
(
"monitor"
,
"public-path-list"
).
split
()
self
.
private_path_list
=
config
.
get
(
"monitor"
,
"private-path-list"
).
split
()
self
.
private_path_list
=
config
.
get
(
"monitor"
,
"private-path-list"
).
split
()
...
@@ -107,7 +106,7 @@ class Monitoring(object):
...
@@ -107,7 +106,7 @@ class Monitoring(object):
self
.
parameter_cfg_file
=
config
.
get
(
"monitor"
,
"parameter-file-path"
).
strip
()
self
.
parameter_cfg_file
=
config
.
get
(
"monitor"
,
"parameter-file-path"
).
strip
()
self
.
config_folder
=
os
.
path
.
join
(
self
.
private_folder
,
'config'
)
self
.
config_folder
=
os
.
path
.
join
(
self
.
private_folder
,
'config'
)
self
.
report_folder
=
os
.
path
.
join
(
self
.
private_folder
,
'report'
)
self
.
report_folder
=
self
.
private_folder
self
.
promise_dict
=
{}
self
.
promise_dict
=
{}
for
promise_folder
in
self
.
promise_folder_list
:
for
promise_folder
in
self
.
promise_folder_list
:
...
@@ -237,14 +236,14 @@ class Monitoring(object):
...
@@ -237,14 +236,14 @@ class Monitoring(object):
else
:
else
:
response
=
urllib2
.
urlopen
(
url
)
response
=
urllib2
.
urlopen
(
url
)
except
urllib2
.
HTTPError
:
except
urllib2
.
HTTPError
:
return
'Unknow Instance'
return
'Unknow
n
Instance'
else
:
else
:
try
:
try
:
monitor_dict
=
json
.
loads
(
response
.
read
())
monitor_dict
=
json
.
loads
(
response
.
read
())
return
monitor_dict
.
get
(
'title'
,
'Unknow Instance'
)
return
monitor_dict
.
get
(
'title'
,
'Unknow
n
Instance'
)
except
ValueError
,
e
:
except
ValueError
,
e
:
print
"Bad Json file at %s"
%
url
print
"Bad Json file at %s"
%
url
return
'Unknow Instance'
return
'Unknow
n
Instance'
def
configureFolders
(
self
):
def
configureFolders
(
self
):
# configure public and private folder
# configure public and private folder
...
@@ -256,7 +255,6 @@ class Monitoring(object):
...
@@ -256,7 +255,6 @@ class Monitoring(object):
jio_private
=
os
.
path
.
join
(
self
.
webdav_folder
,
'jio_private'
)
jio_private
=
os
.
path
.
join
(
self
.
webdav_folder
,
'jio_private'
)
mkdirAll
(
jio_public
)
mkdirAll
(
jio_public
)
mkdirAll
(
jio_private
)
mkdirAll
(
jio_private
)
mkdirAll
(
self
.
status_history_folder
)
try
:
try
:
os
.
symlink
(
self
.
public_folder
,
os
.
path
.
join
(
jio_public
,
'.jio_documents'
))
os
.
symlink
(
self
.
public_folder
,
os
.
path
.
join
(
jio_public
,
'.jio_documents'
))
except
OSError
,
e
:
except
OSError
,
e
:
...
@@ -269,11 +267,9 @@ class Monitoring(object):
...
@@ -269,11 +267,9 @@ class Monitoring(object):
raise
raise
self
.
data_folder
=
os
.
path
.
join
(
self
.
private_folder
,
'data'
,
'.jio_documents'
)
self
.
data_folder
=
os
.
path
.
join
(
self
.
private_folder
,
'data'
,
'.jio_documents'
)
self
.
report_folder
=
os
.
path
.
join
(
self
.
report_folder
,
'.jio_documents'
)
config_folder
=
os
.
path
.
join
(
self
.
config_folder
,
'.jio_documents'
)
config_folder
=
os
.
path
.
join
(
self
.
config_folder
,
'.jio_documents'
)
mkdirAll
(
self
.
data_folder
)
mkdirAll
(
self
.
data_folder
)
mkdirAll
(
config_folder
)
mkdirAll
(
config_folder
)
mkdirAll
(
self
.
report_folder
)
try
:
try
:
os
.
symlink
(
os
.
path
.
join
(
self
.
private_folder
,
'data'
),
os
.
symlink
(
os
.
path
.
join
(
self
.
private_folder
,
'data'
),
os
.
path
.
join
(
jio_private
,
'data'
))
os
.
path
.
join
(
jio_private
,
'data'
))
...
@@ -285,11 +281,6 @@ class Monitoring(object):
...
@@ -285,11 +281,6 @@ class Monitoring(object):
except
OSError
,
e
:
except
OSError
,
e
:
if
e
.
errno
!=
os
.
errno
.
EEXIST
:
if
e
.
errno
!=
os
.
errno
.
EEXIST
:
raise
raise
try
:
os
.
symlink
(
self
.
report_folder
,
os
.
path
.
join
(
jio_private
,
'report'
))
except
OSError
,
e
:
if
e
.
errno
!=
os
.
errno
.
EEXIST
:
raise
def
makeConfigurationFiles
(
self
):
def
makeConfigurationFiles
(
self
):
config_folder
=
os
.
path
.
join
(
self
.
config_folder
,
'.jio_documents'
)
config_folder
=
os
.
path
.
join
(
self
.
config_folder
,
'.jio_documents'
)
...
@@ -397,7 +388,7 @@ class Monitoring(object):
...
@@ -397,7 +388,7 @@ class Monitoring(object):
def
generateReportCronEntries
(
self
):
def
generateReportCronEntries
(
self
):
cron_line_list
=
[]
cron_line_list
=
[]
# We should add the possibility to modify this parameter later from monitor interface
# We should add the possibility to modify this parameter later from monitor interface
report_frequency
=
"*/
3
0 * * * *"
report_frequency
=
"*/
2
0 * * * *"
report_name_list
=
[
name
.
replace
(
'.report.json'
,
''
)
report_name_list
=
[
name
.
replace
(
'.report.json'
,
''
)
for
name
in
os
.
listdir
(
self
.
report_folder
)
if
name
.
endswith
(
'.report.json'
)]
for
name
in
os
.
listdir
(
self
.
report_folder
)
if
name
.
endswith
(
'.report.json'
)]
...
@@ -451,7 +442,7 @@ class Monitoring(object):
...
@@ -451,7 +442,7 @@ class Monitoring(object):
for
service_name
,
promise
in
self
.
promise_items
:
for
service_name
,
promise
in
self
.
promise_items
:
service_config
=
promise
[
"configuration"
]
service_config
=
promise
[
"configuration"
]
service_status_path
=
"%s/%s.status.json"
%
(
self
.
public_folder
,
service_name
)
# hardcoded
service_status_path
=
"%s/%s.status.json"
%
(
self
.
public_folder
,
service_name
)
mkdirAll
(
os
.
path
.
dirname
(
service_status_path
))
mkdirAll
(
os
.
path
.
dirname
(
service_status_path
))
promise_cmd_line
=
[
promise_cmd_line
=
[
...
@@ -463,7 +454,7 @@ class Monitoring(object):
...
@@ -463,7 +454,7 @@ class Monitoring(object):
'--promise_script "%s"'
%
promise
[
"path"
],
'--promise_script "%s"'
%
promise
[
"path"
],
'--promise_name "%s"'
%
service_name
,
'--promise_name "%s"'
%
service_name
,
'--monitor_url "%s/jio_private/"'
%
self
.
webdav_url
,
# XXX hardcoded,
'--monitor_url "%s/jio_private/"'
%
self
.
webdav_url
,
# XXX hardcoded,
'--history_folder "%s"'
%
self
.
status_history
_folder
,
'--history_folder "%s"'
%
self
.
public
_folder
,
'--instance_name "%s"'
%
self
.
title
,
'--instance_name "%s"'
%
self
.
title
,
'--hosting_name "%s"'
%
self
.
root_title
]
'--hosting_name "%s"'
%
self
.
root_title
]
...
@@ -539,13 +530,21 @@ class Monitoring(object):
...
@@ -539,13 +530,21 @@ class Monitoring(object):
# Rotate monitor data files
# Rotate monitor data files
option_list
=
[
option_list
=
[
'daily'
,
'nocreate'
,
'noolddir'
,
'rotate
30
'
,
'daily'
,
'nocreate'
,
'noolddir'
,
'rotate
5
'
,
'nocompress'
,
'extension .json'
,
'dateext'
,
'nocompress'
,
'extension .json'
,
'dateext'
,
'dateformat -%Y-%m-%d'
,
'notifempty'
'dateformat -%Y-%m-%d'
,
'notifempty'
]
]
file_list
=
[
"%s/*.data.json"
%
self
.
data_folder
]
file_list
=
[
"%s/*.data.json"
%
self
.
data_folder
]
self
.
generateLogrotateEntry
(
'monitor.data'
,
file_list
,
option_list
)
self
.
generateLogrotateEntry
(
'monitor.data'
,
file_list
,
option_list
)
# Rotate public history status file, delete data of previous days
option_list
=
[
'daily'
,
'nocreate'
,
'rotate 0'
,
'nocompress'
,
'notifempty'
]
file_list
=
[
"%s/*.history.json"
%
self
.
public_folder
]
self
.
generateLogrotateEntry
(
'monitor.service.status'
,
file_list
,
option_list
)
# Add cron entry for SlapOS Collect
# Add cron entry for SlapOS Collect
command
=
"%s %s --output_folder %s --collector_db %s"
%
(
self
.
python
,
command
=
"%s %s --output_folder %s --collector_db %s"
%
(
self
.
python
,
self
.
collect_script
,
self
.
data_folder
,
self
.
collector_db
)
self
.
collect_script
,
self
.
data_folder
,
self
.
collector_db
)
...
...
This diff is collapsed.
Click to expand it.
stack/monitor/scripts/run-promise.py
View file @
bd1049cd
...
@@ -65,6 +65,7 @@ def main():
...
@@ -65,6 +65,7 @@ def main():
status_json
[
'title'
]
=
parser
.
promise_name
status_json
[
'title'
]
=
parser
.
promise_name
status_json
[
'instance'
]
=
parser
.
instance_name
status_json
[
'instance'
]
=
parser
.
instance_name
status_json
[
'hosting_subscription'
]
=
parser
.
hosting_name
status_json
[
'hosting_subscription'
]
=
parser
.
hosting_name
status_json
[
'type'
]
=
parser
.
promise_type
# Save the lastest status change date (needed for rss)
# Save the lastest status change date (needed for rss)
status_json
[
'change-time'
]
=
ps_process
.
create_time
()
status_json
[
'change-time'
]
=
ps_process
.
create_time
()
...
@@ -85,9 +86,7 @@ def main():
...
@@ -85,9 +86,7 @@ def main():
os
.
remove
(
parser
.
pid_path
)
os
.
remove
(
parser
.
pid_path
)
def
updateStatusHistoryFolder
(
name
,
status_file
,
history_folder
,
promise_type
):
def
updateStatusHistoryFolder
(
name
,
status_file
,
history_folder
,
promise_type
):
old_history_list
=
[]
history_path
=
os
.
path
.
join
(
history_folder
)
keep_item_amount
=
25
history_path
=
os
.
path
.
join
(
history_folder
,
name
,
'.jio_documents'
)
if
not
os
.
path
.
exists
(
status_file
):
if
not
os
.
path
.
exists
(
status_file
):
return
return
if
not
os
.
path
.
exists
(
history_folder
):
if
not
os
.
path
.
exists
(
history_folder
):
...
@@ -101,25 +100,46 @@ def updateStatusHistoryFolder(name, status_file, history_folder, promise_type):
...
@@ -101,25 +100,46 @@ def updateStatusHistoryFolder(name, status_file, history_folder, promise_type):
else
:
raise
else
:
raise
with
open
(
status_file
,
'r'
)
as
sf
:
with
open
(
status_file
,
'r'
)
as
sf
:
status_dict
=
json
.
loads
(
sf
.
read
())
status_dict
=
json
.
loads
(
sf
.
read
())
filename
=
'%s.%s.json'
%
(
status_dict
[
'start-date'
].
replace
(
' '
,
'_'
).
replace
(
':'
,
''
),
if
promise_type
==
'status'
:
promise_type
)
filename
=
'%s.history.json'
%
name
history_file
=
os
.
path
.
join
(
history_path
,
filename
)
# Remove links from history (not needed)
status_dict
.
pop
(
'_links'
,
None
)
if
not
os
.
path
.
exists
(
history_file
):
with
open
(
history_file
,
'w'
)
as
f_history
:
data_dict
=
{
"date"
:
time
.
time
(),
"data"
:
[
status_dict
]
}
f_history
.
write
(
json
.
dumps
(
data_dict
))
else
:
with
open
(
history_file
,
mode
=
"r+"
)
as
f_history
:
f_history
.
seek
(
0
,
2
)
position
=
f_history
.
tell
()
-
2
f_history
.
seek
(
position
)
#f_history.write(',%s]}' % str(status_dict))
f_history
.
write
(
'%s}'
%
',{}]'
.
format
(
json
.
dumps
(
status_dict
)))
elif
promise_type
==
'report'
:
# keep_item_amount = 3
filename
=
'%s.history.json'
%
(
name
)
copyfile
(
status_file
,
os
.
path
.
join
(
history_path
,
filename
))
copyfile
(
status_file
,
os
.
path
.
join
(
history_path
,
filename
))
# Don't let history foler grow too much, keep xx files
"""
# Don't let history foler grow too much, keep xx files
file_list
=
filter
(
os
.
path
.
isfile
,
file_list = filter(os.path.isfile,
glob
.
glob
(
"%s/*.%s
.json"
%
(
history_path
,
promise_type
))
glob.glob("%s/*.%s.history
.json" % (history_path, promise_type))
)
)
file_count
=
len
(
file_list
)
file_count = len(file_list)
if
file_count
>
keep_item_amount
:
if file_count > keep_item_amount:
file_list
.
sort
(
key
=
lambda
x
:
os
.
path
.
getmtime
(
x
))
file_list.sort(key=lambda x: os.path.getmtime(x))
while
file_count
>
keep_item_amount
:
while file_count > keep_item_amount:
to_delete
=
file_list
.
pop
(
0
)
to_delete = file_list.pop(0)
try
:
try:
os
.
unlink
(
to_delete
)
os.unlink(to_delete)
file_count
-=
1
file_count -= 1
except
OSError
:
except OSError:
raise
raise"""
def
generateStatusJsonFromProcess
(
process
,
start_date
=
None
,
title
=
None
):
def
generateStatusJsonFromProcess
(
process
,
start_date
=
None
,
title
=
None
):
stdout
,
stderr
=
process
.
communicate
()
stdout
,
stderr
=
process
.
communicate
()
...
...
This diff is collapsed.
Click to expand it.
stack/slapos.cfg
View file @
bd1049cd
...
@@ -126,10 +126,10 @@ netaddr = 0.7.18
...
@@ -126,10 +126,10 @@ netaddr = 0.7.18
pbr = 1.9.1
pbr = 1.9.1
plone.recipe.command = 1.1
plone.recipe.command = 1.1
prettytable = 0.7.2
prettytable = 0.7.2
psutil = 4.
1
.0
psutil = 4.
2
.0
pyOpenSSL = 16.0.0
pyOpenSSL = 16.0.0
pyasn1 = 0.1.9
pyasn1 = 0.1.9
pyparsing = 2.1.
3
pyparsing = 2.1.
4
pytz = 2016.4
pytz = 2016.4
requests = 2.10.0
requests = 2.10.0
setuptools = 19.6.2
setuptools = 19.6.2
...
@@ -155,7 +155,7 @@ MarkupSafe = 0.23
...
@@ -155,7 +155,7 @@ MarkupSafe = 0.23
# Required by:
# Required by:
# cryptography==1.3.2
# cryptography==1.3.2
enum34 = 1.1.
5
enum34 = 1.1.
6
# Required by:
# Required by:
# jsonschema==2.5.1
# jsonschema==2.5.1
...
@@ -183,7 +183,7 @@ pycparser = 2.14
...
@@ -183,7 +183,7 @@ pycparser = 2.14
# Required by:
# Required by:
# slapos.core==1.3.15
# slapos.core==1.3.15
supervisor = 3.
2.3
supervisor = 3.
3.0
# Required by:
# Required by:
# slapos.core==1.3.15
# slapos.core==1.3.15
...
...
This diff is collapsed.
Click to expand it.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment