Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
4fd5ab16
Commit
4fd5ab16
authored
Dec 20, 2021
by
Grzegorz Bizon
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Visualize the three tracks in CI/CD time decay blueprint
parent
57efe183
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
13 additions
and
5 deletions
+13
-5
doc/architecture/blueprints/ci_data_decay/index.md
doc/architecture/blueprints/ci_data_decay/index.md
+13
-5
doc/architecture/blueprints/ci_data_decay/pipeline_data_time_decay.png
...ure/blueprints/ci_data_decay/pipeline_data_time_decay.png
+0
-0
No files found.
doc/architecture/blueprints/ci_data_decay/index.md
View file @
4fd5ab16
...
@@ -44,18 +44,26 @@ CI/CD data is subject to
...
@@ -44,18 +44,26 @@ CI/CD data is subject to
[
time-decay
](
https://about.gitlab.com/company/team/structure/working-groups/database-scalability/time-decay.html
)
[
time-decay
](
https://about.gitlab.com/company/team/structure/working-groups/database-scalability/time-decay.html
)
because, usually, pipelines that are a few months old are not frequently
because, usually, pipelines that are a few months old are not frequently
accessed or are even not relevant anymore. Restricting access to processing
accessed or are even not relevant anymore. Restricting access to processing
pipelines that are longer than a few months might help us to move this data to
pipelines that are older than a few months might help us to move this data out
a different storage, that is more performant and cost effective.
of the primary database, to a different storage, that is more performant and
cost effective.
It is already possible to prevent processing builds
[
that have been
It is already possible to prevent processing builds
[
that have been
archived
](
../../../user/admin_area/settings/continuous_integration.md#archive-jobs
)
.
archived
](
../../../user/admin_area/settings/continuous_integration.md#archive-jobs
)
.
When a build gets archived it will not be possible to retry it, but we do not
When a build gets archived it will not be possible to retry it, but we do not
move data from the database.
move data out of the database, it still consumes resources that are scarce in
the primary database.
In order to improve performance and make it easier to scale CI/CD data storage
In order to improve performance and make it easier to scale CI/CD data storage
we might want to follow these three tracks described below.
we might want to follow these three tracks described below.
### Migrate build metadata of archived pipelines
![
pipeline data time decay
](
pipeline_data_time_decay.png
)
1.
Partition builds queuing tables
2.
Archive CI/CD data into partitioned database schema
3.
Migrate archived builds metadata out of primary database
### Migrate archived builds metadata out of primary database
Once a build (or a pipeline) gets archived, it is no longer possible to resume
Once a build (or a pipeline) gets archived, it is no longer possible to resume
pipeline processing in such pipeline. It means that all the metadata, we store
pipeline processing in such pipeline. It means that all the metadata, we store
...
@@ -75,7 +83,7 @@ dataset. Technical evaluation will be required to find the best solution here.
...
@@ -75,7 +83,7 @@ dataset. Technical evaluation will be required to find the best solution here.
Epic:
[
Migrate build metadata of archived pipelines
](
https://gitlab.com/groups/gitlab-org/-/epics/7216
)
.
Epic:
[
Migrate build metadata of archived pipelines
](
https://gitlab.com/groups/gitlab-org/-/epics/7216
)
.
###
Partition archived CI/CD dat
a
###
Archive CI/CD data into partitioned database schem
a
After we move CI/CD metadata to a different store, the problem of having
After we move CI/CD metadata to a different store, the problem of having
billions of rows describing pipelines, build and artifacts, remains. We still
billions of rows describing pipelines, build and artifacts, remains. We still
...
...
doc/architecture/blueprints/ci_data_decay/pipeline_data_time_decay.png
0 → 100644
View file @
4fd5ab16
38.4 KB
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment