Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
548767f7
Commit
548767f7
authored
Feb 01, 2022
by
Grzegorz Bizon
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Update CI/CD time decay blueprint content and update links
parent
2b8b8fa7
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
18 additions
and
17 deletions
+18
-17
doc/architecture/blueprints/ci_data_decay/index.md
doc/architecture/blueprints/ci_data_decay/index.md
+18
-17
No files found.
doc/architecture/blueprints/ci_data_decay/index.md
View file @
548767f7
...
@@ -74,13 +74,13 @@ we might want to follow these three tracks described below.
...
@@ -74,13 +74,13 @@ we might want to follow these three tracks described below.
<!-- markdownlint-disable MD029 -->
<!-- markdownlint-disable MD029 -->
1.
Partition
builds queuing
tables
1.
Partition
CI/CD builds queuing database
tables
2.
Archive CI/CD data into partitioned database schema
2.
Partition CI/CD pipelines database tables
3.
Migrate archived builds metadata out of primary database
3.
Reduce the rate of builds metadata table growth
<!-- markdownlint-enable MD029 -->
<!-- markdownlint-enable MD029 -->
###
Migrate archived builds metadata out of primary database
###
Reduce the rate of builds metadata table growth
Once a build (or a pipeline) gets archived, it is no longer possible to resume
Once a build (or a pipeline) gets archived, it is no longer possible to resume
pipeline processing in such pipeline. It means that all the metadata, we store
pipeline processing in such pipeline. It means that all the metadata, we store
...
@@ -98,15 +98,16 @@ be able to use de-duplication of metadata entries and other normalization
...
@@ -98,15 +98,16 @@ be able to use de-duplication of metadata entries and other normalization
strategies to consume less storage while retaining ability to query this
strategies to consume less storage while retaining ability to query this
dataset. Technical evaluation will be required to find the best solution here.
dataset. Technical evaluation will be required to find the best solution here.
Epic:
[
Migrate archived builds metadata out of primary database
](
https://gitlab.com/groups/gitlab-org/-/epics/7216
)
.
Epic:
[
Reduce the rate of builds metadata table growth
](
https://gitlab.com/groups/gitlab-org/-/epics/7434
)
.
###
Archive CI/CD data into partitioned database schema
###
Partition CI/CD pipelines database tables
After we move CI/CD metadata to a different store, the problem of having
After we move CI/CD metadata to a different store, or reduce the rate of
billions of rows describing pipelines, builds and artifacts, remains. We still
metadata growth in a different way, the problem of having billions of rows
need to keep reference to the metadata we store in object storage and we still
describing pipelines, builds and artifacts, remains. We still need to keep
do need to be able to retrieve this information reliably in bulk (or search
reference to the metadata we might store in object storage and we still do need
through it).
to be able to retrieve this information reliably in bulk (or search through
it).
It means that by moving data to object storage we might not be able to reduce
It means that by moving data to object storage we might not be able to reduce
the number of rows in CI/CD tables. Moving data to object storage should help
the number of rows in CI/CD tables. Moving data to object storage should help
...
@@ -132,9 +133,9 @@ partitioning on the application level.
...
@@ -132,9 +133,9 @@ partitioning on the application level.
Partitioning rarely accessed data should also follow the policy defined for
Partitioning rarely accessed data should also follow the policy defined for
builds archival, to make it consistent and reliable.
builds archival, to make it consistent and reliable.
Epic:
[
Archive CI/CD data into partitioned database schema
](
https://gitlab.com/groups/gitlab-org/-/epics/5417
)
.
Epic:
[
Partition CI/CD pipelines database tables
](
https://gitlab.com/groups/gitlab-org/-/epics/5417
)
.
### Partition
builds queuing
tables
### Partition
CI/CD builds queuing database
tables
While working on the
[
CI/CD Scale
](
../ci_scale/index.md
)
blueprint, we have
While working on the
[
CI/CD Scale
](
../ci_scale/index.md
)
blueprint, we have
introduced a
[
new architecture for queuing CI/CD builds
](
https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908
)
introduced a
[
new architecture for queuing CI/CD builds
](
https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908
)
...
@@ -156,7 +157,7 @@ for builds archival. Instead we should leverage a long-standing policy saying
...
@@ -156,7 +157,7 @@ for builds archival. Instead we should leverage a long-standing policy saying
that builds created more 24 hours ago need to be removed from the queue. This
that builds created more 24 hours ago need to be removed from the queue. This
business rule is present in the product since the inception of GitLab CI.
business rule is present in the product since the inception of GitLab CI.
Epic:
[
Partition
builds queuing
tables
](
https://gitlab.com/groups/gitlab-org/-/epics/7438
)
.
Epic:
[
Partition
CI/CD builds queuing database
tables
](
https://gitlab.com/groups/gitlab-org/-/epics/7438
)
.
## Principles
## Principles
...
@@ -215,9 +216,9 @@ pipelines data, although a user provided partition identifier may be required.
...
@@ -215,9 +216,9 @@ pipelines data, although a user provided partition identifier may be required.
All three tracks can be worked on in parallel:
All three tracks can be worked on in parallel:
1.
[
Migrate archived build metadata to object storage
](
https://gitlab.com/groups/gitlab-org/-/epics/7216
)
.
1.
[
Reduce the rate of builds metadata table growth
](
https://gitlab.com/groups/gitlab-org/-/epics/7434
)
.
1.
[
Partition CI/CD
data that have been archived
](
https://gitlab.com/groups/gitlab-org/-/epics/5417
)
.
1.
[
Partition CI/CD
pipelines database tables
](
https://gitlab.com/groups/gitlab-org/-/epics/5417
)
.
1.
[
Partition CI/CD queuing tables using list partitioning
](
https://gitlab.com/g
itlab-org/gitlab/-/issues/347027
)
1.
[
Partition CI/CD queuing tables using list partitioning
](
https://gitlab.com/g
roups/gitlab-org/-/epics/7438
)
## Status
## Status
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment