with [John Northrup](https://gitlab.com/northrup), and live questions coming in from some of our customers.
## GitLab Components
The following components need to be considered for a scaled or highly-available
environment. In many cases components can be combined on the same nodes to reduce
environment. In many cases, components can be combined on the same nodes to reduce
complexity.
- Unicorn/Workhorse - Web-requests (UI, API, Git over HTTP)
...
...
@@ -57,12 +57,12 @@ infrastructure and maintenance costs of full high availability.
### Basic Scaling
This is the simplest form of scaling and will work for the majority of
cases. Backend components such as PostgreSQL, Redis and storage are offloaded
cases. Backend components such as PostgreSQL, Redis, and storage are offloaded
to their own nodes while the remaining GitLab components all run on 2 or more
application nodes.
This form of scaling also works well in a cloud environment when it is more
cost-effective to deploy several small nodes rather than a single
costeffective to deploy several small nodes rather than a single
larger one.
- 1 PostgreSQL node
...
...
@@ -85,11 +85,11 @@ you can continue with the next step.
### Full Scaling
For very large installations it may be necessary to further split components
for maximum scalability. In a fully-scaled architecture the application node
For very large installations, it might be necessary to further split components
for maximum scalability. In a fully-scaled architecture, the application node
is split into separate Sidekiq and Unicorn/Workhorse nodes. One indication that
this architecture is required is if Sidekiq queues begin to periodically increase
in size, indicating that there is contention or not enough resources.
in size, indicating that there is contention or there are not enough resources.
- 1 PostgreSQL node
- 1 Redis node
...
...
@@ -100,7 +100,7 @@ in size, indicating that there is contention or not enough resources.
## High Availability Architecture Examples
When organizations require scaling *and* high availability the following
When organizations require scaling *and* high availability, the following
architectures can be utilized. As the introduction section at the top of this
page mentions, there is a tradeoff between cost/complexity and uptime. Be sure
this complexity is absolutely required before taking the step into full
...
...
@@ -108,11 +108,11 @@ high availability.
For all examples below, we recommend running Consul and Redis Sentinel on
dedicated nodes. If Consul is running on PostgreSQL nodes or Sentinel on
Redis nodes there is a potential that high resource usage by PostgreSQL or
Redis nodes, there is a potential that high resource usage by PostgreSQL or
Redis could prevent communication between the other Consul and Sentinel nodes.
This may lead to the other nodes believing a failure has occurred and automated
failover is necessary. Isolating them from the services they monitor reduces
the chances of split-brain.
This may lead to the other nodes believing a failure has occurred and initiating
automated failover. Isolating Redis and Consul from the services they monitor
reduces the chances of a false positive that a failure has occurred.
The examples below do not really address high availability of NFS. Some enterprises
have access to NFS appliances that manage availability. This is the best case
...
...
@@ -131,7 +131,7 @@ trade-offs and limits.
This architecture will work well for many GitLab customers. Larger customers
may begin to notice certain events cause contention/high load - for example,
cloning many large repositories with binary files, high API usage, a large
number of enqueued Sidekiq jobs, etc. If this happens you should consider
number of enqueued Sidekiq jobs, and so on. If this happens, you should consider
moving to a hybrid or fully distributed architecture depending on what is causing
the contention.
...
...
@@ -162,32 +162,11 @@ contention due to certain workloads.
#### Reference Architecture
-**Supported Users (approximate):** 10,000
-**Known Issues:** While validating the reference architecture, slow endpoints were discovered and are being investigated. [See issue #64335](https://gitlab.com/gitlab-org/gitlab-foss/issues/64335)
The Support and Quality teams built, performance tested, and validated an
environment that supports about 10,000 users. The specifications below are a
representation of the work so far. The specifications may be adjusted in the
future based on additional testing and iteration.
NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size.
This architecture scales to hundreds of thousands of users and projects and is
the basis of the GitLab.com architecture. While this scales well it also comes
with the added complexity of many more nodes to configure, manage and monitor.
with the added complexity of many more nodes to configure, manage, and monitor.
- 3 PostgreSQL nodes
- 4 or more Redis nodes (2 separate clusters for persistent and cache data)
...
...
@@ -214,3 +193,59 @@ separately:
1.[Configure the GitLab application servers](gitlab.md)
1.[Configure the load balancers](load_balancer.md)
1.[Monitoring node (Prometheus and Grafana)](monitoring_node.md)
## Reference Architecture Examples
These reference architecture examples rely on the general rule that approximately 2 requests per second (RPS) of load is generated for every 100 users.
### 10,000 User Configuration
-**Supported Users (approximate):** 10,000
-**RPS:** 200 requests per second
-**Known Issues:** While validating the reference architecture, slow endpoints were discovered and are being investigated. [gitlab-org/gitlab-ce/issues/64335](https://gitlab.com/gitlab-org/gitlab-ce/issues/64335)
The Support and Quality teams built, performance tested, and validated an
environment that supports about 10,000 users. The specifications below are a
representation of the work so far. The specifications may be adjusted in the
future based on additional testing and iteration.
NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size.
expect(json_response['objects'].first['error']).toinclude('code'=>404,'message'=>"Object does not exist on the server or you don't have permissions to access it")
end
end
context'when downloading one new and one existing lfs object'do
expect(json_response['objects'].first['error']).toinclude('code'=>404,'message'=>"Object does not exist on the server or you don't have permissions to access it")
end
end
context'when downloading one new and one existing LFS object'do
let(:body){download_body(multiple_objects)}
let(:update_lfs_permissions)do
project.lfs_objects<<lfs_object
end
it'responds with status 200'do
expect(response).tohave_gitlab_http_status(200)
end
it_behaves_like'LFS http 200 response'
it'responds with upload hypermedia link for the new object'do
expect(json_response['objects'].first['error']).toinclude('code'=>404,'message'=>"Object does not exist on the server or you don't have permissions to access it")
end
end
context'when user is blocked'do
let(:role){:reporter}
let(:user){create(:user,:blocked)}
it_behaves_like'LFS http 401 response'
end
end
context'when using Deploy Tokens'do
let(:project){create(:project,:repository)}
let(:authorization){authorize_deploy_token}
let(:update_user_permissions){nil}
let(:role){nil}
...
...
@@ -587,25 +549,19 @@ describe 'Git LFS API and storage' do
