info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Keyset pagination
The keyset pagination library can be used in HAML-based views and the REST API within the GitLab project.
You can read about keyset pagination and how it compares to the offset based pagination on our [pagination guidelines](pagination_guidelines.md) page.
## API overview
### Synopsis
Keyset pagination with `ActiveRecord` in Rails controllers:
```ruby
cursor=params[:cursor]# this is nil when the first page is requested
This library adds a single method to ActiveRecord relations: [`#keyset_paginate`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/active_record_keyset_pagination.rb).
This is similar in spirit (but not in implementation) to Kaminari's `paginate` method.
Keyset pagination works without any configuration for simple ActiveRecord queries:
- Order by one column.
- Order by two columns, where the last column is the primary key.
The library can detect nullable and non-distinct columns and based on these, it will add extra ordering using the primary key. This is necessary because keyset pagination expects distinct order by values:
```ruby
Project.order(:created_at).keyset_paginate.records# ORDER BY created_at, id
Project.order(:name).keyset_paginate.records# ORDER BY name, id
Project.order(:created_at,id: :desc).keyset_paginate.records# ORDER BY created_at, id
Project.order(created_at: :asc,id: :desc).keyset_paginate.records# ORDER BY created_at, id DESC
```
The `keyset_paginate` method returns [a special paginator object](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/pagination/keyset/paginator.rb) which contains the loaded records and additional information for requesting various pages.
The method accepts the following keyword arguments:
-`cursor` - Encoded order by column values for requesting the next page (can be `nil`).
-`per_page` - Number of records to load per page (default 20).
-`keyset_order_options` - Extra options for building the keyset paginated database query, see an example for `UNION` queries in the performance section (optional).
The paginator object has the following methods:
-`records` - Returns the records for the current page.
-`has_next_page?` - Tells whether there is a next page.
-`has_previous_page?` - Tells whether there is a previous page.
-`cursor_for_next_page` - Encoded values as `String` for requesting the next page (can be `nil`).
-`cursor_for_previous_page` - Encoded values as `String` for requesting the previous page (can be `nil`).
-`cursor_for_first_page` - Encoded values as `String` for requesting the first page.
-`cursor_for_last_page` - Encoded values as `String` for requesting the last page.
- The paginator objects includes the `Enumerable` module and delegates the enumerable functionality to the `records` method/array.
Example for getting the first and the second page:
```ruby
paginator=Project.order(:name).keyset_paginate
paginator.to_a# same as .records
cursor=paginator.cursor_for_next_page# encoded column attributes for the next page
paginator=Project.order(:name).keyset_paginate(cursor: cursor).records# loading the next page
```
Since keyset pagination does not support page numbers, we are restricted to go to the following pages:
- Next page
- Previous page
- Last page
- First page
#### Usage in Rails with HAML views
Consider the following controller action, where we list the projects ordered by name:
The performance of the keyset pagination depends on the database index configuration and the number of columns we use in the `ORDER BY` clause.
In case we order by the primary key (`id`), then the generated queries will be efficient since the primary key is covered by a database index.
When two or more columns are used in the `ORDER BY` clause, it's advised to check the generated database query and make sure that the correct index configuration is used. More information can be found on the [pagination guideline page](pagination_guidelines.md#index-coverage).
NOTE:
While the query performance of the first page might look good, the second page (where the cursor attributes are used in the query) might yield poor performance. It's advised to always verify the performance of both queries: first page and second page.
Example database query with tie-breaker (`id`) column:
`OR` queries are difficult to optimize in PostgreSQL, we generally advise using [`UNION` queries](../sql.md#use-unions) instead. The keyset pagination library can generate efficient `UNION` when multiple columns are present in the `ORDER BY` clause. This is triggered when we specify the `use_union_optimization: true` option in the options passed to `Relation#keyset_paginate`.
Common `ORDER BY` configurations will be handled by the `keyset_paginate` method automatically so no manual configuration is needed. There are a few edge cases where order object configuration is necessary:
-`NULLS LAST` ordering.
- Function-based ordering.
- Ordering with a custom tie-breaker column, like `iid`.
These order objects can be defined in the model classes as normal ActiveRecord scopes, there is no special behavior that prevents using these scopes elsewhere (kaminari, background jobs).
# SELECT "issues".* FROM "issues" WHERE "issues"."project_id" = 10 ORDER BY relative_position DESC NULLS LAST
scope.keyset_paginate# raises: Gitlab::Pagination::Keyset::Paginator::UnsupportedScopeOrder: The order on the scope does not support keyset pagination
```
The `keyset_paginate` method raises an error because the order value on the query is a custom SQL string and not an [`Arel`](https://www.rubydoc.info/gems/arel) AST node. The keyset library cannot automatically infer configuration values from these kinds of queries.
To make keyset pagination work, we need to configure custom order objects, to do so, we need to collect information about the order columns:
-`relative_position` can have duplicated values since no unique index is present.
-`relative_position` can have null values because we don't have a not null constraint on the column. For this, we need to determine where will we see NULL values, at the beginning of the resultset or the end (`NULLS LAST`).
- Keyset pagination requires distinct order columns, so we'll need to add the primary key (`id`) to make the order distinct.
- Jumping to the last page and paginating backwards actually reverses the `ORDER BY` clause. For this, we'll need to provide the reversed `ORDER BY` clause.
Example:
```ruby
order=Gitlab::Pagination::Keyset::Order.build([
# The attributes are documented in the `lib/gitlab/pagination/keyset/column_order_definition.rb` file
The `add_to_projections` flag tells the paginator to expose the column expression in the `SELECT` clause. This is necessary because the keyset pagination needs to somehow extract the last value from the records to request the next page.
### `iid` based ordering
When ordering issues, the database ensures that we'll have distinct `iid` values within a project. Ordering by one column is enough to make the pagination work if the `project_id` filter is present:
@@ -58,9 +58,7 @@ It's not possible to make all filter and sort combinations performant, so we sho
### Prepare for scaling
Offset-based pagination is the easiest way to paginate over records, however, it does not scale well for large tables. As a long-term solution, keyset pagination is preferred. The tooling around keyset pagination is not as mature as for offset pagination so currently, it's easier to start with offset pagination and then switch to keyset pagination.
To avoid losing functionality and maintaining backward compatibility when switching pagination methods, it's advised to consider the following approach in the design phase:
Offset-based pagination is the easiest way to paginate over records, however, it does not scale well for large database tables. As a long-term solution, [keyset pagination](keyset_pagination.md) is preferred. Switching between offset and keyset pagination is generally straightforward and can be done without affecting the end-user if the following conditions are met:
- Avoid presenting total counts, prefer limit counts.
- Example: count maximum 1001 records, and then on the UI show 1000+ if the count is 1001, show the actual number otherwise.
...
...
@@ -304,7 +302,22 @@ LIMIT 20
##### Tooling
Using keyset pagination outside of GraphQL is not straightforward. We have the low-level blocks for building keyset pagination database queries, however, the usage in application code is still not streamlined yet.
A generic keyset pagination library is available within the GitLab project which can most of the cases easly replace the existing, kaminari based pagination with significant performance improvements when dealing with large datasets.
