Does Spryker plan to improve pagination support for categories with 10k+ products?

Roland
Roland Senior Application Engineer Posts: 2 🧑🏻‍🚀 - Cadet

Currently if there is more than 10k products in category and you try to switch to category page which shows product list ordered >10k you will see `fail whale` and get such error log:
```
Yves.CRITICAL: Spryker\Client\SearchElasticsearch\Exception\SearchResponseException - Search failed with the following reason: Result window is too large, from + size must be less than or equal to: [10000] but was [10008]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting. [reason: all shards failed].
```

From my research increasing the `index.max_result_window` is not recommended approach due to potential large CPU and memory resource needs.
Scroll API docs say:

```
The scroll API is no longer recommend for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).
```

It seems that `search_after` with PIT is currently the most recommended approach which seems not supported by SearchElasticsearch client yet (or am I wrong?). Hence my question.
Do Spryker plan to improve pagination approach for such case in near future?

Cheers.