Large product catalogues

With a large catalogue, the number of products itself is rarely the problem. What matters far more is how Magento or another e-commerce platform stores, calculates, indexes and serves data as the scale grows.

We design new Magento and e-commerce platforms for large catalogues and rebuild existing solutions where filtering, search, the database or the entire store slows down as data volumes grow.

A free 30–60 minute call. We will learn about the scale of your catalogue, its data structure, and the import and indexing processes. We will identify the areas that may be limiting performance and further growth.

Catalogue complexity

A large catalogue is not a single number

We do not define an SKU threshold at which a catalogue becomes “large”.

Actual complexity also depends on the data model, the number and use of attributes, variants, store views, markets, price lists, the indexing approach, the search engine and the integrations that feed the catalogue.

A catalogue with a moderate number of products but complex variants, multiple markets and different pricing models can be more demanding for Magento than a much larger but simpler catalogue.

That is why we do not begin a project assessment by asking, “How many products are there?” What matters more is how the data is structured and how much work Magento must do to process it.

Two scenarios

When a large catalogue requires a specialised approach

01 · New implementation

Large scale from the outset

The first scenario is a new implementation designed to handle large scale from the outset.

The data model, indexing approach and integration architecture are designed before the platform goes into production. This is the right time to avoid decisions that cause no problems with a smaller catalogue but become expensive to change later.

02 · Existing Magento platform

Performance declines as data volumes grow

The second scenario is an existing Magento platform whose performance deteriorates as data volumes grow.

The store used to work properly, but over time filtering and search begin to slow down, database load increases and catalogue operations take longer and longer. In this situation, we need to identify which earlier decision has stopped scaling.

Symptoms and causes

What most often starts causing problems

Typical symptoms include slow filtering and search, increasing database load and deteriorating performance across the entire store.

Complex attributes and variants, multiple store views, separate markets and different price lists add another layer of complexity.

However, the visible problem may have an entirely different cause: a suboptimal data model, inefficient SQL queries, unnecessary data recalculation or indexing, or cache and infrastructure configured in a way that does not match how the platform operates.

We therefore do not assume that a large catalogue must itself be the problem. We also analyse the performance of the entire Magento platform and look for the actual bottleneck.

Data architecture

A data model that withstands growth

At a smaller scale, some architectural decisions have no noticeable consequences. Problems emerge later, when the same operations begin to run across a much larger number of records.

We analyse how products, attributes and the relationships between them are modelled. If the current data structure is the source of the problem, rebuilding it is also an option.

This is not always necessary. Some problems can be solved by changing queries, indexing or the way OpenSearch is used. Rebuilding the model makes sense when further optimisation around it would merely postpone the problem.

Catalogue performance

Database, indexing and search

With large catalogues, these areas are closely interconnected.

We analyse database queries and check whether Magento is retrieving or recalculating data it does not need at that moment. We also review the indexing approach and the use of OpenSearch or Elasticsearch.

For filtering, the number of products is not the only factor. The number of values users can filter by may be just as important.

Project example

In one project, we had to design support for hundreds of thousands of person-related values-authors, translators, illustrators, scriptwriters and characters-used in filters, content pages and relationships with product pages.

Loading complete lists of these values as standard attribute options was no longer scalable. We built a dedicated mechanism using OpenSearch that handled the data without placing the entire cost on Magento’s standard catalogue layer.

Multi-store

Multiple stores, markets and price lists

A large catalogue becomes more demanding when the same products operate across several store views or multiple markets.

Prices, availability, content, catalogue configurations and sales rules may all differ.

In this model, we need to consider not only how the data is stored, but also how many variants of the same data Magento and the search engine must process.

This is particularly important in B2B projects and for platforms operating across several markets at the same time.

Scale in practice

Scale is more than the number of products

~500k

products in the source database

~250k

active products

300k+

person records

3

storefronts

In one of our projects, the source database contained approximately 500,000 products. After organising the data and removing inactive items, the target catalogue contained around 250,000 active products.

The source system also contained more than 300,000 person records, while the platform supported three storefronts.

Moving this number of records was not the greatest challenge.

The real problem was historical data: different representations of the same value treated as separate records, constraints arising from preserving the existing SEO structure, and integration with an ERP system and its long-standing API.

In projects like these, organising and normalising data can be just as important as the development itself.

Scope of work

How we work with large catalogues

The scope depends on the situation.

We can analyse and optimise the data model, database and SQL queries, indexing, OpenSearch, search, filtering, cache and infrastructure.

We also measure performance to verify whether the changes actually solve the problem. Not every measurement needs to become a formal report-it is often a tool used in the team’s day-to-day work.

For large platforms, this is not always a one-off project. The catalogue grows and new markets, price lists and integrations are added, so performance and the way data is handled also require ongoing oversight as the project develops.

Experience

Why Velocity

With large catalogues, knowledge of a single Magento layer is rarely enough.

You need to understand the relationships between the data model, database, indexing, search engine, cache and infrastructure.

We have experience with catalogues containing hundreds of thousands of products, multiple storefronts, complex legacy data and ERP integrations.

The greatest value of this experience becomes clear when deciding whether a problem can be solved through configuration and optimisation or requires a deeper change to the data model or architecture.

Fit

When the problem lies elsewhere

The problem concerns overall performance

If the store is slow but catalogue scale is not a significant part of the problem, Magento performance optimisation is the right starting point.

Magento performance optimisation →

The presentation layer is the problem

If the backend responds quickly but the presentation layer is heavy or difficult to develop, it is worth considering Hyvä for Magento.

Hyvä for Magento →

The platform is unstable

If the platform is generally unstable, errors keep returning and the project’s technical condition is unclear, a Magento takeover and stabilisation may be needed first.

Magento takeover and stabilisation →

A large catalogue alone is not a reason to rebuild. If search, filtering and indexing work correctly, there is no value in optimising the system solely because of the number of products.

Questions

FAQ

VLCT

There is no single threshold. Project complexity also depends on attributes, variants, the number of values, store views, markets, price lists and the integration model. The SKU count alone says little about the actual load on Magento.

VLCT

Yes. We have worked with catalogues of this size. At scale, the data model, indexing, search engine, database and integration with other systems become particularly important.

VLCT

Not always. Sometimes changing queries, the indexing strategy or the way OpenSearch is used is enough. We recommend rebuilding the model when the current structure is the source of the problem.

VLCT

Yes. In migrations like these, organising, mapping and normalising data, as well as preserving the required URLs and relationships, is usually an important part of the project.

Migration and replatforming →
VLCT

Not always. A PIM makes sense when it addresses a genuine problem with organising and managing product data. The SKU count alone is not sufficient reason to implement one.

VLCT

It depends on the source of the problem. Query optimisation has a different scope from rebuilding the data model or migrating a large catalogue. We define the scope after an initial analysis of the platform and its data.