I have a P&S issue. The product data in ElasticSearch is out of sync with the database. I’m tryi

UNTTWV4JK

@UK9N7MP96 I will try to get the propel log/reproduce it on local but it will take some time.

giovanni.piemontese

which version of product-page-search do u have?

this version https://github.com/spryker/product-page-search/releases/tag/3.9.0. fix some memory problem (come from 2 week ago)

Ehsan Zanjani

@UL65CH0MC that fixed the publishing memory issue, but it looks @UNTTWV4JK has memory issue on synchronization part

UNTTWV4JK

@UK9N7MP96 this is the full error ```
Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 12288 bytes) in /data/vendor/propel/propel/src/Propel/Runtime/Connection/StatementWrapper.php on line 238
ErrorException - Exception: FATAL ERROR - Allowed memory size of 2147483648 bytes exhausted (tried to allocate 12288 bytes)
in /data/vendor/propel/propel/src/Propel/Runtime/Connection/StatementWrapper.php (238)

Command: vendor/bin/console sync:data product_abstract 26940

Trace:

0 /data/vendor/spryker/error-handler/src/Spryker/Shared/ErrorHandler/ErrorHandlerEnvironment.php(99): Spryker\Shared\ErrorHandler\ErrorHandler->handleFatal()

1 [internal function]: Spryker\Shared\ErrorHandler\ErrorHandlerEnvironment->Spryker\Shared\ErrorHandler{closure}()

```

giovanni.piemontese

@UK9N7MP96 u are right...

Ehsan Zanjani

@UNTTWV4JK this line is not in the full stack

Ehsan Zanjani

however that doesn’t tell too much

UNTTWV4JK

AFAIK the P&S is building pairs between the storage entries and the entities to determine what need to be saved. 38 entries from the spy_product_abstract_page_search paired with the entities… this will use a lot of memory before starting to save the data in elastic

Ehsan Zanjani

True, but this is happening in publishing part when you run queue:worker:start or queue:task:start event

Ehsan Zanjani

sync:data is responsible to query the product_abstract_page_search or product_abstract_storage to just return the value of data column and write them in to the queue

giovanni.piemontese

I have 2,5Mio Products with 7000Attributes and other stuff and just test on my system the console command synch product abstract and it seems that everything works fine.. .BUT: i have only 1Store, 1Language, 1Currency now..

UNTTWV4JK

we have more

UNTTWV4JK

20 stores, 8 locales, 8 currencies, 2 price types…

giovanni.piemontese

and only 38 products? right?

giovanni.piemontese

or only 38 are published and u have more?

UNTTWV4JK

no, these are the search entries for one product

UNTTWV4JK

is combination of stores/locale

UNTTWV4JK

for one product

giovanni.piemontese

ok understood... yes i know your environment..

I noted on my system that i give as u only 1 id to be published but i see that the command just fetch all products... can u please check if before the memory fatal error u see some events on rmq under sync.search.product?

Ehsan Zanjani

@UNTTWV4JK which version of spryker/product-page-search are you using?

Ehsan Zanjani

I found a bug for sync:data for specific ids here
Spryker/Zed/Synchronization/Business/Export/ExporterPluginResolver.php:72
$ids is missing

Ehsan Zanjani

will report it now

Ehsan Zanjani

@UNTTWV4JK can you reduce the chunk size?

Ehsan Zanjani

you can do it by adding this line into your config_default-production.php

$config[\Spryker\Shared\Synchronization\SynchronizationConstants::EXPORT_MESSAGE_CHUNK_SIZE] = 50;

Ehsan Zanjani

this is the query which works with chunking and avoid any memory issue, if you still get memory issue you can reduce the chunk size

Ehsan Zanjani

\Spryker\Zed\Synchronization\Business\Iterator\SynchronizationDataQueryContainerPluginIterator

UNTTWV4JK

Thanks for suggestions, I will try tomorrow.

UNTTWV4JK

FYI: The memory issue is gone after adding the $ids in the method.

Ehsan Zanjani

Yes that will be fixed in new release for that module, but without id you should not see any memory issue as well