Hi all, I am facing an issue with publish&synchronize out of nowhere where I need your help. Tri

U01F7P3D9NH

Hi all, I am facing an issue with publish&synchronize out of nowhere where I need your help. Triggering all abstracts ends in an infinite loop. F.e. 2000 msg in event queue getting unacked, pushes msgs successfully to sync.storage and then event msgs getting back to ready. No error, no retry. Pushing smaller batches (10-20 abstracts) mostly getting processed. Tried to track them down to the faulty one, thought I found it, but trigger the individual item worked also. With 500k msgs, it is not a good workaround 🙂 Chunksizes are limited already. Any ideas?

U01BZ7Q3XRV

could be memory limit or OOM (out of memory) error if chunks are to big. when it's "only" memory limit, you can try to start more workers in parallel (see config: \Spryker\Shared\Queue\QueueConstants::QUEUE_ADAPTER_CONFIGURATION) instead of having too big chunk sizes. You can also try reducing event queue chunk size and increasing only publish and storage queues chunk sizes (especially writing to storage can usually be much bigger), but this depends a bit on your implementation how and if this is possible by configuration

giovanni.piemontese

product_abstract P+S has in Spryker a problem about a lot of memory allocation and cpu time... the problem is mainly related to the attributes...

how many attributes do u have in your system imported?

Valerii Trots

I'll check php error logs in the container that are not pushed to cloudwatch yet and which you can't check on your own because of that.

Valerii Trots

There are none. 🤔

Valerii Trots

And last OOM killer involvement was on 1st of November.

Valerii Trots

U01F7P3D9NH

@valerii.trots it is reproducable on local env, so you could tell me where to look. @UL65CH0MC it is a difference how many abstracts are pushed in queue? because with 10 abstracts it works fine

giovanni.piemontese

@U01F7P3D9NH, yes, absolutely

U01F7P3D9NH

1,6k attributes in the table shouldn´t break everything 🙂

giovanni.piemontese

on my project i optimized it and reduced the memory to standard ca. 120MB per process and execution time to a few seconds instead minutes and ca,2,5GB for every chunk...

giovanni.piemontese

1,6K is not lot, but can be the problem... There are some parts of codes where spryker make a lot of foreach wihtout sense and load a lot of not necessary data from db etc etc...

giovanni.piemontese

I have also created since 1 year a support ticket and attached the pyz fix, but till now no fix in core...

giovanni.piemontese

even if it is the same problem.. but i suppose yes, it is (i think)

U01F7P3D9NH

i´ll try it now with chunk limit of 5 but it looks kinda the same, storage queues are filled and processed but "main" message stays

giovanni.piemontese

what events are in the main message? can u please get a message and see the type/name?

U01F7P3D9NH

{"listenerClassName":"Spryker\\Zed\\ProductStorage\\Communication\\Plugin\\Event\\Listener\\ProductAbstractStoragePublishListener","transferClassName":"Generated\\Shared\\Transfer\\EventEntityTransfer","transferData":{"id":"6","event":null,"name":null,"foreign_keys":[],"modified_columns":[],"original_values":[],"additional_values":[]},"eventName":"Product.product_abstract.publish"}

giovanni.piemontese

but it is a product storage.. and u said that the storage msgs are consumed properly... or not?

U01F7P3D9NH

I paused one queue so you can see that things are processed

U01F7P3D9NH

I purge event queue and the other queues getting empty

giovanni.piemontese

how it is configured your sync chunk?

U01F7P3D9NH

// ---------- Event
$config[EventConstants::EVENT_CHUNK] = 5;

// ---------- Synchronization
$config[SynchronizationConstants::EXPORT_MESSAGE_CHUNK_SIZE] = 5;
$config[SynchronizationConstants::DEFAULT_SYNC_STORAGE_MESSAGE_CHUNK_SIZE] = 5;
$config[SynchronizationConstants::DEFAULT_SYNC_SEARCH_QUEUE_MESSAGE_CHUNK_SIZE] = 5;

giovanni.piemontese

the problem that i explained before was more related to product_abstract_search and product_concrete_search publish events... but in your case it seems that you have probably timeouts by sync process and therefore u not get the ack status...

U01F7P3D9NH

super strange... how can 100k concrete with chunk 100 running smooth vs 10 abstracts and chunk 5 fail without and sign of high mem or cpu 😄

U01F7P3D9NH

@U01BZ7Q3XRV & @UL65CH0MC Thanks guys for the help. Seems I got trolled by IDE or the Broker. During writing product abstract storage of high variant abstracts (around 5MB per row) it stoped in the middle (7 stores, 6 languages). IDE breakpoint just disappeared without any sign of error, also nothing in the log. Still dont know why, but only with chunksize of 1 the real reason appeared and rabbit container was killed by OOM. With

"memory_limit": 20480m

and nothing less it is working again and chunks are back to 100. @valerii.trots Could you unleash the EC2 beast for the weekend? 😄

Valerii Trots

Please create a support request and my colleagues will take care. Unfortunately I'm not as available as was previously due to position change and internal tasks I have.