What are the Slack Archives?

It’s a history of our time together in the Slack Community! There’s a ton of knowledge in here, so feel free to search through the archives for a possible answer to your question.

Because this space is not active, you won’t be able to create a new post or comment here. If you have a question or want to start a discussion about something, head over to our categories and pick one to post in! You can always refer back to a post from Slack Archives if needed; just copy the link to use it as a reference..

What is the relationship with `publish:trigger-events` and `sync:data` ? If I run `publish:trigger-e

U03T08LE3CP
U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

What is the relationship with publish:trigger-events and sync:data ? If I run publish:trigger-events when do I need to run sync:data?

Β«1

Comments

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    publish:trigger-events triggers all propel entity events (as far as I understand, probably the update event for all entities) which leads to the first step of publish / synchronize being executed: the denormalized data is written to the _storage and _search tables in the database. This then automatically triggers the events that lead to the sync part of the publish / sync: writing all the data from _storage tables to redis and all the data from _search tables to elasticsearch. the sync:data command only does the last part, transfering the data from _storage and _search tables to redis and elasticsearch (while _storage and _search are only naming conventions. I think they can be named differently, and also there are at least 1 or 2 tables with _search suffix that do something different)

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    so if you only want to sync for example from spy_product_concrete_page_search to elasticsearch, sync:data (-r product_concrete I think also works) would be sufficient. if spy_product_concrete_page_search does not contain the latest data, you need publish:trigger-events (-r product_concrete) to publish the data from spy_product to spy_product_concrete_page_search

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    so if I do say, manipulations to image sets in the database directly, should it suffice to publish:trigger-events -r product_abstract_image?

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    yes, it should πŸ‘ (I think πŸ˜› )

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    I'm starting to get really hesitant to make any SQL changes since I cannot seem to get the indices update cleanly..

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    now the imageSets value in glue abstract_products is empty and I don't know why

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    hm ok. So you can go step by step from the database side, and look at the spy_product_abstract_image_storage table and see if it contains the correct data

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    and if yes, the next step would be to look into the storage (probably redis). you can get the redis key for this from the spy_product_abstract_image_storage table

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    or one question: are you querying one specific product or a list of products? if it is a list it might be _search table and elasticsearch instead

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    queyring ong product

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet
    edited May 2023

    /abstract-products/001_4059729207241?include=concrete-products,abstract-product-image-sets,concrete-product-prices,concrete-product-availabilities,category-nodes,concrete-product-image-sets

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    something like that

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    then the data should come from the redis storage which is (or should be) synced from the spy_product_abstract_image_storage table. I'm not 100% sure about the glue part (because I did not use glue myself, yet): You included the image sets, maybe there is also another include needed for the actual image data?

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    I know this is right it just doesn't reflect the db changes

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    while there is only one storage table for abstract images so probably they are combined during publish

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    then you should look at the spy_product_abstract_image_storage table and search for the product to see if this is correct

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    you might also want to look at rabbitmq, if there are any messages on one of the publish.error queues. If so, you can look at the messages in the rabbitmq UI and see if they contain an error message (if something is going wrong during publish)

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    storage looks right

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    error queues are empty

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    then the problem is probably that the data is not updated to redis (so the sync:data part). Depending on your infrastructure you might be able to use some client to connect to it and look at the values. But there is also a UI in Zed to access it under maintenance -> storage I think

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    you can use the key for the data set from the spy_product_abstract_image_storage table and search for it in the storage UI

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    (in Zed it must be an exact match of the key I think)

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    yeah it's redis πŸ™‚

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    but I have tinkered so that I have cli access to it as well

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    ok, I'm not very familiar with the cli (other then deleting everything πŸ˜„ ), but you should be able to search for the key there quite easily

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    I think its just GET ... πŸ˜›

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    the root cause might be that I didn't quite get how the product-image connection works

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    in glue?

  • U03T08LE3CP
    U03T08LE3CP Posts: 185 πŸ§‘πŸ»β€πŸš€ - Cadet

    in the database

  • U01BZ7Q3XRV
    U01BZ7Q3XRV Posts: 148 πŸ§‘πŸ»β€πŸš€ - Cadet

    ah ok