What are the Slack Archives?

It’s a history of our time together in the Slack Community! There’s a ton of knowledge in here, so feel free to search through the archives for a possible answer to your question.

Because this space is not active, you won’t be able to create a new post or comment here. If you have a question or want to start a discussion about something, head over to our categories and pick one to post in! You can always refer back to a post from Slack Archives if needed; just copy the link to use it as a reference..

Hello, we experienced an issue today that might concern us in the future. Our customer has abstract

UPWG9AYH2
UPWG9AYH2 Posts: 509 🧑🏻‍🚀 - Cadet

Hello,
we experienced an issue today that might concern us in the future.
Our customer has abstract products that get drilled down to a concrete product using many super attributes. Today we had 9 in sum, with two options each … which is not that much in my opinion.
However, with 8 attributes we had no problems before, but today with 9 we got a memory allocation issue in the AttributePermutationGenerator while generating the attribute permutations for the product detail page. The limit of 1024MB was exceeded.
So, yes, this one case is solvable increasing the memory limit. But I am worry about further limitations since it’s very likely that we get additional attributes.
Also, this permutation generation stuff seems not to progress linear because the problem at all is not a linear problem. The memory consumption will explode at a specific point.

So my question here: What are the limitations? Any experiences from other projects handling many super attributes?

Best

«1

Comments

  • Jeremy Fourna
    Jeremy Fourna Lead Product Manager Posts: 130 🧑🏻‍🚀 - Cadet

    @UPWG9AYH2 could you create a support ticket for it, we will need to solve this problem.
    cc: @valerii.trots

  • I can confirm that issue

  • U0145CP9Q6Q
    U0145CP9Q6Q Posts: 123 🧑🏻‍🚀 - Cadet
    edited November 2020

    @UPWG9AYH2 try to make super attributes as simple text types. Without options. If possible. I had the same problem and as i remember it helped

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    I'm facing the same issue. Changing the super attributes' type doesn't help.
    What's the progress on this one? Is there a ticket number I can refer to?

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    Doesn't work. After updating the packages, I'm getting redis connect errors. So I tried to resetup the docker env and ended up with this:

    Store: AT | Environment: docker.dev
    Build Yves frontend
    
    
    
      spryker-b2b-demo-shop@1.0.0 yves /data
  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    Seems that the shop-ui module depends on a missing npm package.

  • UPWG9AYH2
    UPWG9AYH2 Posts: 509 🧑🏻‍🚀 - Cadet
  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice

    Thanks Ingo, you were faster. 🙂

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    So the recommended solution is to downgrade the shop-ui and lose updatability? Am I getting it right?

  • UPWG9AYH2
    UPWG9AYH2 Posts: 509 🧑🏻‍🚀 - Cadet

    As far as i can see, the solution is to add the missing dependency to your project level package.json

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice

    You need to fix package.json as

    Unfortunately we cannot force npm dependencies update using composer.
    
  • U01LE4BMBK7
    U01LE4BMBK7 Posts: 241 🧑🏻‍🚀 - Cadet

    thanks for the link @valerii.trots

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet
    edited February 2021

    Thanks. We're one step further now. But now the Jenkins jobs aren't set up anymore.

  • UPWG9AYH2
    UPWG9AYH2 Posts: 509 🧑🏻‍🚀 - Cadet

    Whats the error message? Seems to me like the usual stuff that happen when updates gettin applied 🙂

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    No error message. When I open scheduler.spryker.local, there are no jobs. Even after running docker/sdk console scheduler:setup manually.

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    edited February 2021

    docker/sdk console scheduler:setup -vvv?

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    Seems to work now. Here's what I did:

    docker/sdk cli composer require filp/whoops spryker/product-storage:"^1.31.0" spryker-shop/product-detail-page:"^3.12.0" spryker/product:"^6.22.1" spryker/product-attribute:"^1.8.1"
    docker/sdk cli npm install password-validator
    

    It didn't work until I ran the composer update without the --update-with-dependencies option.
    I needed to add the filp/whoops package, too, since it was moved to dev dependencies recently, which obviously is wrong.

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice

    which obviously is wrong

    Nope. It's just a part of some updates\migration path that isn't taken for the project yet.

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    At first glance, it looked as if it would work, but it doesn't. Still failing at the same place:

    [console] [cli] - [2021-02-09T09:01:29.657080+00:00] Zed.CRITICAL: ErrorException - FATAL ERROR - Allowed memory size of 4294967296 bytes exhausted (tried to allocate 20480 bytes) in "/data/vendor/spryker/product/src/Spryker/Zed/Product/Business/Product/Variant/AttributePermutationGenerator.php::55" {"exception":"[object] (ErrorException(code: 0): FATAL ERROR - Allowed memory size of 4294967296 bytes exhausted (tried to allocate 20480 bytes) at /data/vendor/spryker/product/src/Spryker/Zed/Product/Business/Product/Variant/AttributePermutationGenerator.php:55)
    
  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice
    edited February 2021

    How many super attributes do you use?

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    There are products with seven super attributes. There might be even more.

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice

    This "fix" works with at least 9 super attributes.
    Could you please provide exact steps to reproduce?

    @jeremy.fourna FYI.

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    I don't think that it's safe to say works with at least x super attributes. It's not only about the number of super attributes, but also the amount of different values. You could have for example two super attributes only, but if each of them is having ten values, things will go crazy again.
    And that's my actual issue. The combination of super attributes and their values makes the php memory consumption grow exponentially.

    The steps that I'm doing are pretty basic. Once I'm having everything in the db, I simply call docker/sdk console publish:trigger-events. In rabbitmq I can see the publish messages coming in and being processed. Once it gets to the abstract product publish messages, it will exceed the memory. The queue is getting stuck then, i. e. it's picking the same bulk of 500 messages over and over again and retries forever.

  • UPWG9AYH2
    UPWG9AYH2 Posts: 509 🧑🏻‍🚀 - Cadet

    My “experience” with this issue was, that the problem is growing more worse with different attributes than with different values.

  • Valerii Trots
    Valerii Trots SRE @ Spryker Sprykee Posts: 1,654 ✨ - Novice

    I may be wrong but this "fix" has been developed specifically for Ingo's case as a quick win. Proper fix will follow as a big refactoring of the area.

    @jeremy.fourna could you confirm this?

  • Jeremy Fourna
    Jeremy Fourna Lead Product Manager Posts: 130 🧑🏻‍🚀 - Cadet
  • Jeremy Fourna
    Jeremy Fourna Lead Product Manager Posts: 130 🧑🏻‍🚀 - Cadet

    @valerii.trots the solution is not supposed to be a quick fix, we normally solved the root cause

  • U01LKKBK97T
    U01LKKBK97T Posts: 287 🧑🏻‍🚀 - Cadet

    Can't confirm, unfortunately.