Good Morning, does anybody in this round has some practical experience with performance of P&S p

UPWG9AYH2

Good Morning,
does anybody in this round has some practical experience with performance of P&S process for on-demand requested items?
How long takes it in real to get the data from saving the database entry to redis/elasticsearch and is this timing stable? So for my theoretical understanding, the data could “wait” up to one minute until a new worker instance for p&s is started. But in practical it often does not feel that long.
Is there are also a way to speed this process up or to “enforce” a “synchronized” way?

Background: Fetching a customer specific price from an external system for a product triggered by a frontend call should immediately be available for further processes in redis/elasticsearch, so that the next frontend action can work with this prices from redis/elasticsearch. Here it is reasonable to have longer waiting time in frontend until the price is fetched. In my opinion this conflicts with the async behaviour of the whole spryker p&s concept. But maybe there is any concept to solve this problem.
Does anybody has experience with this?

Best

giovanni.piemontese

Hi Ingo, can i ask why do u want save/sync the customer prices for products in redis/ES? We have a similar concept, but we save the customer prices in own db redis after fetching (for some ttl) and then we have in post-save-plugins in cart just a plugin expander that read these data from redis if exists, otherwise perforn a new sap call, and then the price in cart are just the correct customer prices according logged customer.
Of course, there is more than one expander plugin that u have to write.. i.e. also for quick-order price-resolver etc...

UPWG9AYH2

Hi Giovanni, we also have/had this idea to save it for a while in an extra space in redis. But my idea here would be to have minimal implementation effort and use spryker ootb concepts as much as possible. So one idea would be to save customer specific prices as a new price dimension in spryker. But this requires the whole process from saving it in the db and publish it to the storage/es … this takes time and you cant ensure that on the next call the price is already published and ready to use … but if it would be there, everything else would work out of the box i think.

giovanni.piemontese

Yes, understood... But u have to think/analyse it in according other aspects...
For example we have ca. 50K customers and ca. 3M Products...
In the case i choice for ootb solution with price dimension i think i have in a couple of months sure some problems... .. And.. don't forget also that in most cases the customer price can change every day (as in our case)... it means that the problem is not only bound to p+s for publish, but also for unpublish case..

UPWG9AYH2

sounds reasonable in your case … but if i think of the code that must be changed to work spryker as expected … for example what about adding this product with this price to the cart? the complete cart calculation stack must be deactivated and post/pre plugins must set the correct price for the item?

giovanni.piemontese

and in worst case i have in db saved 50 000 * 3 000 000 = 150Miliard Kombination... Not possible with some frameworks and db engine..

No. the price calculation stack works further... we have in db saved only for all 3M products the Brutto price and in post save stack run own plugins that update the item prices according sap price data.. and then spryker (standard) after post save plugin run again the calculation... we have always right calculated cart....
-> The only place to get central in your code the price applying in cart is in postSavePlugins Stack in CartDependencyProvider..

p.s.: of course in the case u have some additional cost from shipment or payment etc... it will be a little bit complicated...

UPWG9AYH2

and in worst case i have in db saved 50 000 * 3 000 000 = 150Miliard Kombination... Not possible with some frameworks and db engine..

Would be the case if every combination is requested in the end. In our case, a customer just get the specific price if he enter the pdp … but yes, in long term, doesn’t seem feasible

No. the price calculation stack works further... we have in db saved only for all 3M products the Brutto price and in post save stack run own plugins that update the item prices according sap price data..

So you use the merchant related prices feature on top?

spryker (standard) after post save plugin run again the calculation... we have always right calculated cart....

That means, there is also calculation logic on the spryker side? In our case the prices that get requested are calculated by the erp and dont need to pass calculation stack … but only if these are requested in the end. if not, the standard prices should be calculated (with calculation stack eventually)

giovanni.piemontese

1. Ok
2. We don't use merchant related prices. We have just the DEFAULT Price imported to avoid to change some business logic. But thanks the post save plugin the price are always right and customer specific.
3. In my case i need to pass the calculation stack (standard spryker do it after post save plugins) because as said the prices from erp (sap) can have an impact on shipment surcharge, payment surcharge and some other related prices... In the case, for some reason, sap middleware just not reply, the system will calculate everything with default prices saved in db (in our case brutto price).
   p.s.: we are full in B2B concept and closed shop.

giovanni.piemontese

i.e, addToCart Operation (the same is for removeFromCart and for ReloadCart etc...)

UPWG9AYH2

Okay, i hope i got it now:
1. load the prices into a custom redis k:v structure when making an ERP request (like on pdp page)
2. on cart actions like addtocart, tell a postsaveplugin to fetch these prices from the custom redis structure instead of loading it from the default spryker db (?)

UPWG9AYH2

Can you detect if the prices in the cart are now sap or spryker default prices?

UPWG9AYH2

I could also think of a unfavorable situation when there are mixed prices (default for some items, sap related for some other)

giovanni.piemontese

1. yes
2. make a erp call... (without to know if there is a cache or not). Business logic there has a complex cache logic.. in the case the request is full cached then return just the data from redis, otherwise make the erp call and save the cache from response..
   p.s..: we have bulk request but we split the cache on single product

ps.2: yes i have a flag...

UPWG9AYH2

okay, sounds reasonable to me and since the cart just calculates only when there is a action on it, the effort seems to be moderate at all

giovanni.piemontese

yes.. 🙂 if u want we can speak about it in details in private.. i can show u how is implemented.. maybe u can save a lot of concept etc...

Andriy Netseplyayev

very interesting thread, indeed 🙂
saving ERP prices into redis - how fast did you come to this approach? Sounds like an optimisation, some shops try to keep it “on the fly” if response times are good though! In that case, do you specify/manage the ttl of the stored key in redis?

That could also be interesting in a multi-store environment

giovanni.piemontese

Hi Andriy,
yes, we have a SapMiddlewareApi Bundle that in our System is the heart bundle... Then we have another bundle called SapMiddleareApiCache...
In the Request Class we have of course some stack plugins registered..
Every Request is just configurable to enable the cache and give a TTL for Redis..
For example for price we have 1day, for some other requests we have i.e. just 2 hours or similar..

Of course we extended the Redis Adapter to allow TTL in setex and combination of setMulti + expire and to properly separate the configuration of other redis (KV, Storage) from this redis sap.

It is very a huge of code, very difficult to explain just in few words...

About Mult-Store Env I think that is just the cache key generation to take care on Store (or maybe just configure a relation 1-1 store-db_redis

Stanislav Matveyev

Have you considered other options like using DB spy_price*_storage calls directly by PriceProductStorageClient from Postgres/MariaDB?

giovanni.piemontese

Hi Stanislav,

we have ca, 3Mio Products and ca. 50K Customers.. in worst case I have ca. 150Miliarde Prices... Und die Preise ändern sich fast jeden Tag von SAP...

it means that I see a potential bottleneck using the storage table... and (important): the writing in this redis db (sap cache) must be sync otherwise (in case of worker problems or to much events) i have a lot of request to SAP for prices already requested, but not yes synced in redis.

giovanni.piemontese

don't forget (VERY IMPORTANT) -> SAP is transactional system and slow for real-time request.. it means that to much request to sap can be a bottleneck problem...

Andriy Netseplyayev

Hey @UL65CH0MC @UPWG9AYH2 I hope your SAP integrations work well ☀ ! I just wanted to check with you - which PHP libraries are you using in your integrations, maybe you could recommend any or the other way round - to stay away from? Thanks in advance!

giovanni.piemontese

Hi Andriy,
it is very complicated about library etc..
(i will create a Docker Image on Docker Hub for all, this is my goal).

But of course we can have a call and i can explain u what we have done.. and what libraries we use in our middleware project..

UPWG9AYH2

Hi @UKJSE6T47, we are currently on other things and had no time working on this topic in the meantime. So, no useful input from me yet. But I’ll come back later to this topic again 🙂