Hello everyone, because we have so much data that would need to be pushed to the Storage-Redis, we

UQ958DA4U

Hello everyone,

because we have so much data that would need to be pushed to the Storage-Redis, we opted to switch it out for a direct database connection as outlined here: https://documentation.spryker.com/docs/howto-replace-key-value-storage-with-database
Now we've got the problem that we need a reliable way to set this up with docker/sdk. Is there any way to pass new environment variables to the Yves container without manually patching the generator scripts?

Thanks in advance.

giovanni.piemontese

how much data do u have?

UQ958DA4U

Enough that Redis is not a viable solution any longer. We are talking milions of products.

giovanni.piemontese

Ok. we have 26GB of data in redis and it works well..

We have ca. 3Mio Product Abstract, 3Mio Product Concrete, 5Mio Product Images, 3Mio Price, 3Mio Product Sales Unit, 3Mio Product Quantity, a lot of Mio Product Stocks + Availability, 1000 Categories (only 1 Tree) etc etc etc.. etc...... and without Redis the System will be very slow...

giovanni.piemontese

if u have 100Mio products, maybe can (a little bit) understand your question.. even if u can scale up with Storage CTE and more worker parallel the P+S ...

UQ958DA4U

We are simply at a point where Redis cannibalizes all our cluster resources for itself due to data duplication. 30GB would not be enough and we are growing.

UQ958DA4U

Also, I don't see how a SELECT on an indexed table would be much slower than a KV access with a properly indexed and sized database. The idea of decoupling by default is nice, but we've clearly found a limit here. We were also advised by Spryker to consider this approach if Redis starts exploding.

giovanni.piemontese

ok, of course i don't know your data size and other aspects... I think that request to redis are faster than request to zed and forward to db

Imagine u have Yves Server/Instance that for every glossary or storage request have to call the zed (because only zed can communicate with db) and then zed server/Instance have to make another request (in best case) to another server where db is running... i think that the result can be that the system is a little bit slower (maybe)...

Other side u save up a lot of rmq messages, php process for syncing etc.. and your data are sure synced always...

UQ958DA4U

The way the tutorial sets it up, the Storage driver is exchanged, so no Zed calls are made. The Yves frontend connects directly to the database. I agree that constant REST requests would be an overwhelming overhead.

UQ958DA4U

also, we're still using the publish process but just not the sync process that pushes the data to Redis. So we're still working with denormalized data, it's just not read from Redis but from the _storage tables in the DB instead.

giovanni.piemontese

yes i see...
i did not know that

StorageDatabase

have a self connectionProvider to communicate via client layer with db...

giovanni.piemontese

then would be an acceptable solution...

UQ958DA4U

that's really all besides my original question unfortunately 😉
if we only did SQL queries through Zed, we wouldn't have the issue mentioned in the post as that of course already gets the credentials passed in via environment by docker/sdk.

giovanni.piemontese

I think u have include the database env located here docker/generator/src/templates/env/database (i.e. for zed here -> docker/generator/src/templates/env/application/zed.env.twig:25 ) in docker/generator/src/templates/env/application/yves.env.twig too... and then in docker/generator/src/templates/application/yves.yml.twig u have to extends the depends_on with database too..

sprymiker

Locally it can just use the same database credentials from primary database.
You can still separate dev and cloud config on PHP side to take different environment variables.

sprymiker

Aha… So you problem is that Yves/Glue does not have database env variables at all.

In this case I would suggest to use this functionality:
https://github.com/spryker/docker-sdk/blob/master/docs/99-deploy.file.reference.v1.md#image-environment

image:
  ...
  environment:
        STORAGE_DATABASE_HOST: database
        STORAGE_DATABASE_USER: spryker
        STORAGE_DATABASE_PASSWORD: secret

But only for dev deploy.yml

UQ958DA4U

thanks