How can we approach debugging ``` Fatal error: Uncaught PhpAmqpLib\Exception\AMQPConnectionClosedExc

UK5DS29L2

How can we approach debugging

UKGT7RC7P

maybe check the rabbit mq logs?
when does it happen?

UK5DS29L2

that happens when running
vendor/bin/console -vvv queue:task:start <rabbitqueuename>

UK5DS29L2

but no issues seen in rabbit logs

UKGT7RC7P

and the container/host running the command can see rabbit otherwise?

UKGT7RC7P

could it be a configuration issue?

UK5DS29L2

yes, most of the queues run fine

UK5DS29L2

eventually they hit a queue item that causes this error, and then the queue cannot progress

UK5DS29L2

could it be a configuration issue?

well that's one of the guesses, but I have no idea what to look for

UKGT7RC7P

if the other queues run fine then I guess not

UKGT7RC7P

sounds like Rabbit is dying though?

UKGT7RC7P

you could maybe enable event logging and see which event is causing the issue?

UKGT7RC7P

UK5DS29L2

I tried to change heartbeat to 0,
\Spryker\Client\RabbitMq\RabbitMqConfig::AMQP_STREAM_CONNECTION_READ_WRITE_TIMEOUT to 3600
changed event bulk size from 500 to 1
but still the problem can be reproduced

UK5DS29L2

and dies way earlier than 1 hour

UKGT7RC7P

we had issues with rabbit running out of memory and dying (too many events)

UKGT7RC7P

but if you can isolate a queue item that reliably causes the error that sounds look a good first step I guess, if not the problem is elsewhere

UK5DS29L2

ah, good idea, let me check the memory usage (although I think I would see this in logs)

UKHQ50XFX

RabbitMQ might block requests, e.g. for memory reasons. then it blocks additional requests (even consuming messages) for a while until cause for blocking is gone.
about memory: you should set the memory threshold (set_vm_memory_high_watermark) to 80% of the available memory of the RabbitMQ instance.
when using RabbitMQ Docker image, there is the env var RABBITMQ_VM_MEMORY_HIGH_WATERMARK for this purpose.
see also https://www.rabbitmq.com/memory.html

UK5DS29L2

going to check another thing (reconnect instead of throwing the expection) and see how that goes. if that fails I will try to scale the server up and see how that goes

UK5DS29L2

so reconnection didn't work which proven what @UKGT7RC7P suggested (memory issues) and i rescaled rabbit host to more ram, which allowed the queue to run successfully

UK5DS29L2

which is really really bad, as there was no meaningful errors anywhere to prove it (e.g. what ES does when it's out of RAM is to shout about it in logs)

UKKEX8U6L

Is it locally? We saw this problem on AWS because connection on load balancer get closed if the open connection idles for a too long time and doesn't send any data

UK5DS29L2

no, this is on live DO server, and it wasn't because of the long requests, as I was able to keep the connection alive for longer time

UK5DS29L2

well, WAS. we already scaled up which helped