Hello team,
We have deployed the EMQ X broker in a Kubernetes cluster and are publishing messages with the retain flag set to true. However, when our application subscribes to topics with retained messages, the client is getting disconnected, and the EMQ X broker logs are indicating timeouts.
Here’s the Broker logs
2023-02-15T12:59:57.458837+00:00 [warning] line: 417, mfa: emqx_alarm:do_actions/3, msg: alarm_is_deactivated, name: <<"conn_congestion/test_subscriber_client/debug1">>
2023-02-15T13:05:30.853409+00:00 [warning] line: 411, message: <<"connection congested: #{buffer => 4096,clientid => <<\"test_subscriber_client\">>,conn_state => connected,connected_at => 1676466330601,high_msgq_watermark => 8192,high_watermark => 1048576,memory => 1107856,message_queue_len => 625,peername => <<\"10.240.2.120:46418\">>,pid => <<\"<0.14814.12>\">>,proto_name => <<\"MQTT\">>,proto_ver => 5,recbuf => 131072,recv_cnt => 17,recv_oct => 2094,reductions => "...>>, mfa: emqx_alarm:do_actions/3, msg: alarm_is_activated, name: <<"conn_congestion/test_subscriber_client/debug1">>
2023-02-15T13:05:45.861316+00:00 [warning] clientid: test_subscriber_client, line: 881, mfa: emqx_connection:handle_info/2, msg: socket_error, peername: 10.240.2.120:46418, reason: timeout
2023-02-15T13:05:45.861834+00:00 [warning] line: 417, mfa: emqx_alarm:do_actions/3, msg: alarm_is_deactivated, name: <<"conn_congestion/test_subscriber_client/debug1">>
2023-02-15T13:06:23.396232+00:00 [warning] line: 411, message: <<"connection congested: #{buffer => 4096,clientid => <<\"test_subscriber_client\">>,conn_state => connected,connected_at => 1676466383149,high_msgq_watermark => 8192,high_watermark => 1048576,memory => 1034368,message_queue_len => 686,peername => <<\"10.240.2.120:37348\">>,pid => <<\"<0.15511.12>\">>,proto_name => <<\"MQTT\">>,proto_ver => 5,recbuf => 131072,recv_cnt => 11,recv_oct => 2094,reductions => "...>>, mfa: emqx_alarm:do_actions/3, msg: alarm_is_activated, name: <<"conn_congestion/test_subscriber_client/debug1">>
2023-02-15T13:06:38.406390+00:00 [warning] clientid: test_subscriber_client, line: 881, mfa: emqx_connection:handle_info/2, msg: socket_error, peername: 10.240.2.120:37348, reason: timeout
2023-02-15T13:06:38.407009+00:00 [warning] line: 417, mfa: emqx_alarm:do_actions/3, msg: alarm_is_deactivated, name: <<"conn_congestion/test_subscriber_client/debug1">>
2023-02-15T13:07:31.725278+00:00 [warning] line: 411, message: <<"connection congested: #{buffer => 4096,clientid => <<\"test_subscriber_client\">>,conn_state => connected,connected_at => 1676466436725,high_msgq_watermark => 8192,high_watermark => 1048576,memory => 1574912,message_queue_len => 0,peername => <<\"10.240.2.120:38596\">>,pid => <<\"<0.16218.12>\">>,proto_name => <<\"MQTT\">>,proto_ver => 5,recbuf => 131072,recv_cnt => 38,recv_oct => 2969,reductions => 15"...>>, mfa: emqx_alarm:do_actions/3, msg: alarm_is_activated, name: <<"conn_congestion/test_subscriber_client/debug1">>
2023-02-15T13:07:48.240397+00:00 [warning] clientid: test_subscriber_client, line: 881, mfa: emqx_connection:handle_info/2, msg: socket_error, peername: 10.240.2.120:38596, reason: timeout
2023-02-15T13:07:48.240889+00:00 [warning] line: 417, mfa: emqx_alarm:do_actions/3, msg: alarm_is_deactivated, name: <<"conn_congestion/test_subscriber_client/debug1">>
Here’s the client side logs
2023-02-15 18:37:53.231: Info: MQTT - Authentication enabled
2023-02-15 18:37:54.028: Debug: test_subscriber_client Connected
2023-02-15 18:38:32.132: Debug: Disconnected Client Reason : NormalDisconnection, Exception: , Exception: MQTTnet.Exceptions.MqttCommunicationException: An existing connection was forcibly closed by the remote host.
---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
at MQTTnet.Implementations.MqttTcpChannel.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at MQTTnet.Adapter.MqttChannelAdapter.ReceiveAsync(CancellationToken cancellationToken)
at MQTTnet.Adapter.MqttChannelAdapter.ReceivePacketAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at MQTTnet.Adapter.MqttChannelAdapter.WrapAndThrowException(Exception exception)
at MQTTnet.Adapter.MqttChannelAdapter.ReceivePacketAsync(CancellationToken cancellationToken)
at MQTTnet.Client.MqttClient.TryReceivePacketsAsync(CancellationToken cancellationToken)
I would appreciate any support in troubleshooting and resolving this issue. If needed, I can provide additional details to assist.
Thank you.