# Alarm

EMQX Broker has built-in monitoring and alarm functionality. Currently it supports monitoring of CPU occupancy, system and process memory occupancy, number of processes, rule engine resource status, cluster partition and healing, and it can alarm on these metrics. Both activation and deactivation of alarms will generate an alarm log and the Broker will publish an MQTT message with the topic of $SYS/brokers/<Node>/alarms/activate or $SYS/brokers/<Node>/alarms/deactivate. Users can subscribe to the topics of $SYS/brokers/+/alarms/avtivate and $SYS/brokers/+/alarms/deactivate to get alarm notifications.

The Payload of an alarm notification message is in Json format and contains the following fields:

FieldTypeDescription
namestringAlarm name
detailsobjectAlarm details
messagestringHuman-readable alarm instructions
activate_atintegerA UNIX timestamp in microseconds representing when the alarm was activated
deactivate_atinteger / stringA UNIX timestamp in microseconds representing when the alarm was deactivated. The value of this field for the activated alarm is infinity.
activatedbooleanWhether the alarm is activated

Taking the alarm of high system memory usage as an example, you will receive a message in the following format:

An alarm will not be generated repeatedly. That means if the high CPU usage alarm has been activated the same alarm will not appear again while high CPU is maintained. The alarm will be automatically deactivated when the monitored metric returns to normal. However, it also supports manual deactivation by the user (if the user clearly does not care about the alarm). Users can view current alarms (activated alarms) and historical alarms (deactivated alarms) on the Dashboard, and they can also use the HTTP API provided by EMQX to Query and manage alarms.

EMQX Broker allows users to adjust the alarm function to a certain extent to meet actual needs. The following configuration items are currently available:

Configuration itemTypeDefault ValueDescription
sysmon.os.cpu_check_intervalduration60sCheck interval for CPU usage
sysmon.os.cpu_high_watermarkpercent80%The high watermark of the CPU usage, which is the threshold to activate the alarm
sysmon.os.cpu_low_watermarkpercent60%The low watermark of the CPU usage, which is the threshold to deactivate the alarm
sysmon.os.mem_check_intervalduration60sCheck interval for memory usage
sysmon.os.sysmem_high_watermarkpercent70%The high water mark of the system memory usage. The alarm is activated when the total memory occupied by the application reaches this value
sysmon.os.procmem_high_watermarkpercent5%The high water mark of the process memory usage. The alarm will be activated when the memory occupied by a single process reaches this value
sysmonn.vm.process_check_intervalduration30sCheck interval for the number of processes
sysmon.vm.process_high_watermarkpercent80%The high watermark of the process occupancy rate, that is, the alarm is activated when the ratio of the number of created processes to the maximum number limit reaches this value
sysmon.vm.process_low_watermarkpercent60%The low water mark of the process occupancy rate, that is, the alarm is deactivated when the ratio of the number of created processes to the maximum number limit drops to this value
sysmonn.vm.long_gcdisabled/durationdisabledEnable Long GC monitoring
sysmon.vm.long_scheduledisabled/durationdisabledEnable Long Schedule monitoring
sysmon.vm.large_heapdisabled/bytesizedisabledEnable Large Heap monitoring
sysmon.vm.busy_portbooleantrueEnable Busy Distribution Port monitoring
sysmonn.top.num_itemsinteger10The number of top processes per monitoring group
sysmon.top.sample_interlvalduration2sSpecifies how often process top should be collected
sysmon.top.max_procsinteger1000000Stop collecting data when the number of processes in the VM exceeds this value
sysmonn.top.db_hostnamestring""Hostname of the PostgreSQL database that collects the data points
sysmonn.top.db_portinteger5432Port of the PostgreSQL database that collects the data points
sysmon.top.db_usernamestring"system_monitor"Username of the PostgreSQL database
sysmon.top.db_passwordstring"system_monitor_password"EMQX user password in the PostgreSQL database
sysmon.top.db_namestring"postgres"PostgreSQL database name