TIBCO Mashery® Local Monitoring Guide

(1)

Monitoring Guide

Version 4.x

November 2020

(2)

TIBCO Documentation and Support Services . . . . 3

Monitoring Mashery Local 4.x . . . .4

Node Metrics . . . . 5

Component Monitoring . . . . 6

Traffic Manager

. . . .

⁶

MySQL

. . . .

¹¹

Memcache

. . . .

¹⁵

Memcache Loader

. . . .

²⁸

Status of Cloud Sync Services, Developer, Tokens and Logs

. . . .

²⁸

Legal and Third-Party Notices . . . . 31

(3)

TIBCO Documentation and Support Services

How to Access TIBCO Documentation

Documentation for TIBCO products is available on the TIBCO Product Documentation website, mainly in HTML and PDF formats.

The TIBCO Product Documentation website is updated frequently and is more current than any other documentation included with the product. To access the latest documentation, visit https://docs.tibco.com.

Product-Specific Documentation

Documentation for TIBCO products is not bundled with the software. Instead, it is available on the TIBCO Documentation site at TIBCO Mashery

^®

Local documentation page.

The following documents for this product can be found on the TIBCO Documentation website:

●

TIBCO Mashery

^®

Local Release Notes

●

TIBCO Mashery

^®

Local Installation and Configuration Guide

●

TIBCO Mashery

^®

Local Migration and Upgrade Guide

●

TIBCO Mashery

^®

Local Reporting Services

For information on TIBCO Cloud Integration with Mashery, refer to Integrating with TIBCO Mashery.

How to Contact TIBCO Support

You can contact TIBCO Support in the following ways:

●

For an overview of TIBCO Support, visit http://www.tibco.com/services/support.

●

For accessing the Support Knowledge Base and getting personalized content about products you are interested in, visit the TIBCO Support portal at https://support.tibco.com.

●

For creating a Support case, you must have a valid maintenance or support contract with TIBCO. You also need a user name and password to log in to https://support.tibco.com. If you do not have a user name, you can request one by clicking Register on the website.

How to Join TIBCO Community

TIBCO Community is the official channel for TIBCO customers, partners, and employee subject matter

experts to share and access their collective experience. TIBCO Community offers access to Q&A forums,

product wikis, and best practices. It also offers access to extensions, adapters, solution accelerators, and

tools that extend and enable customers to gain full value from TIBCO products. In addition, users can

submit and vote on feature requests from within the TIBCO Ideas Portal. For a free registration, go to

https://community.tibco.com.

(4)

Monitoring Mashery Local 4.x

Each node of a Mashery Local cluster has the following main components - Traffic Manager, MySQL and MemCache.

To ensure reliability of the Mashery Local cluster, it is important to monitor these components and set up alerts when resources go low either at the node level or at the process level, for each application. At the node level, resources such as CPU, Memory, Disk and Network I/O can be monitored. At the process level, each process-specific resource can be monitored. For example, monitoring the percentage of available memory in MemCache is important to ensure reliability and optimal performance of MemCache.

Data Collection

Statistics are collected in the following ways:

●

by making HTTP calls from your monitoring system.

●

by forwarding log files to your monitoring system.

Both ways are needed as some statistics can be collected only by making HTTP calls and other statistics by forwarding logs from the host machine where the logs are generated.

Monitoring System

A monitoring system is a log processing software that may provide log collection agents like Splunk

forwarder for Splunk or FileBeat for elastic search. Please consult your devops team on which monitoring

solution to use.

(5)

Node Metrics

The health of the node of a Mashery Local cluster can be monitored by tracking the use of resources like CPU, Memory, Disk and Network I/O. Alerts can be set if the resources are going low below certain levels.

For example, alerts can be set if:

●

Available CPU resources are below 50%,

●

Swap file is being used which means that RAM is fully utilized, or

●

Available disk is space is below 20%

The above numbers are just examples. Contact your Mashery Customer Success team for your specific threshold values.There are many third-party tools available to collect these metrics, for example, ELK users can use MetricBeat.

Diagnostic Recipe / Alerts

Metrics Notes

CPU resources are low Consider adding more nodes to distribute traffic Swap file usage has increased and available

memory is low Consider increasing the RAM size for the node

Disk space running low Check log file sizes and consider purging logs.

Check expired token count in MySQL and

consider deleting more aggressively if the count is high.

Consider increasing the disk space.

(6)

Component Monitoring

●

Traffic Manager

●

MySQL

●

Memcache

●

Memcache Loader

●

Status of Cloud Sync - Services, Developer, Tokens and Logs

Traffic Manager

Jetty Pool Connection Statistics

Traffic Manager maintains a thread pool to process incoming requests. Monitoring the statistics of this pool can help in determining the health of the traffic manager.

The stats can be collected by adding a cron job to every node of Mashery Local cluster. The cron job makes a cURL call and appends the results in a log file which can be forwarded to the log processing system. This is available for Mashery Local 4.4 and above for both Docker and OVA versions. If using the Docker version, log into the container

^ml-tm

before running the following command.

curl -s --compressed -H "Accept: application/json" -X GET http://127.0.0.1:8082/jmx/

com.mashery.proxy.management:name=JettyConnectionPool,type=ConectionPool

Connection Pool Statistics

{

"results": [{

"connectionData": [{

"Connections": 0,

"ConnectorUptimeSec": 2195, "MaxConnections": 1,

"MaxConnectionDuration": 0.056,

"ConnectorUptime": " 0 hours 36 minutes 35 seconds", "Port": 80,

"MaxRequestsPerConnection": 1, "TotalRequests": 36,

"ConnectionDurationStdDev": 0.0075163440961300175, "RequestsPerConnectionStdDev": 0.0,

"Keystore": "Non SSL",

"TotalConnectionDuration": " 0 hours 0 minutes 0 seconds", "AvgConnectionDuration": 0.008638888888888889,

"TotalConnections": 36,

"AvgRequestsPerConnection": 1.0, "TotalConnectionDurationSec": 0 }],

"threadPoolIdleThreads": 48, "threadPoolMaxThreads": 500, "threadPoolMinThreads": 50, "threadPoolThreads": 50 }]

}

(7)

Parameters

Name Field Notes Details

Max requests threads

threadPoolMaxThreads

Denotes the maximum

concurrent requests possible on this Traffic Manager instance

Inital number of threads

threadPoolMinThreads

Threads to be created on startup Current created threads

threadPoolThreads

Number of threads

created in the pool Idle Threads

threadPoolIdleThreads

Number of threads

created but are idle (at the time of taking the snapshot)

A high value of idle threads denotes low utilization of this instance. Check if other Traffic Manager instances are over-loaded.

Total connections since

start

Connections

Number of

connection requests to this instance Avg time for requests

AvgConnectionDuration

Average time taken

by the this instance to server traffic

A low value denotes high throughput

Diagnostic Recipe / Alerts

Metric Field Name / Computation Notes Efficiency

threadPoolThreads/

This ratio provides the utilization of a Traffic Manager. Though it may depend upon traffic patterns, a desired range is 75%-80%.

Low water mark - 80%

High water mark - 90%

A continuous breach of high water mark

indicates that this Traffic Manager instance

is over loaded.

(8)

Metric Field Name / Computation Notes Under-

utilization

threadPoolIdleThreads/

A very high ratio denotes that the Traffic Manager is handling fewer calls and might indicate deficiency in load balancing rules.

However, planning for peak usage is very important and during other times, it is possible that nodes show under-utlization.

Max connections

connectionData.MaxConnections

This is the maximum connections open at the same time, thereby giving the peak usage value. An alert can notify if this value is within 80%. Upon receiving this alert, consider increasing the number of nodes.

Increasing the threadPoolMaxThreads is an option but it depends if there are enough CPU and Memory resources to handle a higher number of concurrent requests in the same node. A higher number of concurrent requests will require more resources by Garbage Collection Processes.

Old Garbage Collection Statistics

The stats can be collected by adding a cron job to every node of the Mashery Local cluster. The cron job calls a URL and appends the results in a log file that can be forwarded for further processing. This is available in Mashery Local 4.3 and above for the OVA version only.

For example:

curl -s --compressed -H "Accept:

application/json" -X GET

http://127.0.0.1:8082/jmx/java.lang:name=PS%20MarkSweep,type=GarbageCollector

The default configuration to run the Traffic Manager does not specify any specific Garbage Collector and JVM chooses "PS MarkSweep" as the old generation garbage collector. For more information about Garbage Collection Tuning, see the HotSpot Virtual Machine Tuning Guide for your version of Java Standard Edition in Oracles's online documentation.

{

"results": [{

"LastGcInfo": { "gcInfo": {

"compositeType": {

"description": "CompositeType for GC info for PS MarkSweep", "array": false,

"typeName": "sun.management.PS MarkSweep.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData"

},

"startTime": 544366, "endTime": 544426, "duration": 60,

"memoryUsageBeforeGc": { "PS Old Gen": { "max": 716177408, "committed": 44040192, "used": 21829792, "init": 41943040 }

},

"memoryUsageAfterGc": { "PS Old Gen": { "max": 716177408, "committed": 61865984,

(9)

"used": 11530416, "init": 41943040 }

}, "id": 2 },

"compositeType": {

"description": "CompositeType for GC info for PS MarkSweep", "array": false,

"typeName": "sun.management.PS MarkSweep.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData"

} },

"CollectionCount": 2, "CollectionTime": 100, "Valid": true,

"MemoryPoolNames": ["PS Eden Space", "PS Survivor Space", "PS Old Gen"], "Name": "PS MarkSweep",

"ObjectName": {

"domain": "java.lang", "pattern": false, "domainPattern": false, "propertyPattern": false, "propertyListPattern": false, "propertyValuePattern": false, "keyPropertyList": {

"name": "PS MarkSweep", "type": "GarbageCollector"

},

"keyPropertyListString": "type=GarbageCollector,name=PS MarkSweep", "canonicalKeyPropertyListString": "name=PS

MarkSweep,type=GarbageCollector",

"canonicalName": "java.lang:name=PS MarkSweep,type=GarbageCollector"

} }]

}

Diagnostic Recipe / Alerts

See the Oracle Garbage Collection Tuning guide for more information.

New Garbage Collection Statistics

The stats can be collected by adding a cron job to every node of Mashery Local cluster. The cron job calls a URL and appends the results in a log file that can be forwarded for further processing. This is available for Mashery Local 4.3 and above for the OVA versions only.

curl -s

--compressed -H "Accept: application/json" -X GET

http://127.0.0.1:8082/jmx/java.lang:type=GarbageCollector,name=PS%20Scavenge

The default configuration to run the Traffic Manager does not specify any specific Garbage Collector and JVM chooses "PS Scavenge" as the default New generation garbage collector. For more information about Garbage Collection Tuning, see the HotSpot Virtual Machine Tuning Guide for your version of Java Standard Edition in Oracles's online documentation.

{

"results": [{

"LastGcInfo": { "gcInfo": {

"compositeType": {

"description": "CompositeType for GC info for PS Scavenge", "array": false,

"typeName": "sun.management.PS Scavenge.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData"

},

"startTime": 2045258, "endTime": 2045263, "duration": 5,

"memoryUsageBeforeGc": { "PS Survivor Space": {

(10)

"max": 3670016, "committed": 3670016, "used": 1148864, "init": 2621440 },

"Metaspace": { "max": -1,

"committed": 40239104, "used": 36625520, "init": 0

},

"PS Eden Space": { "max": 350224384, "committed": 155713536, "used": 155713536, "init": 15728640 }

},

"memoryUsageAfterGc": { "PS Survivor Space": { "max": 3670016, "committed": 3670016, "used": 2022880, "init": 2621440 },

"Metaspace": { "max": -1,

"committed": 40239104, "used": 36625520, "init": 0

},

"PS Eden Space": { "max": 350224384, "committed": 157286400, "used": 0,

"init": 15728640 }

}, "id": 62 },

"compositeType": {

"description": "CompositeType for GC info for PS Scavenge", "array": false,

"typeName": "sun.management.PS Scavenge.GcInfoCompositeType", "className": "javax.management.openmbean.CompositeData"

} },

"CollectionCount": 62, "CollectionTime": 372, "Valid": true,

"MemoryPoolNames": ["PS Eden Space", "PS Survivor Space"], "Name": "PS Scavenge",

"ObjectName": {

"domain": "java.lang", "pattern": false, "domainPattern": false, "propertyPattern": false, "propertyListPattern": false, "propertyValuePattern": false, "keyPropertyList": {

"name": "PS Scavenge", "type": "GarbageCollector"

},

"keyPropertyListString": "type=GarbageCollector,name=PS Scavenge",

"canonicalKeyPropertyListString": "name=PS Scavenge,type=GarbageCollector", "canonicalName": "java.lang:name=PS Scavenge,type=GarbageCollector"

} }]

}

Diagnostic Recipe / Alerts

(11)

For more information about Garbage Collection Tuning, see Oracle's Garbage Collection Tuning guide.

MySQL

Get Status

The status can be obtained periodically by calling the URL:

https://<IP address of the node>:5480/service/mashery/cgi/mysql_status_cgi.py

The response is a JSON that has the status of the mysql server.

MySQL Statistics

{

"Connection_errors_accept": 0, "Connection_errors_internal": 0,

"Connection_errors_max_connections": 0, "Connection_errors_peer_address": 0, "Connection_errors_select": 0, "Connection_errors_tcpwrap": 0, "Connections": 234,

"Delayed_insert_threads": 0,

"Innodb_buffer_pool_dump_status": "Dumping", "Innodb_buffer_pool_load_status": "Buffer", "Innodb_buffer_pool_resize_status": "", "Innodb_buffer_pool_pages_data": 1802, "Innodb_buffer_pool_bytes_data": 29523968, "Innodb_buffer_pool_pages_dirty": 0, "Innodb_buffer_pool_bytes_dirty": 0, "Innodb_buffer_pool_pages_flushed": 2834, "Innodb_buffer_pool_pages_free": 6302, "Innodb_buffer_pool_pages_misc": 87, "Innodb_buffer_pool_pages_total": 8191, "Innodb_buffer_pool_read_ahead_rnd": 0, "Innodb_buffer_pool_read_ahead": 0,

"Innodb_buffer_pool_read_ahead_evicted": 0, "Innodb_buffer_pool_read_requests": 3247529, "Innodb_buffer_pool_reads": 1603,

"Innodb_buffer_pool_wait_free": 0,

"Innodb_buffer_pool_write_requests": 279100, "Innodb_data_fsyncs": 11193,

"Innodb_data_pending_fsyncs": 0, "Innodb_data_pending_reads": 0, "Innodb_data_pending_writes": 0, "Innodb_data_read": 29266432, "Innodb_data_reads": 2006, "Innodb_data_writes": 13727, "Innodb_data_written": 126338560, "Innodb_dblwr_pages_written": 2799, "Innodb_dblwr_writes": 148,

"Innodb_log_waits": 0,

"Innodb_log_write_requests": 36527, "Innodb_log_writes": 10540,

"Innodb_os_log_fsyncs": 10722, "Innodb_os_log_pending_fsyncs": 0, "Innodb_os_log_pending_writes": 0, "Innodb_os_log_written": 33955840, "Innodb_page_size": 16384,

"Innodb_pages_created": 134, "Innodb_pages_read": 1668, "Innodb_pages_written": 2834, "Innodb_row_lock_current_waits": 0, "Innodb_row_lock_time": 0,

"Innodb_row_lock_time_avg": 0, "Innodb_row_lock_time_max": 0, "Innodb_row_lock_waits": 0, "Innodb_rows_deleted": 11712, "Innodb_rows_inserted": 11924, "Innodb_rows_read": 4709484, "Innodb_rows_updated": 90,

(12)

"Innodb_num_open_files": 76,

"Innodb_truncated_status_writes": 0, "Innodb_available_undo_logs": 128, "Max_used_connections": 6,

"Max_used_connections_time": "2019-03-25", "Performance_schema_accounts_lost": 0, "Performance_schema_cond_classes_lost": 0, "Performance_schema_cond_instances_lost": 0, "Performance_schema_digest_lost": 0,

"Performance_schema_file_classes_lost": 0, "Performance_schema_file_handles_lost": 0, "Performance_schema_file_instances_lost": 0, "Performance_schema_hosts_lost": 0,

"Performance_schema_index_stat_lost": 0, "Performance_schema_locker_lost": 0,

"Performance_schema_memory_classes_lost": 0, "Performance_schema_metadata_lock_lost": 0, "Performance_schema_mutex_classes_lost": 0, "Performance_schema_mutex_instances_lost": 0, "Performance_schema_nested_statement_lost": 0, "Performance_schema_prepared_statements_lost": 0, "Performance_schema_program_lost": 0,

"Performance_schema_rwlock_classes_lost": 0, "Performance_schema_rwlock_instances_lost": 0, "Performance_schema_session_connect_attrs_lost": 0, "Performance_schema_socket_classes_lost": 0,

"Performance_schema_socket_instances_lost": 0, "Performance_schema_stage_classes_lost": 0, "Performance_schema_statement_classes_lost": 0, "Performance_schema_table_handles_lost": 0, "Performance_schema_table_instances_lost": 0, "Performance_schema_table_lock_stat_lost": 0, "Performance_schema_thread_classes_lost": 0, "Performance_schema_thread_instances_lost": 0, "Performance_schema_users_lost": 0,

"Qcache_queries_in_cache": 0, "Queries": 415232,

"Slow_launch_threads": 0, "Slow_queries": 0,

"Threads_cached": 2, "Threads_connected": 4, "Threads_created": 6, "Threads_running": 1, "Uptime": 19807,

"Uptime_since_flush_status": 19807,

"ingestion_time": "2019-03-25T15:07:38+00:00"

}

Parameters

Name Data Field Unit

Data

Type Details Connection_errors_ac

cept

Connection_errors_accep

t

N/A Numbe

r The number of errors that occurred during calls to accept() on the listening port.

Connection_errors_m

ax_connections

Connection_errors_max_c

onnection

s N/A Numbe

r The number of connections refused because the server max_connections limit was reached.

Connection_errors_pe

er_address

Connection_errors_peer_

address

N/A Numbe

r The number of errors that occurred

while searching for connecting client

IP addresses.

(13)

Name Data Field Unit

Data

Type Details Innodb_pages_read

Innodb_pages_read

N/A Numbe

r The number of pages read from the InnoDB buffer pool by operations on InnoDB tables.

Innodb_rows_deleted

N/A Numbe

r The number of rows deleted from InnoDB tables.

Innodb_rows_inserted

N/A Numbe

r The number of rows inserted into InnoDB tables.

Innodb_rows_read

N/A Numbe

r The number of rows read from InnoDB tables.

Innodb_rows_update

d

Innodb_rows_updated

N/A Numbe

r The number of rows updated in InnoDB tables.

Max_used_connection

s

Max_used_connections

N/A Numbe

r The maximum number of connections that have been in use simultaneously since the server started.

Max_used_connection

s_time

Max_used_connections_ti

me

N/A Time The time at which

Max_used_connections reached its current value.

Queries

^Queries

N/A Numbe

r The number of statements executed by the server. This variable includes statements executed within stored programs, unlike the Questions variable. It does not count

COM_PING or COM_STATISTICS commands. The discussion at the beginning of this section indicates how to relate this statement-counting status variable to other such variables.

Slow_queries

N/A Numbe

r The number of queries that have taken more than long_query_time seconds. This counter increments regardless of whether the slow query log is enabled. For information about that log, see Section 5.4.5, "The Slow Query Log" .

Threads_cached

N/A Numbe

r The number of threads in the thread cache.

Threads_connected

N/A Numbe

r The number of currently open

connections.

(14)

Name Data Field Unit

Data

Type Details

Threads_created

N/A Numbe

r The number of threads created to handle connections. If

Threads_created is big, you may want to increase the

thread_cache_size value

. The cache miss rate can be calculated as Threads_created/

Connections.

Threads_running

N/A Numbe

r The number of threads that are not sleeping.

Uptime

^Uptime

Secon

ds Numbe

r The number of seconds that the server has been up.

Uptime_since_flush_s

tatus

Uptime_since_flush_stat

us

Secon

ds Numbe

r The number of seconds since the most recent FLUSH STATUS statement.

Runbooks

Metrics Computation Notes

Buffer

utilization Innodb_buffer_pool_byte s_data/

(Innodb_page_size*Innod b_buffer_pool_pages_tota l)

If buffer utilization is very high (for example, exceeds 98%) run the MySQLTuner script to determine the new value for

innodb_buffer_pool_size

and then increase the value of the parameter

in the /etc/my.cnf file.

Rejected

connections Max_used_connections /

Max_Connections Max Connections can be seen from /etc/my.cnf file. If the ratio is high (for example, above 90%), consider

increasing the number of max connections in the /etc/

my.cnf file. On the contrary, if the ratio is very low (for example, less than 10%), consider decreasing the number of max connections in the /etc/my.cnf file)

Available

Cache Qcache_free_blocks /

Qcache_total_blocks If the free space is less (for example, 20%), consider increasing the value of the parameter

query_cache_size

in /etc/my.cnf

Replication Status

Replication status of a "follower" node can be retrieved by periodically calling the URL:

https://<IP Address of the node>:5480/service/mashery/cgi/replication_status.py

The response is in JSON format.

{

"replication_status": { "restore": {

"error": false, "errors": "", "log": "",

"complete": false },

"replication": { "last_error": "",

(15)

"seconds_behind_master": "", "follower_io_running": "", "follower_sql_running": ""

} },

"error": null }

Diagnostic Recipe / Alerts

Metrics Computation Notes

seconds_behind

_master None An alert can be generated if the

value of seconds_behind_master

exceeds more than 120 (2 min). Seriously consider increasing the

. However, before increasing the value, make sure that there is enough RAM available. To calculate the used RAM, add the memory allocated to the traffic manager, memcache and the database.

To view the memory used by traffic manager, run the command

^{ps -ef |}

grep java

. Look at the parameter

^-xMx

.

To view the memory used by memcache, add the CACHESIZE of all the configuration files for various memcache pools. The names of the config files are available in the memcache section.

To view the memory used by MySQL, run the top command see the value of RES for the process mysql. Another way to compute memory

requirements by MySQL is to use MySQL Calculator.

Memcache

The memcache status can be periodically collected using the URL:

https://<IP address of the node>:5480/service/mashery/cgi/memcached_status.py

The memcahe stats consists of the following pools:

●

counter (master node only)

●

service

●

mapi

●

packager

●

content

●

httpsclientsecurity

The statistics for each pool are shown below.

Memcache Statistics

{

"counter": {

"lru_maintainer_juggles": 9441668.0, "lru_crawler_starts": 675240.0, "auth_cmds": 0.0,

"crawler_items_checked": 0.0, "reclaimed": 0.0,

"rejected_connections": 0.0, "get_expired": 0.0,

"curr_items": 0.0, "pid": 26638.0, "malloc_fails": 0.0,

"time_in_listen_disabled_us": 0.0, "slab_reassign_evictions_nomem": 0.0, "expired_unfetched": 0.0,

"moves_to_cold": 0.0,

(16)

"lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0, "port": 11212,

"lru_bumps_dropped": 0.0, "max_connections": 10240.0, "cas_hits": 0.0,

"uptime": 9442506.0, "moves_to_warm": 0.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0,

"decr_hits": 0.0, "direct_reclaims": 0.0, "cmd_touch": 0.0, "incr_hits": 0.0, "version": "1.5.4", "active_slabs": 0.0,

"limit_maxbytes": 268435456.0, "total_items": 0.0,

"bytes_written": 3773743.0, "incr_misses": 0.0,

"accepting_conns": 1.0,

"rusage_system": 55.626542999999998, "slab_reassign_running": 0.0,

"slabs": {

},

"log_watcher_sent": 0.0, "moves_within_lru": 0.0, "get_flushed": 0.0,

"slab_reassign_busy_items": 0.0, "slabs_moved": 0.0,

"cmd_get": 0.0,

"total_malloced": 0.0, "curr_connections": 10.0, "log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0,

"host": "localhost", "threads": 4.0, "get_hits": 0.0, "cmd_set": 0.0,

"libevent": "2.0.22-stable", "conn_yields": 0.0,

"get_misses": 0.0, "evicted_active": 0.0, "reserved_fds": 20.0, "bytes_read": 3762137.0, "hash_bytes": 524288.0,

"slab_reassign_chunk_rescues": 0.0, "evicted_unfetched": 0.0,

"cas_badval": 0.0, "cmd_flush": 0.0,

"lrutail_reflocked": 0.0, "evictions": 0.0,

"bytes": 0.0,

"slab_reassign_busy_deletes": 0.0, "crawler_reclaimed": 0.0,

"slab_reassign_inline_reclaim": 0.0, "connection_structures": 11.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0,

"rusage_user": 429.89164599999998, "time": 1559600959.0,

"delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0,

"total_connections": 156766.0

(17)

},

"service": {

"moves_to_cold": 1338990.0, "lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0, "port": 11214,

"limit_maxbytes": 134217728.0, "total_items": 1407989.0, "bytes_written": 38287775.0, "incr_misses": 0.0,

"slabs": { "1": {

"cas_hits": 0.0, "used_chunks": 1.0, "touch_hits": 0.0, "cmd_set": 31474.0, "cas_badval": 0.0, "free_chunks": 10921.0, "slab_no": "1",

"total_chunks": 10922.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 96.0,

"chunks_per_page": 10922.0, "delete_hits": 0.0,

"free_chunks_end": 0.0, "mem_requested": 88.0 },

"20": {

"cas_hits": 0.0, "used_chunks": 20.0, "touch_hits": 0.0, "cmd_set": 60358.0, "cas_badval": 0.0, "free_chunks": 127.0, "slab_no": "20", "total_chunks": 147.0,

(18)

"total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 7104.0, "chunks_per_page": 147.0, "delete_hits": 0.0, "free_chunks_end": 0.0, "mem_requested": 117174.0 }

},

"cmd_get": 1601164.0,

"bytes": 2368985.0,

"slab_reassign_inline_reclaim": 0.0, "connection_structures": 8.0,

"hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0,

"rusage_user": 971.62729000000002, "time": 1559600959.0,

"total_connections": 188302.0 },

"mapi": {

"moves_to_cold": 81041.0, "lru_crawler_running": 0.0, "slab_global_page_pool": 0.0, "hash_is_expanding": 0.0,

(19)

"port": 11211,

"uptime": 9442506.0, "moves_to_warm": 13088.0, "touch_hits": 0.0,

"delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0,

"slabs": { "1": {

"free_chunks_end": 0.0, "mem_requested": 271.0 },

"13": {

"cas_hits": 0.0, "used_chunks": 0.0, "touch_hits": 0.0, "cmd_set": 1.0, "cas_badval": 0.0, "free_chunks": 708.0, "slab_no": "13", "total_chunks": 708.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 1480.0, "chunks_per_page": 708.0, "delete_hits": 0.0, "free_chunks_end": 0.0, "mem_requested": 0.0 }

},

"cmd_get": 31500.0,

"total_malloced": 3145728.0, "curr_connections": 11.0,

(20)

"log_worker_written": 0.0, "log_watcher_skipped": 0.0, "touch_misses": 0.0,

"bytes": 39421.0,

"slab_reassign_inline_reclaim": 0.0, "connection_structures": 13.0, "hash_power_level": 16.0, "log_worker_dropped": 0.0, "auth_errors": 0.0,

"rusage_user": 607.44365400000004, "time": 1559600959.0,

"content": {

(21)

"slabs": {

},

"cmd_get": 0.0,

"bytes": 0.0,

"rusage_user": 433.868042, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0,

"httpsclientsecurity": {

"lru_bumps_dropped": 0.0, "max_connections": 10240.0,

(22)

"cas_hits": 0.0, "uptime": 9442506.0, "moves_to_warm": 0.0, "touch_hits": 0.0, "delete_misses": 0.0, "listen_disabled_num": 0.0, "slab_reassign_rescues": 0.0, "cas_misses": 0.0,

"slabs": { "2": {

"total_chunks": 8738.0, "total_pages": 1.0, "decr_hits": 0.0, "incr_hits": 0.0, "get_hits": 0.0, "chunk_size": 120.0, "chunks_per_page": 8738.0, "delete_hits": 0.0,

"free_chunks_end": 0.0, "mem_requested": 105.0 }

},

"cmd_get": 0.0,

"bytes": 105.0,

"slab_reassign_busy_deletes": 0.0,

(23)

"crawler_reclaimed": 0.0,

"rusage_user": 482.107708, "time": 1559600959.0, "delete_hits": 0.0, "pointer_size": 64.0, "decr_misses": 0.0,

"packager": {

"slabs": { "1": {

(24)

"free_chunks_end": 0.0, "mem_requested": 94.0 }

},

"cmd_get": 0.0,

"bytes": 94.0,

"rusage_user": 485.45619900000003, "time": 1559600959.0,

"total_connections": 31485.0 }

}

Parameters

Name Data Field Unit Data Type Notes Details

Pool Name

^pool_name

N/A String Possible values

counter, services, mapis, httpclientsecuri ty, packager, content

The pool name maps to

a Pool

(25)

Name Data Field Unit Data Type Notes Details

pool

^pool

N/A Number The port

number on which the pool is listening for Memcached function calls Possible values 11211, 11212, 11213, 11214, 11215,11216

11211 = mapis, 11212 = counter, 11213 = content, 11214 = services, 11215 = packages, 11216 = httpsclientsecurity

Particular slab

^slab_class

N/A Number For each

Memcache, there could be multiple slabs, slab_class denotes which slab it is.

Process id

^pid

N/A Number Process id of

memcached services- there will be one PID per memcached pool

Up time

^uptime

N/A Number Time since start

of process - denotes how long the process has been active.

Maximum

connections

max_connectio ns

N/A Number Max

connections allowed for this pool

Total connections

total_connect ions

N/A Number Total number of

successful connections since start Current client

connections

curr_connecti ons

N/A Number Number of

Memcached

clients

connected at

this point

(26)

Name Data Field Unit Data Type Notes Details

Total GETs

^cmd_get

N/A Number Number of

times a Get request was made- includes both successful and unsuccessful

Total SETs

^cmd_set

N/A Number Number of

times a Set request was made - includes both successful and unsuccessful

Total Get hits

^get_hits

N/A Number Number of

successful GET requets

Total GET misses

^get_misses

N/A Number Number of

failed "get"

requests because nothing was cached for requested key.

get hits

^get_hits

N/A Number Number of hits

for the slab - this should be >

sn_cmd_set

chunk size

^chunk_size

Bytes Number Size of item

chunks.

Number of chunks in

a page

chunks_per_pa

ge

Bytes Number Total number of chunks in a Page - This will compute as chunk_size*tota l_chunks = 1MB approximately.

Diagnostic Recipe / Alerts

You can keep track of the health of the Memcached pools.

(27)

Metric Field Name / Computation Notes

Fill ratio/percent

bytes/limit_maxbytes

Low water mark 85%; High water mark 90%.

If the Fill ratio is very low (say 10%) consider reducing the CACHESIZE in the configuration file.

Eviction Rate

evictions

It is vital that for the Services pool, the number of evictions be

⁰

. If evictions > 0, then increase the

^CACHESIZE

of the services memcache pool in file

/etc/memcached-services.conf

. With other pools, it is NOT crucial to have evictions be

⁰

, but is always preferred.

Connection Limit

Reached listen_disabled_num This is a poorly named counter which describes how many times you've reached maxconns. Each time memcached hits

maxconns it will delay new connections, which means you'll possibly get timeouts. If

listen_disabled_num > 0, increase MAXCONN in the memcache configuration files.

How to change connection limit and cache size

Memcache parameters are generally present in a configuration file present in the /etc folder. The number of max connections from traffic manager to a memcache pool are determined by the parameter

^MAXCONN

. The max memory allocated to the memcache pool is determined by the parameter

^CACHESIZE

.

Locate the configuration file for the memcache pool for which you want to change the value. The configuration files along with the pool name is as follows.

Pool Name Port Conf File MAXCONN CACHESIZE

MAPIs 11211

/etc/memcached.conf

10240 384M

Counter 11212

/etc/memcached-count.conf

10240 256M

Content 11213

/etc/memcached-

content.conf

10240 384M

Services 11214

/etc/memcached-

services.conf

10240 128M

Packages 11215

/etc/memcached-

packager.conf

10240 128M

HTTPsClientSecur

ity 11216

/etc/memcached-

httpsclientsecurity.conf

10240 128M

To get the settings into effect, restart the Mashery Local nodes. That will trigger restart of the memcache

and traffic manager will use the newly-warmed up cache.

(28)

Memcache Loader

Memcache loader loads data from the MySQL database to Memcache so that Traffic Manager can get faster access to configuration data and process incoming traffic calls. Memcache loader is invoked via a cron job specified in the script /etc/cron.d/cache-loader. Every time memcache loader runs, it records the count of the following data - Services, Packages, Plans, Methods, MethodOverrides, ResponseFilters, Mapis, DevClasses, HttpsClientProfile, Trust and Identity in the log files.

03/Jun/2019:18:53:14 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Mapi, primary_param=updatedmapi=10, area_param=, action=end countMapisLoaded=0

03/Jun/2019:18:53:14 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=DevClass, primary_param=updateddevclass=10, area_param=, action=end countDevClassesLoaded=0

03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258; done loading Plans; countPlansLoaded=0

03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258; done loading Method Overrides;

countMethodOverridesLoaded=0

03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258; done loading Methods; countMethodsLoaded=0

03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Packager, primary_param=updatedpackager=30, area_param=, action=processing areaid=258areaid=258; done loading Response Filters;

countResponseFiltersLoaded=0

03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=Services, primary_param=, area_param=, action=end

countServicesLoaded=0

03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=HttpsClientSecurity,

primary_param=updatedhttpsclientsecurity=30, area_param=, action=end countHttpsClientProfileLoaded=9

03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=HttpsClientSecurity,

primary_param=updatedhttpsclientsecurity=30, area_param=, action=end countTrustLoaded=0 03/Jun/2019:18:53:15 +0000 log_level=INFO, tid=c7df27c2-92ee-4155-95f7-0af5604acf61, stack=java, loader_class=HttpsClientSecurity,

primary_param=updatedhttpsclientsecurity=30, area_param=, action=end countIdentityLoaded=0

Diagnostic Recipe / Alerts

The logs are written in the file

/var/log/mashery/memcacheload.log

. At every run the file is overwritten (you can change this by modifying the file

/etc/cron.d/cache-loader

). This log file can be forwarded to your log processing software.

When to receive alerts:

●

If the logs are not being generated

●

The last log statement generated was more than 30 minutes ago (The memcache loader runs every 10 minutes.)

●

If the count of each type of data consistently stays at 0 even though configuration data has been modified in the API Control Center

In case the memcache loader is not running, consider stopping the

javaproxy

(

service javaproxy stop

), restarting

memcache

(

service memcached restart

) and starting

javaproxy

(

service javaproxy start

)

Status of Cloud Sync Services, Developer, Tokens and Logs

Cloud Sync downloads configuration data that is updated in API Control Center. The configuration data is

made up of Services, (Service) Keys, Packages, PackageKeys, Apps, Classes, HttpsClientSecurity. The cloud

sync also uploads logs and synchronizes tokens.

(29)

The status for "Cloud Sync" can be periodically collected using the URL:

https://<IP address of the node>:5480/service/mashery/cgi/cloud_sync_status_cgi.py

The response is a JSON that shows the time stamp of last successful sync for each type of sync. The following is a sample response.

Cloud Sync Response

{

"Logs": {

"start": "2019-06-12 15:08:01", "is_success": 1,

"end": "2019-06-12 15:08:02", "details": null,

"record": 0 },

"PackageKeys": {

"start": "2019-06-12 15:15:02", "is_success": 1,

"end": "2019-06-12 15:15:02", "details": null,

"record": 0 },

"Keys": {

"start": "2019-06-12 15:15:02", "is_success": 1,

"end": "2019-06-12 15:15:02", "details": null,

"record": 0 },

"Apps": {

"start": "2019-06-12 15:15:02", "is_success": 1,

"end": "2019-06-12 15:15:02", "details": null,

"record": 0 },

"Tokens": {

"start": "2019-06-12 15:22:02", "is_success": 1,

"end": "2019-06-12 15:22:02", "details": "0",

"record": 0 },

"Classes": {

"start": "2019-06-12 15:15:02", "is_success": 1,

"end": "2019-06-12 15:15:02", "details": null,

"record": 0 },

"HttpsClientSecurity": {

"start": "2019-06-12 15:15:02", "is_success": 0,

"end": "2019-06-12 15:15:02", "details": "MOM responded with 596

596\ncom.mashery.proxy.onpremloader.OperationException: MOM responded with 596 596\n

\tat

com.mashery.proxy.onpremloader.OnpremLoader.doSyncHttpsClientSecurity(OnpremLoader.java :2817)\n\tat

com.mashery.proxy.onpremloader.OnpremLoader.syncHttpsClientSecurity(OnpremLoader.java:2 747)\n\tat com.mashery.proxy.internal.onpremloader.OnpremLoaderManager

$LoaderTask.runInterruptibly(OnpremLoaderManager.java:318)\n\tat com.mashery.proxy.internal.onpremloader.OnpremLoaderManager

$LoaderTask.run(OnpremLoaderManager.java:257)\n\tat

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n",

"record": 0 },

"Services": {

"start": "2019-06-12 15:15:02",

(30)

"is_success": 1,

"end": "2019-06-12 15:15:02", "details": null,

"record": 0 },

"Packages": {

"start": "2019-06-12 15:15:02", "is_success": 1,

"end": "2019-06-12 15:15:02", "details": null,

"record": 0 }

}

Parameters

Name Data Field Unit

Data

Type Details

start

^start

N/A datetim

e Start time

end

^end

N/A datetim

e End time

is_success

^is_success

N/A boolean Indicates if the sync operation was successful

record

^record

N/A integer Number of records downloaded

details

^details

N/A text Message from the sync operation

Diagnostic Recipe / Alerts

You can monitor the sync of each type of configuration data (Services and their Keys, Packages and their Keys, Apps, HttpsClientSecurity, Tokens, and Logs). An alert can be set if a successful sync has not happened three times in a row. That is, if the last successful sync happened more than three times of the cloud sync interval in the past, an alert can be generated.

Symptom Action

The parameter "details" contains "MOM

responded with 596". Call TIBCO Support immediately, as it indicates that TIBCO Cloud may be unhealthy.

The parameter "details" indicates that Mashery

OnPrem Manager (MOM) cannot be reached. Check network connectivity of the Mashery Local node to the internet.

No Last successful CloudSync in last three tries. See the log files for possible clues on failures.

(31)

Legal and Third-Party Notices

SOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCO SOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSED TIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHER TIBCO SOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENT FOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATE AGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALLATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN THE LICENSE FILE) OR IF THERE IS NO SUCH SOFTWARE LICENSE AGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE “LICENSE” FILE(S) OF THE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALL CONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME.

ANY SOFTWARE ITEM IDENTIFIED AS THIRD PARTY LIBRARY IS AVAILABLE UNDER SEPARATE SOFTWARE LICENSE TERMS AND IS NOT PART OF A TIBCO PRODUCT. AS SUCH, THESE

SOFTWARE ITEMS ARE NOT COVERED BY THE TERMS OF YOUR AGREEMENT WITH TIBCO, INCLUDING ANY TERMS CONCERNING SUPPORT, MAINTENANCE, WARRANTIES, AND INDEMNITIES. DOWNLOAD AND USE OF THESE ITEMS IS SOLELY AT YOUR OWN DISCRETION AND SUBJECT TO THE LICENSE TERMS APPLICABLE TO THEM. BY PROCEEDING TO DOWNLOAD, INSTALL OR USE ANY OF THESE ITEMS, YOU ACKNOWLEDGE THE FOREGOING DISTINCTIONS BETWEEN THESE ITEMS AND TIBCO PRODUCTS.

This document is subject to U.S. and international copyright laws and treaties. No part of this document may be reproduced in any form without the written authorization of TIBCO Software Inc.

TIBCO, the TIBCO logo, the TIBCO O logo, and TIBCO Mashery are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries.

All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

This software may be available on multiple operating systems. However, not all operating system platforms for a specific software version are released at the same time. Please see the readme.txt file for the availability of this software version on a specific operating system platform.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS.

CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT. TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S) DESCRIBED IN THIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHER DOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND "READ ME" FILES.

This and other products of TIBCO Software Inc. may be covered by registered patents. Please refer to TIBCO's Virtual Patent Marking document (https://www.tibco.com/patents) for details.

Copyright

^©

TIBCO Mashery® Local Monitoring Guide

Monitoring Guide

Version 4.x

November 2020

Contents

TIBCO Documentation and Support Services . . . . 3

Monitoring Mashery Local 4.x . . . .4

Node Metrics . . . . 5

Component Monitoring . . . . 6

. . . .

. . . .

. . . .

. . . .

. . . .

Legal and Third-Party Notices . . . . 31

TIBCO Documentation and Support Services

How to Access TIBCO Documentation

Documentation for TIBCO products is available on the TIBCO Product Documentation website, mainly in HTML and PDF formats.

The TIBCO Product Documentation website is updated frequently and is more current than any other documentation included with the product. To access the latest documentation, visit https://docs.tibco.com.

Product-Specific Documentation

Documentation for TIBCO products is not bundled with the software. Instead, it is available on the TIBCO Documentation site at TIBCO Mashery

Local documentation page.

The following documents for this product can be found on the TIBCO Documentation website:

TIBCO Mashery

Local Release Notes

TIBCO Mashery

Local Installation and Configuration Guide

TIBCO Mashery

Local Migration and Upgrade Guide

TIBCO Mashery

Local Reporting Services

For information on TIBCO Cloud Integration with Mashery, refer to Integrating with TIBCO Mashery.

How to Contact TIBCO Support

You can contact TIBCO Support in the following ways:

For an overview of TIBCO Support, visit http://www.tibco.com/services/support.

For accessing the Support Knowledge Base and getting personalized content about products you are interested in, visit the TIBCO Support portal at https://support.tibco.com.

For creating a Support case, you must have a valid maintenance or support contract with TIBCO. You also need a user name and password to log in to https://support.tibco.com. If you do not have a user name, you can request one by clicking Register on the website.

How to Join TIBCO Community

TIBCO Community is the official channel for TIBCO customers, partners, and employee subject matter

experts to share and access their collective experience. TIBCO Community offers access to Q&A forums,

product wikis, and best practices. It also offers access to extensions, adapters, solution accelerators, and

tools that extend and enable customers to gain full value from TIBCO products. In addition, users can

submit and vote on feature requests from within the TIBCO Ideas Portal. For a free registration, go to

https://community.tibco.com.

Monitoring Mashery Local 4.x

Each node of a Mashery Local cluster has the following main components - Traffic Manager, MySQL and MemCache.

Data Collection

Statistics are collected in the following ways:

by making HTTP calls from your monitoring system.

by forwarding log files to your monitoring system.

Both ways are needed as some statistics can be collected only by making HTTP calls and other statistics by forwarding logs from the host machine where the logs are generated.

Monitoring System

A monitoring system is a log processing software that may provide log collection agents like Splunk

forwarder for Splunk or FileBeat for elastic search. Please consult your devops team on which monitoring

solution to use.

Node Metrics

The health of the node of a Mashery Local cluster can be monitored by tracking the use of resources like CPU, Memory, Disk and Network I/O. Alerts can be set if the resources are going low below certain levels.

For example, alerts can be set if:

Available CPU resources are below 50%,

Swap file is being used which means that RAM is fully utilized, or

Available disk is space is below 20%

The above numbers are just examples. Contact your Mashery Customer Success team for your specific threshold values.There are many third-party tools available to collect these metrics, for example, ELK users can use MetricBeat.

Diagnostic Recipe / Alerts

Metrics Notes

CPU resources are low Consider adding more nodes to distribute traffic Swap file usage has increased and available

memory is low Consider increasing the RAM size for the node

Disk space running low Check log file sizes and consider purging logs.

Check expired token count in MySQL and

consider deleting more aggressively if the count is high.

Consider increasing the disk space.

Component Monitoring

Traffic Manager

MySQL

Memcache

Memcache Loader

Status of Cloud Sync - Services, Developer, Tokens and Logs

Traffic Manager

Jetty Pool Connection Statistics

Traffic Manager maintains a thread pool to process incoming requests. Monitoring the statistics of this pool can help in determining the health of the traffic manager.

before running the following command.