Monitor Boomi Runtime with AWS Cloudwatch
AWS Cloudwatch
AWS CloudWatch can be used to monitor logs and metrics. To use AWS CloudWatch with a Boomi runtime there need to be two agents involved. First, the amazon-cloudwatch-agent will be used to send data to CloudWatch and the second agent will be collectd, which will be used to retrieve JMX properties from the Boomi runtime.
This guide will cover how to manually enable CloudWatch and the latter section will provide a script to automate the process. The most current versions of the scripts can be found on GitHub.
The installation is set up to be used for RHEL (Red Hat) v8. The guide will first cover all the parts. The latter section will cover what needs to be executed from the command line.
Before getting started add IAM Role to an EC2 instance
The CloudWatch Agent uses a policy that is attached to a role, and then the role is attached to the EC2 instance. CloudWatchAgentAdminPolicy is the AWS-managed policy that will be used. A copy of the policy is enclosed within the repo, amazon-cloudwatch-agent.json.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"ec2:DescribeVolumes",
"ec2:DescribeTags",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"logs:CreateLogStream",
"logs:CreateLogGroup"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
}
]
}Installing collectd
collectd is a powerful and robust service that is used to collect metrics. It’s built on C and uses plug-ins for it’s functionality. We’ll be covering the plug-ins for logging of collectd, java, and jmx.
First, we will want to install a few packages. I’ll be assuming RHEL as the linux distribution. Please update the install packages for the desired Linux distribution.
# Install the Amazon Linux Extras package
sudo yum install -y amazon-linux-extras
# Install a local version of java.
# You can use the Boomi's runtime version of Java as an additional option.
amazon-linux-extras install -y java-openjdk11
# Install collectd and additional packages
amazon-linux-extras install -y collectd
sudo yum install -y collectd-java
sudo yum install -y collectd-generic-jmxOnce, that is installed, next create a symlink for libjvm.so. There will likely be two or more libjvm.so files on the server. To find all locations execute locate libjvm.so. collectd is assuming that it is located within /usr/lib64/ and will fail to load if it is not found. There are other ways to achieve this same configuration but I found this to be the easiest to implement.
# Look for the libjvm.so that's within the java-11-openjdk directory.
# The directory is from the java-openjdk11 that was installed earlier.
libjvm_location=$(locate libjvm.so | grep -m 1 'java-11-openjdk')
ln -s $libjvm_location /usr/lib64/libjvm.soNext, modify the collectd.conf file and overwrite what is currently within /etc/collectd.conf.
Within collectd.conf there are 4 plugins being used. A log file will write the collectd logs to /var/log/collectd.log. This file can be helpful to troubleshoot issues. The LogLevel can be changed to info, which will populate the log file with metrics. The conf file below has it set to warning to limit the amount of data within the file. The write_log plugin defines the format that data is written to in the collectd.log file.
The java plugin has a java and a GenericJMX plugin within. They are used to monitor the JMX properties. The connection section defines how to connect to the Boomi runtime.
The network plugin acts as a server for the Amazon CloudWatch Agent to listen to for metrics.
Once complete, the systemd service must be enabled and started.
systemctl enable collectd
systemctl start collectdcollectd has now been set up.
Installing Amazon CloudWatch Agent
# Install Amazon CloudWatch Agent
sudo yum install -y amazon-cloudwatch-agentamazon-cloudwatch-agent will get installed to /opt/aws/amazon-cloudwatch-agent/bin. There are a few ways to start amazon-cloudwatch-agent. bin/amazon-cloudwatch-agent-config-wizard will start the auto-config. This is helpful if you do not know what you want.
For us, we are going to use the config file that is below and implement it by running the following command. The user that the server is being run as is defined. Additionally, under the collectd metrics, the connection that was referenced earlier under network is defined.
# Set the CloudWatch Agent's directory
AWS_CLOUDWATCH_AGENT_HOME="/opt/aws/amazon-cloudwatch-agent/bin"
# Create or overwrite what is in the amazon-cloudwatch-agent.json
# with what is below
sudo vi $AWS_CLOUDWATCH_AGENT_HOME/amazon-cloudwatch-agent.json
$AWS_CLOUDWATCH_AGENT_HOME/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:$AWS_CLOUDWATCH_AGENT_HOME/amazon-cloudwatch-agent.json
# Enable and start the amazon-cloudwatch-agent service
sudo systemctl enable amazon-cloudwatch-agent
sudo systemctl start amazon-cloudwatch-agentUsed the amazon-cloudwatch-agent.json.
Bash Script for Automation
The previous sections outline the manual steps. The aws-cloudwatch-boomi-monitor-rhel-8.sh script can automate those steps. Place the bash script, collectd.conf, and amazon-cloudwatch-agent.json files into the same directory by cloning the repo. Then execute the aws-cloudwatch-boomi-monitor-rhel-8.sh bash script. There are 2 variables that are at the beginning of the script: BOOMI_INSTALL_DIR and CLOUDWATCH_LOG_GROUP_NAME. Both of these must be updated.
#!/bin/sh
# Run script as root
# Developed for RHEL 8
# Variables
BOOMI_INSTALL_DIR=/home/ec2-user/Boomi_AtomSphere/Molecule/Molecule_cloudwatch_molecule
CLOUDWATCH_LOG_GROUP_NAME=boomi-molecule-cloudwatch-02
# Install collectd
echo "Installing java and collectd ..."
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y
/usr/bin/crb enable
dnf install -y java-11-openjdk
dnf install -y collectd
dnf install -y collectd-java
dnf install -y collectd-generic-jmx
# Disable SELinux for collectd
echo "Disabling SELinux..."
setenforce 0
sed -i "s%SELINUX=enforcing%SELINUX=disabled%g" /etc/selinux/config
# Looking for libjvm.so from the java-11-openjdk package that was installed
LIBJVM_SYMLINK=/usr/lib64/libjvm.so
if [ -L ${LIBJVM_SYMLINK} ] && [ -e ${LIBJVM_SYMLINK} ]; then
echo "Synlink to libjvm.so already exists. Skipping..."
else
libjvm_location=$( find / -name libjvm.so | grep -m 1 'java-11-openjdk')
echo "libjvm_location: $libjvm_location"
ln -s $libjvm_location /usr/lib64/libjvm.so
fi
# Load collectd config file
HOST_NAME=$(hostname | xargs)
echo "LoadPlugin logfile
LoadPlugin java
LoadPlugin network
LoadPlugin write_log
<Plugin logfile>
LogLevel warning
File \"/var/log/collectd.log\"
Timestamp true
PrintSeverity false
</Plugin>
<Plugin \"java\">
# required JVM argument is the classpath
# JVMArg \"-Djava.class.path=/installpath/collectd/share/collectd/java\"
# Since version 4.8.4 (commit c983405) the API and GenericJMX plugin are
# provided as .jar files.
JVMARG \"-Djava.class.path=/usr/share/collectd/java/collectd-api.jar:/usr/share/collectd/java/generic-jmx.jar\"
LoadPlugin \"org.collectd.java.GenericJMX\"
<Plugin \"GenericJMX\">
# Memory usage by memory pool.
<MBean \"memory\">
ObjectName \"java.lang:type=Memory,*\"
InstancePrefix \"memory-heap\"
#InstanceFrom \"name\"
<Value>
Type \"memory\"
#InstanceFrom \"\"
Table false
Attribute \"HeapMemoryUsage.max\"
InstancePrefix \"HeapMemoryUsage.max\"
</Value>
<Value>
Type \"memory\"
#InstanceFrom \"\"
Table false
Attribute \"HeapMemoryUsage.used\"
InstancePrefix \"HeapMemoryUsage.used\"
</Value>
</MBean>
<MBean \"garbage_collector\">
ObjectName \"java.lang:type=GarbageCollector,*\"
InstancePrefix \"gc-\"
InstanceFrom \"name\"
<Value>
Type \"invocations\"
#InstancePrefix \"\"
#InstanceFrom \"\"
Table false
Attribute \"CollectionCount\"
</Value>
<Value>
Type \"total_time_in_ms\"
InstancePrefix \"collection_time\"
#InstanceFrom \"\"
Table false
Attribute \"CollectionTime\"
</Value>
</MBean>
<MBean \"os\">
ObjectName \"java.lang:type=OperatingSystem\"
InstancePrefix \"os-\"
InstanceFrom \"name\"
<Value>
Type \"guage\"
InstancePrefix \"os-open-file-\"
Attribute \"OpenFileDescriptorCount\"
</Value>
<Value>
Type \"guage\"
InstancePrefix \"os-load-average\"
Attribute \"SystemLoadAverage\"
</Value>
</MBean>
<Connection>
Host \"${HOST_NAME}\"
ServiceURL \"service:jmx:rmi://localhost:5002/jndi/rmi://localhost:5002/jmxrmi\"
Collect \"memory\"
Collect \"garbage_collector\"
</Connection>
</Plugin>
</Plugin>
<Plugin \"write_log\">
Format JSON
</Plugin>
<Plugin network>
<Server \"127.0.0.1\" \"25826\">
SecurityLevel None
</Server>
</Plugin>
" | tee /etc/collectd.conf
# Install Amazon Cloudwatch Agent
echo "Installing Amaon Cloudwatch Agent ..."
dnf install -y wget
wget https://s3.amazonaws.com/amazoncloudwatch-agent/redhat/amd64/latest/amazon-cloudwatch-agent.rpm
sudo rpm -U ./amazon-cloudwatch-agent.rpm
AWS_CLOUDWATCH_AGENT_HOME="/opt/aws/amazon-cloudwatch-agent/bin"
INTERNAL_IP_ADDRESS=$(hostname -I | sed 's/\./_/g' | xargs)
echo "{
\"agent\": {
\"metrics_collection_interval\": 30,
\"run_as_user\": \"root\"
},
\"logs\": {
\"logs_collected\": {
\"files\": {
\"collect_list\": [
{
\"file_path\": \"${BOOMI_INSTALL_DIR}/logs/*.container.${INTERNAL_IP_ADDRESS}.log\",
\"log_group_name\": \"${CLOUDWATCH_LOG_GROUP_NAME}\",
\"log_stream_name\": \"{hostname}\",
\"timestamp_format\": \"%b %d, %Y %I:%M:%S %p %Z\",
\"multi_line_start_pattern\": \"{datetime_format}\"
}
]
}
}
},
\"metrics\": {
\"metrics_collected\": {
\"collectd\": {
\"metrics_aggregation_interval\": 60,
\"service_address\": \"udp://127.0.0.1:25826\",
\"collectd_security_level\": \"none\"
},
\"disk\": {
\"measurement\": [
\"used_percent\"
],
\"metrics_collection_interval\": 30,
\"resources\": [
\"*\"
]
},
\"mem\": {
\"measurement\": [
\"mem_used_percent\"
],
\"metrics_collection_interval\": 30
}
}
}
}" | tee $AWS_CLOUDWATCH_AGENT_HOME/amazon-cloudwatch-agent.json
$AWS_CLOUDWATCH_AGENT_HOME/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:$AWS_CLOUDWATCH_AGENT_HOME/amazon-cloudwatch-agent.json
echo "Setting up systemd for collectd and amazon-cloudwatch-agent"
systemctl enable collectd
systemctl stop collectd
systemctl start collectd
systemctl enable amazon-cloudwatch-agent
systemctl stop amazon-cloudwatch-agent
systemctl start amazon-cloudwatch-agent
echo "Installation complete!"Setting Up Cloudwatch
Go to Cloudwatch and use the following path to start monitoring the JMX metrics. Metrics -> All Metrics -> Browse -> CWAgent -> ImageId, InstanceId, InstanceType, instance, type, type_instance (there are multiple that look similar)
Troubleshooting
Two log files can be useful for determining errors.
# Amazon CloudWatch Logs
vi /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log
# collectd log file.
# Update LogLevel within collectd.conf to info to see more detail
vi /var/log/collectd.logAdditional Links
Installing CloudWatch Agent on EC2
How to Better Monitor Your Custom Application Metrics Using Amazon CloudWatch Agent
The article was originally posted at Boomi Community.