CloudWatch is great – if you require very basic logging and monitoring for the Amazon Web Services (AWS) cloud, at least.
However, the reality is that most teams need more than basic logging and monitoring. They may also need to perform log analytics on data sources from outside AWS, which CloudWatch doesn’t support.
That’s why, although CloudWatch may be one tool in your log analytics strategy, it probably should not be the only one. Keep reading for guidance on how to extend your log management operations beyond CloudWatch to ensure more log analytic depth, breadth and actionability than you can get from CloudWatch alone.
What is CloudWatch?
CloudWatch is Amazon's primary monitoring and logging service built into the AWS cloud. It can collect logs and metrics from all AWS services and workloads hosted on AWS from CloudTrail and EKS, to Route53, to individual EC2 instances running the CloudWatch agent, to Lambda functions. CloudWatch, through CloudWatch Logs Insights, provides basic search and analytics capabilities, such as visualizations, to help you interpret log and metrics data. And it lets you configure alarms, which can alert you to anomalies or sudden changes in workload performance patterns.
The pros and cons of CloudWatch for log analytics
As a cloud monitoring and log analytics service, CloudWatch has some obvious advantages:
- It’s built into the AWS cloud, so you can start using it instantaneously.
- It’s well integrated with most AWS services, which means minimal effort is required to begin collecting logs and metrics.
- It has a consumption-based pricing model. You don’t have to pay anything upfront to use CloudWatch, and you pay for what you use, based on the amount of data you ingest and the number of metrics, alarms, and features you utilize.
On the other hand, there are clear limitations to CloudWatch:
- Cost
- Limitations on Retention
- It only supports AWS: you can’t use CloudWatch log analytics to help monitor workloads hosted in other clouds or on-premises, leading to limits on the ability to centralize all log data and embrace Data Lake philosophy with other data sources
- CloudWatch logs insights analytics features are basic. CloudWatch may be able to alert you to significant anomalies within your logs, for example, but it lacks the data integration depth and correlation features necessary to recognize very complex patterns or perform root-cause analysis across larger and multiple data sources.
While CloudWatch certainly takes a role in most AWS logging and monitoring workflows, it doesn’t usually make sense to rely on CloudWatch alone.
Read: Leveraging Amazon S3 Cloud Object Storage for Analytics
Steps to better log analytics
Instead, most teams should extend their log analytics and monitoring strategies via steps such as the following.
Centralize log analytics across all of your clouds
If you use multiple public clouds at once – as most businesses do today – you need a log analytics strategy that lets you collect and analyze log data from across all of those clouds including hybrid cloud for those who are extending on-premises applications and infrastructure into the public clouds.
Your approach must enable you not simply to collect and analyze logs from each cloud individually but also have the critical ability to correlate and compare log data.
If you want to know, for example, how an application hosted in Azure performs compared to one hosted in AWS, you need multi-cloud-friendly log analytics. You don’t get that from CloudWatch.
Collect and analyze all log data
Even if all of your workloads run in AWS, you may not be able to collect and analyze as much data from them as you would like when using CloudWatch. CloudWatch only supports specific predefined log and metrics types.
So, extend your log analytics strategy beyond CloudWatch by deploying tools that give you complete control over which log and metrics data you expose and how you interact with it. Don’t restrict yourself to choosing from a predefined list of data types.
You can certainly make use of CloudWatch and also leverage ChaosSearch by exporting CloudWatch logs, moving them to S3 and indexing these logs and other log data stored in S3.
Additionally, you may choose to bypass CloudWatch altogether and push logs directly into a more comprehensive and powerful analytics platform. ChaosSearch allows you to do this as it can index any data stored in S3 that is in the log, JSON, or CSV format. There is a vast ecosystem of log shippers and tools to transport data to cloud object storage (Amazon S3) from Logstash and beats, Fluentd, Fluentbit to Vector, Segment.io, Cribl.io or programmatically from Boto3.
Watch this quick demo to learn how to analyze JSON logs with ChaosSearch:
Store data efficiently – and cost-effectively
As we noted, storing log and metrics data in AWS CloudWatch may not always be the most practical or cost-effective approach – especially if you need to retain data for an extended period due to compliance obligations or the untold value the long tail of data may bring for security use cases, forensics, or customer and product analytics.
For that reason, choose a log analytics strategy that gives you the flexibility to store your data wherever, and for as long as you would like. Even if you use CloudWatch to perform initial data collection, unlock additional value by storing all data centrally in Amazon S3 enabling analytics with a more powerful platform like ChaosSearch.
Read: Why Data Retention Windows Fail
Fine-tune your alarms
While CloudWatch supports basic alerting functionality in the form of alarms, CloudWatch alerts are just that – basic. They enable a limited amount of granularity, which means it’s difficult to define different alerts for different parts of your workloads. It’s also hard to configure highly dynamic alarms that factor in complex contextual data before determining whether to fire off an alert or not.
So, instead of relying on CloudWatch as your primary alerting and monitoring tool, look for an external solution that provides more control over alerts with easy integration with any system that supports RESTful webhooks and platforms like PagerDuty, OpsGenie, Slack, Microsoft Teams, and ServiceNow.
Make your log data actionable
CloudWatch can help you visualize your log data, but it’s not very useful if you need to search through the data or run complex queries on it.
To do these things, you’ll need to extend your log analytics strategy with other tools that support sophisticated log queries and that you can use to parse multiple logs at once. CloudWatch’s query engine just isn’t powerful enough to deliver deep, granular insights in many cases.
Read: AWS Monitoring Challenges: Avoiding a Rube Goldberg Approach to AWS Management [VIDEO]
Conclusion
In short, CloudWatch is a valuable tool for gaining a quick overview of the status of AWS workloads. But it’s rarely sufficient on its own as the foundation for a complete log analytics and cloud monitoring strategy. To achieve the latter, you’ll need a log analytics solution that offers features like multicloud support, sophisticated alert configuration, storage flexibility and more.
Additional Resources
Read the Blog: AWS vs GCP: Top Cloud Services Logs to Watch and Why
Listen to the Podcast: Differentiate or Drown: Managing Modern-Day Data
Check out the Whitepaper: DevOps Forensic Files: Using Log Analytics to Increase Efficiency
