Amazon’s native monitoring and management service AWS CloudWatch is great for basic monitoring and alerts. However, on its own, it may not be the best solution for analyzing log data at scale — especially if you need to analyze data outside of AWS. Many teams may find themselves restricted by retention issues and basic analytic features with Amazon CloudWatch logs for troubleshooting use cases.
Whether cloud infrastructure logs across AWS services, applications, microservices, container logs, Lambda functions, security telemetry data, or network device logs, CloudWatch can be tough to use under the weight of non-stop log streams. Since cloud-based applications and infrastructure generate millions (even billions) of logs – teams might be losing out on a wealth of insights if they have to make retention tradeoffs.
Let’s dive into why log analytics are important, some of the challenges with the CloudWatch log process, and how to overcome them.
READ: Leveraging Amazon S3 Cloud Object Storage for Analytics
Why Log Analytics is Important for Cloud-Native Companies
If you’re analyzing logs in CloudWatch, chances are you might be looking for answers for ITOps, DevOps, security or customer analytics purposes. Let’s elaborate on each use case:
- ITOps: Platform and site reliability engineers analyze IT logs from applications, devices, systems and network infrastructure. This helps them monitor and adjust IT service delivery, to improve performance and reliability.
- DevOps Analysis: DevOps engineers analyze IT logs to track and optimize the software development lifecycle. This helps them speed up releases, reduce bugs, and improve collaboration between development and operations teams.
- Security Analysis: Security administrators analyze logs of events such as user authentication attempts, firewall blocking actions, file integrity checks, and the detection of malware. They use their findings to predict, assess and respond to threats, and assist compliance efforts.
- Customer Analytics: Marketing managers and BI analysts study customers’ digital footprints, including website clicks, asset downloads, service requests and purchases. Analyzing these logs helps describe customer behavior, identify intent, predict behavior and prescribe action.
These use cases have a common problem: processing data at scale.
This issue crops up across a wide variety of observability, monitoring and dashboarding tools such as Elasticsearch, CloudWatch and Datadog. Fortunately, organizations can supplement these tools with alternatives that allow for powerful log analytics at scale with unlimited data retention.
WATCH: Choosing an Analytical Cloud Data Platform: Trends, Strategies & Tech Considerations
CloudWatch Log Management Pain Points
To execute properly on these use cases, organizations must be able to access large volumes and wide varieties of log data for analysis. One of the biggest challenges with CloudWatch is that it can quickly become expensive and requires care in how it is deployed and utilized at scale. Furthermore, while CloudWatch is great for creating alarms and monitoring real-time application performance – it’s not ideal for deeper troubleshooting use cases.
When it comes to troubleshooting and root cause analysis, CloudWatch has a complex UI. Once organizations collect a high enough volume of logs, filtering and searching in the CloudWatch interface becomes far too complicated. Finding the root cause of an error involves scrolling manually through pages and pages of CloudWatch log groups to locate the specific invocation that threw an error.
Even then, data integration might still be a problem. CloudWatch lacks the data integration depth and correlation features necessary to recognize very complex patterns or perform root cause analysis across larger and multiple data sources.
In addition, querying and scaling data isn’t the best use case for CloudWatch. Once teams reach terabyte-scale (and need log retention beyond a short period of time, such as a few days or a week), CloudWatch simply becomes impractical. This is especially true if you need a longer retention period for compliance reasons or to tap into the value of long-term log storage for the use cases described above.
READ: Achieving Better CloudWatch Log Insights
How to Improve CloudWatch Log Workflows with ChaosSearch
New cloud-based platforms can alleviate some of the most common bottlenecks with log data:
“They rapidly and dramatically compress indexed data, which is critical given the high number and small content of logs. Users can automatically discover, normalize, and catalog all those log files and assemble metadata to improve query planning—all with a smaller footprint than predecessors such as Lucene. The log data remains in place, but presents a single logical view with familiar visualization tools the user already knows (such as Kibana via open APIs).”
- Kevin Petrie, VP of Research, Eckerson Group
To search data or run complex queries, organizations need to extend their log analytics strategy with other tools like ChaosSearch that support sophisticated log queries and parsing multiple logs at once. Getting started is easy:
- Send logs directly to Amazon S3: Move logs to Amazon S3 and index all log data stored in S3. Sending logs to Amazon S3 allows you to control costs and realize longer-term value from logs due to the economy and scale of S3.
- Connect to ChaosSearch: Grant ChaosSearch read-only access to the raw log buckets. From there, teams can create a new bucket for Chaos Index® to make their data fully searchable or create a few object groups and views.
- Analyze logs via Elastic or SQL APIs: Investigate issues with infrastructure and applications in the ChaosSearch console via Kibana (for troubleshooting), Superset (for relational analytics), Elastic or SQL API.
Using ChaosSearch, organizations can take advantage of unlimited data retention with greater price transparency at scale. Combining CloudWatch with ChaosSearch can help organizations achieve best-in-class monitoring, as well as a deeper understanding of all of the systems that drive business growth.
Additional Resources
Read the Blog: Eliminate Data Transfer Fees from Your AWS Log Costs
Listen to the Podcast: Making the World's AWS Bills Less Daunting
Check out the Whitepaper: The New World of Data Lakes, Data Warehouses and Cloud Data Platforms
