Work, education, and many of our leisure activities moved online in 2020. A catalyst, the global pandemic made the Internet and the SaaS services we use foundational. Ensuring that these systems are operating optimally and securely is of paramount importance.
There's no going back. Each person on the planet now consumes more and more bandwidth than the year prior. This brings new threats to content delivery networks and security platforms.
During the challenging COVID19 crisis, Cloudflare added to their core web application performance and security features, providing safe DNS for families in its 1.1.1.1 mobile app and service.
Cloudflare has a long history of being able to handle scale dealing with the largest DDOS attacks in the history of the Internet - they're built for it.
ChaosSearch is also built for scale, and we are happy to have Cloudflare Enterprise customers who are ingesting TB’s of Cloudflare log data daily to securely harvest actionable insights. All with the data residing in their own Amazon S3 environments. 
Challenges at Scale
Dealing with daily data ingest rates of terabyte and greater scale via other log analytics platforms is extremely challenging for a number of reasons.
From sizing your cluster, outlining storage requirements, estimating the number of shards and nodes, to dealing with failures, the care and feeding of an ELK stack (self-hosted or hosted) or other search platforms, can be daunting.
If you are utilizing other hosted solutions, you may be offloading some of the administrative tasks, but costs and your ability to keep more than a few days of data can quickly become a problem.
ChaosSearch, due to its underlying indexing technology, can index TBs or PBs of data and create an index that is a tiny fraction of the size of the source data, while providing full text search and analytics on this index.
Once the index is created, there is no need for the source data to be maintained, and you can delete it or push it off to Glacier for longer term storage.
ChaosSearch is a fully managed service requiring zero administration and providing the ability to index data at scales that typically choke other solutions.
Cloudflare and ChaosSearch Together
Cloudflare has built-in DDoS protection to help fire-fight attacks and provide actionable insights. Adjustments to your Cloudflare web application firewall add more power to your security capabilities.
With ChaosSearch indexing your Cloudflare logs, you will be able to:
- Monitor, detect, and alert on suspicious activity or patterns.
- Quickly view and identify top IPs, allowing you to neutralize attacks as they happen.
- Monitor and set performance thresholds to ensure that end-users and customers have the best customer experience possible.
- Leverage powerful full-text search with automatic indexing of all fields, providing the ability to “find a needle in a haystack” at TB or PB scale.
- Integrate monitoring and alerting with other systems like Slack, PagerDuty, Jira, OpsGenie, ServiceNow via webhooks, allowing you to proactively bring the additional power of these systems. Automate your workflow based upon monitors that you configure in ChaosSearch.
See all of this in action…
My colleagues Tom O’Connell and Kevin Davis demonstrated all of this power of ChaosSearch and Cloudflare this week. You can replay the webcast recording here:
About the Author
Dave Armlin is VP of Customer Success and Solutions Architecture at ChaosSearch. Dave has spent his 25+ year career building, deploying, and evangelizing Enterprise and Cloud based solutions for companies globally and could not be more excited about the impact ChaosSearch can have on organizations around the globe today.
