Fastly is one of the world’s largest and fastest-growing CDN platforms. They provide their customers with a powerful platform that offers a content distribution network (CDN), the ability for their customers to instantly auto-scale their applications without modifying existing infrastructure, DDoS protection, mitigation service, and a slew of other advanced edge technology features.
For Fastly customers that leverage Amazon Web Services (AWS), the platform can take advantage of the streaming capabilities by sending all Fastly logs to an Amazon S3 bucket. Fastly has made it extremely easy for AWS users to send their logs to the S3 endpoint for long-term analysis on network traffic, which is invaluable when debugging customer issues, security incidents, and understanding user access and growth scaling patterns. This saves the user a step by not having to write code that will continually be downloading and uploading to an object-store. However, you will still need to find a solution to actually query these logs in a fast and cost-efficient way. Commonly recommended solutions to this problem involve moving your data out of Amazon S3 and into an Elasticsearch cluster. But as your log volume grows and your data retention requirements increase, it quickly becomes expensive and time-consuming to continually manage Elasticsearch.
This is where CHAOSSEARCH delivers. Integrate within minutes to your Fastly log bucket, index your data, and write those indexes back into YOUR Amazon S3 account. We provide you with an Elasticsearch API and a fully integrated Kibana interface without ever having to move your Fastly logs. Now, you have the ability to query months and years of your Fastly log data without expensive-to-operate databases and retention limitations. All the data lives within your Amazon S3 account, allowing you to manage the lifecycle of that data based on your needs.
You can configure Fastly in a few simple steps — below are the ones that we followed to start streaming our logs to an S3 bucket:
Step 1 - Fill out the recommended field names:
Step 2 - Add your Bucket name, AWS Access key, and Secret key:
Step 3 - Complete the set-up by selecting Classic for our logline format, Format Version Default for the Placement, and a GZIP level of 9 to increase compression and savings on Amazon S3 storage.
From here, I can go from raw data to insights within minutes using the CHAOSSEARCH service with these logs on my Amazon S3 account. First, I will create an object group, which is a logical grouping of all the logs that I want to include within an index. In this case, the only logs that exist within this bucket are my Fastly logs so I can leave the discovery regex as the default wildcard.
In the next section CHAOSSEARCH will auto-discover the data format of the Fastly logs, if the log files are compressed, and what the structure will be once we apply our regex to parse the logs.
Because the CHAOSSEARCH platform separates storage from compute, we can scale up indexing immediately and process huge amounts of data without the legacy constraints of a distributed database like Elasticsearch. And because of this we can now go into the CHAOSSEARCH fully integrated Kibana interface and start getting answers from our data.
I can now create a series of visualizations to help me understand my usage patterns, such as identifying the requests over time and the corresponding endpoints.
I quickly see which were the top non-200 status codes that were hitting my site.
I can use the native Kibana functionality to now group all these visualizations together in a single dashboard — giving my engineers quick access to all these insights. Due to the power of the CHAOSSEARCH platform, we can go from raw data directly to insights within minutes. We don’t need to spend any time creating index mappings or figuring out what the appropriate schema layout needs to be. No longer do you need to move your data into an ELK stack, or ETL into a data analytics platform. CHAOSSEARCH provides you the ability to retain and query an unlimited amount of data for an unlimited amount of time leveraging the power and cost-effectiveness of Amazon S3.
If you are a Fastly customer and are looking to get more valuable insight from your logs, or reign in the cost of your growing Elasticsearch cluster reach out today for a free trial and see how quickly you can get answers to your data.
