This blog post discusses optimizing data pipelines in cloud environments using AWS services. It outlines a serverless data pipeline architecture using AWS Glue, Amazon S3, and Amazon QuickSight. The post explains how to extract, transform, and load data from various sources into a centralized data lake on S3 using AWS Glue jobs. It provides tips for optimizing AWS Glue performance and scaling resources. The article also covers strategies for improving data delivery to QuickSight using SPICE caching. Finally, it demonstrates how to automate and orchestrate the entire pipeline using AWS Step Functions and CloudWatch event triggering, ensuring timely updates of QuickSight datasets for up-to-date insights. The post uses real-world examples from Helsinki’s public transit data to illustrate these concepts.