CloudGuild · Blog · Cheat sheets · Lessons · Certifications
Amazon Athena & AWS Glue — Serverless Data Querying and ETL
Athena is a serverless SQL query service for analyzing data in S3, while Glue is a fully managed ETL service that prepares data for analytics.
What it is
- Amazon Athena: Serverless interactive query service that enables you to analyze data in Amazon S3 using standard SQL.
- AWS Glue: Managed ETL service that discovers, prepares, and transforms data for analytics.
When I reach for it
- Use Athena when you need to quickly analyze large datasets in S3 without provisioning servers.
- Use Glue when you need to automate data preparation and transformation workflows, especially for data lakes.
Key architectural decisions
- Choose Athena for ad-hoc querying with minimal setup; consider data formats like Parquet or ORC for better performance.
- Use Glue to create and manage ETL jobs, leveraging its data catalog for schema discovery and metadata management.
- Decide between using Glue Crawlers to automatically catalog data or manually managing schemas in Athena.
Gotchas & exam traps
- Remember that Athena charges per query based on the amount of data scanned, so optimize your queries and data formats.
- Glue may have limitations on job execution time; ensure your jobs can complete within the allowed duration.
The architect view
- Leverage Athena for fast, serverless analytics, while using Glue for robust data pipeline management.
- Keep data in S3 organized and partitioned to optimize costs and performance for both services.