The low cost of getting started with cloud services can easily evolve into a significant expense down the road. That's challenging for teams developing data pipelines, particularly when rapid changes in technology and workload require a constant cycle of redesign. How do you deliver scalable, highly available products while keeping costs in check?
With this practical guide, author Sev Leonard provides a holistic approach to designing scalable data pipelines in the cloud. Intermediate data engineers, software developers, and architects will learn how to navigate cost/performance trade-offs and how to choose and configure compute and storage. You'll also pick up best practices for code development, testing, and monitoring.
By focusing on the entire design process, you'll be able to deliver cost-effective, high-quality products. This book helps you:
- Reduce cloud spend with lower cost cloud service offerings and smart design strategies
- Minimize waste without sacrificing performance by rightsizing compute resources
- Drive pipeline evolution, head off performance issues, and quickly debug with effective monitoring
- Set up development and test environments that minimize cloud service dependencies
- Create data pipeline code bases that are testable and extensible, fostering rapid development and evolution
- Improve data quality and pipeline operation through validation and testing
With over 20 years of experience in the technology industry Sev brings a breadth of experience spanning circuit design for Intel microprocessors, user-driven application development, and data platform development at both small and large scale. Throughout his career Sev has been a writer, speaker, and teacher along with his technical contributions, seeking to pass on what he has learned and make technology education accessible to all. Sev's experience developing cloud data pipelines across multiple cloud service providers in large-scale batch and real-time environments, alongside his established record of writing and teaching, make him uniquely qualified to write Cost-effective Data Pipelines. Sev's hands-on experience as a data-engineer coupled with his ability to synthesize ideas provide him both with the subject matter expertise to speak on the topics in Cost-effective Data Pipelines and to elucidate these advanced concepts to readers. Sev's focus on providing actionable, hands-on content in his classes, tutorials, and interactive sessions guarantees an approach that readers will be able to quickly put into practice.
Lisainfo e-raamatute kohta