In this book we argue for the need for a new approach to data provenance and explain how recent advancements in data processing workflow automation present an opportunity to address this need. We introduce descriptive dataflow operators - a novel approach based on integrating descriptive workflow languages with a data modeling Domain-Specific Language (DSL). We review the current workflow automation technologies and propose a DSL that supports complex data transformations, enhances reproducibility, and enables precise data lineage tracking. Within the framework we introduce the concept of descriptive dataflow operators for more flexible and expressive data transformations.
Modern healthcare increasingly relies on complex data pipelines to process diverse diagnostic information, clinical records, and research data. This growing complexity, combined with emerging AI/ML applications and stricter regulatory oversight, demands sophisticated approaches to data preparation, documentation, and validation. Healthcare organizations face mounting pressure to ensure granular traceability and reproducibility of their data transformations while maintaining regulatory compliance. These challenges are particularly acute in research settings, where data provenance and quality validation become critical for scientific reproducibility and regulatory adherence.
Given the increasing complexity of healthcare data, data ingestion and transformation workflows present significant technical challenges, particularly in ensuring the reproducibility and seamless integration of diverse datasets for ML and AI model development.
We introduce the Dorieh Data Platform as an exemplar implementation of a DSL, providing a comprehensive framework for reproducible research. The platform's infrastructure supports robust data lineage documentation, validation, and error logging, making it a powerful tool for healthcare data analysis by ensuring transparent, auditable data processes and regulatory conformance.
We show how to apply this framework to analyze healthcare claims data quality, revealing insights into inconsistencies and deficiencies. Our approach demonstrates the potential for improved data management and accountability in scientific research, underscoring the necessity for precise, reproducible data transformation methodologies to produce reliable research outcomes.
Chapter
1. Introduction.- Part I. The Need, the Opportunity and the
Solution.
Chapter
2. The Need for Complete Data Provenance.
Chapter
3. The
Opportunity: Advancement of Workflow Automation.
Chapter
4. Solution:
Descriptive Dataflow Operators.
Chapter
5. Language Design.- Part II. How to
Implement It.
Chapter
6. Proof of Concept Implementation.
Chapter
7. Sample
Application: Building ML-Ready.
Chapter
8. Dorieh Medicare Claims Data
Pipeline.
Chapter
9. What is Next?.- Part III. The Architecture of Trust:
Regulation, Provenance and Compliance-As-Code.
Chapter
10. Trust and
Regulation as Delegated Understanding.
Chapter
11. The Burden Spiral: When
Technology Exceeds Human Oversight.
Chapter
12. The Provenance Imperative.-
Chapter
13. From Provenance to Computable Trust.- Part IV. Classification of
Data Transformations.
Chapter
14. A Taxonomy of Data Transformations.-
Appendix.
Michael Bouzinier is a Senior Research Software Engineer within University Research Computing. He has over 30 years of diverse experience in software research and development and 10 years as a professional educator. His intellectual interests include semiotics, natural language processing and text analytics, data visualization, evolutionary and medical genetics, computer simulations, and explainable AI. Throughout his career, he has worked and led diverse international teams, successfully collaborating with developers and researchers from within the US, UK, Sweden, Finland, Belgium, The Netherlands, and Japan.
Dmitry Etin is a digital health strategist specializing in interoperability and health data management at the intersection of technology, medicine, and policy. He guides organizations and policymakers through the complexities of health data governance and large-scale interoperability initiatives, bringing practical solutions to critical challenges in digital health.He is deeply involved in shaping the European Health Data Space, working with the European Commission and supporting the European Medicines Agency in enabling an interoperable European Medicines Regulatory Network. Dmitry is also involved in several Horizon-funded research initiatives, accelerating the adoption of interoperable EHR systems in the EU. As a co-founder of Forome, an open-source initiative for genomics, health data management, and regulatory compliance, he helps develop data provenance and analysis tools for complex healthcare and clinical research cases.
Naeem Khoshnevis is a Research Software Engineer within University Research Computing. In this role, Naeem designs, builds, and optimizes software applications for researchers across Harvard University. Naeem has a superior mathematical and numerical analysis background and has developed, documented, debugged, extended, and refactored numerous scientific software applications for research groups, helping them successfully carry out their projects.
Max Shad is the Director of Engineering at the Kempner Institute for the Study of Natural and Artificial Intelligence and University Research Computing and Data (RCD) at Harvard University. In this role, he leads the computational program of the Kempner Institute, ensuring the provision of advanced Research Computing (RC) tools/services and expert Research Software Engineering (RSE) support. His efforts are instrumental in leveraging High-Performance Computing (HPC), particularly in Machine Learning (ML) and AI research, to facilitate pioneering discoveries in AI, ML, and computational biology.
Scott Yockel is the University Research Computing Officer at Harvard. In this role, Scott works with researchers across campus to develop and champion a university-wide research computing strategy in support of Harvards research mission. He is focused on identifying emerging needs, engaging with faculty, school, and university leadership to articulate those needs, and identifying possible solutions and funding mechanisms. He is spearheading the implementation of these initiatives and articulating their success with concrete measures.
