Data Engineering Manager

Overview The Data Engineering Manager is responsible for • Design and build a metadata-driven file ingestion framework on Databricks aligned with medallion architecture • Lead onboarding of datasets using pattern-based ingestion (not one-off pipelines) • Define and enforce data contracts including schema, keys, and data quality rules • Implement scalable ingestion patterns supporting header / no-header / schema evolution scenarios • Drive table-driven configuration model to eliminate dependency on static YAML-based onboarding • Integrate ingestion framework with governance tools (e.g., DataHub) for lineage, discovery, and ownership • Ensure strong data quality, auditability, and observability through centralized logging and control tables • Collaborate with business and source system teams to understand data semantics and define keys • Establish best practices for file ingestion, schema management, and incremental processing • Lead vendor teams to ensure delivery aligns with target architecture and platform standards Education • BLD - Bachelor's Level Degree in computer science required • 15 years of experience in a complex computing environment may be considered in lieu of degree. Certification/Licensure • Databricks Certification (Preferred) Experience • 8+ years in data engineering with strong experience in modern data platforms (Azure preferred) • Hands-on expertise in Databricks and PySpark for large-scale data processing • Strong understanding of medallion architecture (Bronze / Silver / Gold) • Experience designing metadata-driven or configuration-driven data pipelines • Deep knowledge of data ingestion patterns (batch, file-based, incremental loads) • Strong experience with data modeling concepts (keys, SCD, merge strategies) • Experience implementing data quality frameworks and validation rules • Proficiency in SQL and distributed data processing concepts • Experience with Azure Data Lake Storage (ADLS) and file-based ingestion patterns • Familiarity with CI/CD, Git, and deployment practices in data engineering Preferred Experience • Experience with data catalog and governance tools (DataHub, Collibra, Alation, etc.) • Exposure to customer 360 / MDM / CRM data integration (Salesforce, Dynamics, etc.) • Experience working in healthcare or regulated environments (HIPAA, PHI handling) • Familiarity with Unity Catalog and data access governance models • Experience integrating data platforms with workflow tools like ServiceNow for access management • Exposure to event-driven ingestion patterns (file triggers, streaming, etc.) • Experience working in large-scale migration or modernization programs ,enterprise data platforms • Experience enabling self-service data access and governance • Familiarity with API-driven data integration patterns We provide market-competitive compensation packages, inclusive of base pay, incentives, and benefits. The base pay rate for Full Time employment is:$116,729.60-$216,777.60. Additional compensation may be available for this role such as shift differentials, standby/on-call, overtime, premiums, extra shift incentives, or bonus opportunities.