Job Summary:
We are seeking a highly experienced Databricks Architect to lead the design and implementation of a scalable Lakehouse architecture supporting payer analytics, regulatory reporting, and advanced healthcare use cases. This role will play a critical part in enabling data-driven decision-making across value-based care, risk adjustment, and quality measures such as HEDIS.
The ideal candidate will have deep expertise in Databricks, Apache Spark, AWS cloud services, and healthcare data standards, along with a strong focus on data governance, security, and performance optimization.
Key Responsibilities:
• Lead the design and implementation of Databricks Lakehouse architecture for payer analytics and reporting
• Architect scalable and high-performance data pipelines using Apache Spark and Delta Lake integrated with AWS services
• Enable advanced analytics use cases including value-based care, risk adjustment, HEDIS measures, and population health
• Leverage Gen AI and Agentic AI capabilities to accelerate code development, testing, and deployment processes
• Ensure data security, governance, and compliance (HIPAA, PHI) using Unity Catalog, encryption, and private networking
• Define and enforce data architecture standards, best practices, and governance frameworks
• Collaborate with business and analytics teams to translate requirements into robust data solutions
• Optimize performance, scalability, reliability, and cloud cost efficiency for large-scale healthcare data workloads
• Provide technical leadership, mentorship, and guidance to engineering teams
• Drive CI/CD, automation, and DevOps practices within the data platform
Required Skills & Qualifications:
• 10+ years of experience in data engineering, data architecture, or related roles
• Strong hands-on experience with Databricks Lakehouse platform
• Deep expertise in Apache Spark (PySpark/Scala/SQL) and Delta Lake
• Extensive experience with AWS services (S3, Glue, Lambda, Redshift, etc.)
• Strong understanding of data governance and security frameworks, including HIPAA and PHI handling
• Experience with Unity Catalog and secure data access patterns
• Familiarity with healthcare data standards such as FHIR, HL7, EDI (X12)
• Experience designing and optimizing large-scale distributed data systems
• Strong problem-solving, communication, and stakeholder management skills
Preferred / Good to Have:
• Experience implementing AWS HealthLake, Clinical Data Platforms, or Healthcare Data Lakes
• Exposure to Terminology Servers and Master Patient Index (MPI) solutions
• Experience with real-time/streaming pipelines (Kafka, Kinesis)
• Knowledge of data quality frameworks and observability tools
• Certifications in Databricks or AWS
• Experience working in payer/provider healthcare environments
Key Competencies:
• Strategic thinking and architecture design
• Leadership and mentoring
• Stakeholder collaboration
• Innovation with emerging technologies (AI/ML, GenAI)
• Strong focus on data quality, compliance, and governance