As part of our Innovation Team, we are seeking a Certified Senior Databricks Engineer / Tech Lead with 7–8 years of hands-on experience in building scalable data platforms. This role will focus on designing, building, and operationalizing data solutions on the Databricks platform to accelerate advanced analytics and AI use cases.
Key Responsibilities:
- Architect, develop, productionize and maintain end to end solutions in Databricks
- Implement and optimize ETL/ELT processes for structured and semi-structured data
- Leverage Delta Lake for ACID transactions, data versioning, and time-travel features
- Drive adoption of the Lakehouse architecture to unify data warehousing and AI/ML workloads
- Implement CI/CD pipelines using Databricks Repos, Asset Bundles, and integration with DevOps tools
- Configure and enforce Unity Catalog for secure, governed access to data assets
- Design and implement data quality and validation frameworks to ensure trusted data
- Lead performance tuning and optimization efforts for Spark jobs and queries
- Integrate with external systems such as Kafka, Event Hub, and REST APIs for real-time and batch processing
- Collaborate with data scientists and business stakeholders to build feature-rich datasets and reusable assets
- Troubleshoot and debug complex data workflows in development and production environments
- Guide junior engineers and contribute to best practices in data engineering and platform usage
- Ensure platform security, access controls, and compliance with enterprise data governance standards.
Required Skills:
- Expertise in Apache Spark and Databricks platform
- Experience with Databricks Lakehouse architecture
- Delta Lake concepts
- Proficient in PySpark, SQL, and Delta Lake
- Strong knowledge of Data Engineering concepts
- Experience with data ingestion, ETL/ELT pipelines
- Familiarity with Unity Catalog and data governance
- Hands-on with Databricks Notebooks and Jobs
- CI/CD automation with Databricks Repos and DevOps, Asset Bundles
- Databricks Asset Bundle implementation knowledge
- Strong understanding of performance tuning in Spark
- Data quality and validation framework implementation
- Experience in handling structured, semi-structured data
- Proficient in debugging and troubleshooting
- Collaboration with data scientists and analysts
- Good understanding of security and access control
- Experience with Mosaic AI or Databricks ML capabilities
- Exposure to streaming pipelines using Structured Streaming
- Familiarity with data observability and lineage tools.