Back to Open Positions
Databricks Team Lead (PySpark)
(Data Engineering)
Full timeBangalore7 to 10 Years01 Opening
Job Description
- Lead a team of Data Engineers and provide technical guidance and mentorship.
- Design and implement scalable data solutions using Databricks and PySpark.
- Define architecture, coding standards, and best practices, and oversee development, testing, deployment, and support activities.
- Collaborate with business and technical stakeholders to gather requirements and define solutions.
- Optimize data pipelines for performance, scalability, and reliability.
- Manage sprint planning, task allocation, and delivery tracking, and support production environments to resolve critical issues.
- Drive continuous improvement initiatives across the data platform.
Skills and Experience Required
- Databricks and Apache Spark, with 4+ years hands on Databricks experience
- Strong expertise in PySpark and advanced SQL with data modeling
- Delta Lake, Unity Catalog, and Databricks Workflows
- ETL and ELT pipeline design, data warehousing, and Lakehouse architecture
- Proficiency in Python programming
- Azure Data Factory, Azure Data Lake Storage, and similar cloud data services
- RAG architectures, Azure OpenAI, OpenAI APIs, and GenAI frameworks such as LangChain or LlamaIndex
- Git, CI/CD pipelines, and DevOps practices, plus data governance, security, and performance optimization
- Databricks certification, with team leadership and Agile or Scrum delivery experience