Senior Databricks Engineer (Contract)
| Job Type | Contract |
| Area | Western Cape, South AfricaCape Town, South Africa |
| Sector | IT - Data |
| Salary | R350 – R450 Per Hour |
| Start Date | Approximately 1 month |
| Job Ref | VR/26813 |
- Description
Swan IT is searching for a Senior Databricks Engineer for a 10-month contract in Cape Town. The successful candidate should have strong Databricks expertise and will join our Data & Analytics team.
Responsibilities:
- Design, build, and maintain scalable data pipelines using Databricks (Spark, Delta Lake, Databricks SQL).
- Develop efficient ETL/ELT workflows for batch data processing.
- Optimise data processing for performance, scalability, and cost in cloud environments (Azure).
- Work closely with business stakeholders, analysts, and data scientists to deliver trusted data assets.
- Implement data quality, lineage, and governance practices.
- Support CI/CD integration for data workflows.
- Participate in architectural discussions, data modelling, and integration patterns.
- Troubleshoot, monitor, and continuously improve existing pipelines and infrastructure.
Requirements:
- 3–7 years’ experience in data engineering.
- Strong hands-on experience with Databricks (Spark, Delta Lake, Databricks SQL).
- Databricks or cloud certifications (e.g., Databricks Certified Data Engineer Associate/Professional) are required.
- Bachelor’s or Master’s in Computer Science, Information Systems, Engineering, or a related field is a plus, but not required
- Solid SQL and Python programming skills.
- Experience with cloud storage solutions (Azure Data Lake, AWS S3, or GCP Storage).
- Familiarity with orchestration tools (e.g., Airflow, Databricks Workflows).
- Strong understanding of distributed systems and data warehousing principles.
- Experience with version control (Git) and CI/CD practices.
- Experience with metadata driven coding concepts.
- SAP BW experience is highly advantageous.
- Exposure to streaming tools (Kafka, Event Hubs, Pub/Sub) – nice to have, not core.
- Familiarity with Infrastructure as Code (Terraform, Bicep, or CloudFormation).
- Knowledge of MLOps or machine learning pipelines is a plus but not required.
- Understanding of modern data architectures (data lakehouse, data mesh).