Senior Data Engineer - UAE
Position Overview
Location: Abu Dhabi, UAE
Work Requirement: Client requirement to have people onsite with office visits.
Key Responsibilities
The successful candidate will be responsible for:
- Implementing Databricks pipelines using Delta Live Tables (declarative pipelines).
- Rewriting proprietary Foundry transforms into optimised PySpark on Databricks.
- Moving from Foundry's build system to modern CI/CD (IaC-managed Databricks workspaces).
- Performing performance tuning on Databricks Photon to match or exceed Foundry interactivity.
- Contributing to the reconciliation engine (Palantir vs Databricks output equivalence).
- Managing pipeline rebuild, validation, internal testing, and UAT preparation.
Required Skills & Experience
Candidates must demonstrate strong hands-on experience in the following areas:
- Proven production-level Databricks experience.
- Strong hands-on coding capability.
- Experience building and optimizing modern data pipelines.
- Ability to work within pod-based delivery teams.
- Experience owning pipeline migration and rebuild activities from design through validation.
Databricks
- Databricks hands-on experience.
- Delta Live Tables.
- Unity Catalog.
- Workflows.
- Production experience required (not lab-based experience).
Data Engineering & Development
- PySpark.
- Advanced SQL.
- Data modelling.
- Schema design.
- Medallion architecture.
- Lakehouse architecture.
DevOps & Deployment
- Modern CI/CD for data pipelines.
- Infrastructure as Code (IaC).
- Git-based deployment.
Communication
- Strong written and verbal English skills.
- Ability to interact directly with Databricks Professional Services (DBX).
Preferred Qualifications
The following experience is highly desirable:
Certifications
- Databricks Certified Data Engineer Professional.
- Associate certification at minimum.
Domain Experience
- Financial services.
- Audit.
- Asset management.
Platform Experience
Prior Palantir Foundry exposure, including:
- Code Repo.
- Code Workbook.
- Pipeline Builder.
Migration Experience
Experience with migrations from legacy platforms such as:
- Informatica.
- Hyperion.
- Oracle.
- Palantir Foundry.
- Databricks.
Skills
Candidates with any of the following experience will be advantageous:
- Azure Data Factory (ADF).
- Azure Synapse.
- dbt.
- Airflow.
- Equivalent orchestration tools.
Note: Client estate is Azure-heavy.