Data Developer

Position Details

Job Title: Data Developer (Databricks)

Location: Abu Dhabi, UAE

Work Model: Onsite (5 Days Per Week)

Start Date: ASAP

End Date: 31 October

Position Overview

The Data Developer Consultant will support the final phase of the organisation's Data Platform Modernisation Programme.

This role is focused on delivering the remaining technical activities required to:

• Complete the migration from legacy technologies.
• Standardise engineering practices.
• Prepare the platform for long-term operational support.
• Ensure full operational readiness before transition into business-as-usual (BAU) support.

The engagement is intended to provide experienced technical expertise to execute the "last mile" migration activities, address technical debt, complete platform remediation work, and finalise platform stabilisation efforts.

Key Responsibilities The successful consultant will be responsible for the following:

Data Platform Modernization

• Refactor legacy data models and workflows to align with current engineering standards and best practices.
• Migrate data ingestion processes from legacy implementations to modern Databricks-based solutions.
• Support the decommissioning of legacy catalogues and platform components.
• Complete migration and remediation activities required to finalise the organisation's data platform transformation.

Databricks Engineering

• Review and optimise Databricks cluster utilisation.
• Implement cluster policies and performance optimisation improvements.
• Design, develop, and maintain scalable data pipelines.
• Develop and maintain data ingestion frameworks.
• Support the migration of ingestion processes to approved Databricks patterns and frameworks.

Development & Quality Assurance

• Participate in testing, validation, deployment activities, and code reviews.
• Ensure all deliverables meet defined requirements, standards, and acceptance criteria.
• Perform testing, validation, and quality assurance activities before delivery.
• Contribute to continuous improvement initiatives.

Documentation & Knowledge Transfer

• Document technical processes, standards, and operational workflows.
• Develop and maintain the required business context layer to support operational readiness.
• Contribute to knowledge transfer and support future operational teams.

Team Collaboration

• Support existing team members with:

  • New data ingestion pipelines
  • Data models
  • Platform enhancements
• Work closely with the Data Engineering team to ensure alignment with engineering standards and governance frameworks.
• Support activities required to transition the platform from implementation into steady-state operations.

Required Skills & Experience

Mandatory Experience

Databricks (Minimum 5 Years Required)

Candidates must have a minimum of 5 years' hands-on Databricks development experience, including the following:

• Delta Live Tables (DLT) / Spark Declarative Pipelines
• Auto Loader
• Databricks Asset Bundles (DABs)
• Databricks Cluster Optimization
• Cluster Policy Management

Cloud Environment

• Minimum 5 years of Databricks development experience on AWS environments.
• Strong understanding of cloud-based data engineering architectures.

Additional Technical Skills

Azure Data Factory (ADF)

• 1 to 3 years of experience.

Source Control

• Strong Git experience (mandatory).
• Strong understanding of source control best practices.

Database & Programming Skills

• Strong SQL development skills.
• Experience in SQL optimisation and performance tuning.
• Strong Python programming skills.

Testing & DevOps

• Experience with testing and validation practices.
• Experience with CI/CD deployment processes.
• Familiarity with DevOps methodologies and engineering practices.

Working Model

The consultant will work within the existing data engineering organisation under the team's established Agile delivery framework.

Delivery Structure

• Scoped work packages will be assigned to the consultant.
• Each consultant will be assigned to and managed by an existing member of the Data Engineering team.
• Consultants will operate within established governance, standards, and delivery processes.
• Task prioritisation and workload management will be coordinated through the assigned team lead.

Agile Participation

Consultants will actively participate in:

• Sprint Planning
• Backlog Refinement
• Daily Stand-Ups
• Sprint Reviews
• Sprint Retrospectives
• Other Agile delivery ceremonies as required