We are looking for an experienced Data Engineer to design, develop, and maintain reliable, scalable, and secure data pipelines supporting analytics, reporting, and AI initiatives. The role involves building production-grade data solutions on Azure and Microsoft Fabric, implementing data transformations, optimizing pipeline performance, and ensuring data quality, metadata management, and governance across the data lifecycle.
The ideal candidate will have strong expertise in SQL, Python, PySpark, and modern cloud data platforms, along with hands-on experience in Lakehouse and Medallion architecture.
Key Responsibilities
- Design and develop batch, incremental, CDC, and streaming data ingestion pipelines using Azure Data Factory and Microsoft Fabric.
- Build Silver- and Gold-layer data transformations using SQL, PySpark, and Spark notebooks.
- Implement dimensional data models, data marts, surrogate keys, and Slowly Changing Dimensions (SCDs).
- Integrate data validation, quality checks, reconciliation, and exception-handling mechanisms into data pipelines.
- Maintain metadata, data lineage, and documentation using Microsoft Purview.
- Implement pipeline orchestration, scheduling, dependency management, error handling, and alerting.
- Optimize data pipelines, partitioning, Delta/Parquet formats, and compute utilization for performance and cost efficiency.
- Apply security controls, encryption, data masking, and secure credential management in accordance with applicable data protection requirements.
- Maintain source control, automated testing, CI/CD pipelines, and DataOps practices.
- Support production operations, troubleshoot incidents, maintain runbooks, and facilitate knowledge transfer.
Required Skills and Qualifications
- 4+ years of experience in Data Engineering or a related field.
- Strong proficiency in SQL, Python, and PySpark/Spark.
- Hands-on experience with Azure Data Factory, Microsoft Fabric, Databricks, or Azure Synapse Analytics.
- Strong understanding of Lakehouse/Medallion architecture and Delta/Parquet data formats.
- Experience with dimensional modelling, surrogate keys, and SCD implementation.
- Knowledge of Azure DevOps or GitHub, CI/CD, source control, and DataOps practices.
- Familiarity with Microsoft Purview, metadata management, data lineage, and data quality tools.
- Awareness of UAE data protection and regulatory requirements, including UAE PDPL, the Dubai Data Law, and DESC ISR.
- Strong communication, documentation, troubleshooting, and collaboration skills.
Preferred: Experience working with UAE government organizations or regulated-sector environments.
To apply for this job email your details to anup@praxisconsultants.in