Our customer in the Energy sector is looking for a Senior Data Engineer.
Job scope:
Senior Data Engineer will be responsible for understanding the source of data and developing well-designed, testable and efficient data ingestion and data quality pipelines using software development best practices.
The Senior Data Engineer will have in charge supporting our core data services to serve data science, analytics and corporate reporting
Primary Responsibilities:
- Contribute to the strategy for the Azure based data platform to support the business while optimizing performance and minimizing cost.
- Design and build the processes required for optimal extraction in batch or streaming, transformation, and loading of data using a variety of languages and technologies such Python, Spark, SSIS, Azure Data Factory or Azure Synapse Analytics and Databricks.
- Propagate architectural direction to various teams and work streams, balancing the strategic approach within the agile methodology.
- Partner with stakeholders and project teams to deliver pipelines in an iterative and incremental manner, fostering an environment of learning and collaboration.
- Perform unit tests and conduct reviews with other team members to make sure code is rigorously designed, elegantly coded, and effectively tuned for performance
- Lead root cause analysis, prioritize and manage data quality and remediation, and ensure data integrity to all downstream data systems.
- Continuously work to improve the integrity of solutions.
Profile:
Requirements:
- Good experience with bringing data from variety of sources (Databases, APIs, Files, Applications) into an Azure based centralized data lake or manipulating the available data to build additional data sets for integration, analytics and reporting purposes.
- Mastery of T-SQL and experience other forms of SQL
- Experience of ETL technical design, automated data quality testing, QA and documentation, data warehousing, data modelling and data wrangling
- Experienced with maintaining data quality throughout the lifecycle of the data.
- Expertise in Azure DevOps automation tools like Pipelines and experience in deploying applications to at least one of the major public cloud provider such Azure, AWS, GCP
- Excellent oral and written communication skills in English
- Deadline-driven mentality
- Team player and process-oriented spirit
Nice to have:
- Technical delivery background on Business Intelligence and machine learning constructions
- IT Infrastructures project experience
- ITIL v3 processes certification or seasoned experience
Qualities to be successful:
- Are open to suggestions, collaborative, and thrive in team environments.
- You absolutely love providing support and you are passionate about helping users
- You are optimistic, curious, resourceful and inspired to solve user issues
- You have a flair for building relationships and ability to influence others
- You are empathetic towards understanding what a user needs and why they need it
Work experience: +5 years
Minimum education level: Master's degree Computer Science, Telecommunication or scientific related field
