Data Engineer For Malaysia

Kamlax Global, Malaysia is hiring for Data Engineer

Work Location: KL,Malaysia

Visa will be provided.

Responsibilities·

Design, develop, document and implement end-to-end data pipelines and data integration processes, both batch and real-time. This include data analysis, data profiling, data cleansing, data lineage, data mapping, data transformation, developing ETL / ELT jobs and workflows, and deployment of data solutions.

Monitor, recommend, develop and implement ways to improve data quality including reliability, efficiency and cleanliness, and to optimize and fine-tune ETL / ELT processes.

Recommend, execute and deliver best practices in data management and data lifecycle processes, including modular development of ETL / ELT processes, coding and configuration standards, error handling and notification standards, auditing standards, and data archival standards.

Prepare test data, assist to create and execute test plans, test cases and test scripts.

Collaborate with Data Architect, Data Modeler, IT team members, SMEs, vendors and internal business stakeholders, to understand data needs, gather requirements and implement data solutions to deliver business goals.

BAU support for any data issues and change requests, document all investigations, findings, recommendations and resolutions.

Qualifications

Bachelor in IT, Computer Science or Engineering.

At least 3+ years solid and hands-on experience in real-time event/data streaming experience using Kafka/Spark/Spark Stream/Flink ecological environment development, proficient in basic knowledge and data development technology related to data processing, relevant experience in real-time data processing in large volume environment. Have experience with Kafka and Confluent Platform is highly preferred.

At least 3+ years of solid hands-on development experience with ETL development to transform complex data structure in multiple data sources environment.

Strong programming on Python and T-SQL ETL/ELT programming.

Experience on Azure Databricks for ETL/ELT development and big data analytics programming in Python

Strong Experience with various of ETL/ELT frameworks, data warehousing concepts, data management framework and data lifecycle processes.

Solid understanding on Azure Data Management Solution including Azure Data Factory, Azure Databricks, Azure Blob storage (Gen2) and Azure Synapse