Data Engineer

DCG
  • Post Date: October 6, 2024
    • 12876
  • Applications 0
  • Views 8
Job Overview

As a recruitment company, DCG understands that every business is powered by experienced professionals. Our management style and partnership approach enable us to meet your needs and provide continuous support. Due to our ongoing growth and the large number of recruitment projects we undertake for our partners, we are currently looking for Data Engineer.
Responsibilities:Design, manage, and optimize ETL processes using Azure Data Factory for batch data processing.Use Databricks and PySpark to process and analyze large datasets across distributed environments.Create automation workflows and perform complex data manipulations using Python.Write and optimize SQL queries to efficiently retrieve and process data from various sources.Implement real-time data ingestion solutions using Azure Event Hub and Spark Streaming.Manage and interact with NoSQL databases, particularly CosmosDB, to support scalable data storage and retrieval.Prioritize and enhance batch data processing workflows, ensuring high performance and reliability.Collaborate with cross-functional teams, provide insights, and maintain thorough documentation of all data processes.
Requirements:Minimum 3 years of experience in similar positionEnglish level: B2, C1Strong experience with Azure Data Factory for orchestrating ETL processes.Proficiency in Databricks and PySpark for distributed data processing.Advanced skills in Python for data manipulation and automation.Solid knowledge of SQL for writing complex queries and optimizing data retrieval.Experience with Azure Event Hub for real-time data ingestion.Knowledge of CosmosDB for working with NoSQL databases.Understanding of Spark Streaming for processing real-time data streams.Batch processing experience is more critical than streaming processing for this role.Familiarity with Azure Synapse is a plus, but not mandatory.Good to Have: Knowledge of Azure DevOps for CI/CD pipeline automation, experience in the Aviation Industry, and familiarity with Copilot as a tool for AI-driven development assistance.
What we offer:100% remote jobHourly rate up to 130zł/hPrivate medical careCo-financing for the sport cardTraining & learning opportunitiesConstant support of dedicated consultantTeam-building events organized by DCGEmployee referral programLong-term cooperation

Job Detail
Shortlist Never pay anyone for job application test or interview.

Related Jobs (5079)

Safety Information

  • Safety Tips For Candidate

    DYOR.

  • Safety Tips For Candidate

    Always check the employer\'s offer.