Epicareer Might not Working Properly
Learn More
V

Data Engineer

Salary undisclosed

Apply on


Original
Simplified

Position: Data Engineer

Location: Remote

Duration: Long term contract

Skills that Client wants to see in the resume:
Databricks, Python, Pyspark, Azure, AWS & Terraform Experience

Job Summary:

We are seeking a highly skilled and experienced Data Engineer with expertise in Databricks, Python, PySpark, and cloud platforms such as Azure and AWS. The ideal candidate will also have experience in infrastructure-as-code using Terraform. As a Data Engineer, you will play a crucial role in designing, implementing, and optimizing our data pipelines and infrastructure to support our data-driven decision-making processes.

Key Responsibilities:

  • Design, build, and maintain scalable data pipelines using Databricks, Python, and PySpark to process and analyze large volumes of data.
  • Collaborate with data scientists, analysts, and business teams to understand data requirements and implement solutions.
  • Optimize and troubleshoot data pipelines for performance, scalability, and reliability.
  • Deploy and manage cloud infrastructure on both Azure and AWS for data processing and storage.
  • Implement infrastructure-as-code using Terraform to automate the provisioning and management of cloud resources.
  • Ensure data security and compliance with industry regulations and internal policies.
  • Monitor, maintain, and improve ETL processes, ensuring high availability of data pipelines.
  • Collaborate with DevOps teams to implement CI/CD for data infrastructure.
  • Develop and maintain data models and ensure efficient data storage strategies.
  • Participate in code reviews, performance tuning, and mentoring of junior engineers.

Required Qualifications:

  • Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent experience).
  • 10+ years of experience as a Data Engineer or in a similar role.
  • Proficiency in Databricks, Python, and PySpark for data processing and analytics.
  • Strong experience with Azure and AWS cloud platforms, including storage, compute, and networking services.
  • Experience in infrastructure-as-code tools, particularly Terraform, to automate cloud deployments.
  • Strong SQL skills and familiarity with relational and non-relational databases.
  • Experience in ETL development and data pipeline orchestration.
  • Familiarity with version control systems (e.g., Git) and CI/CD pipelines.
  • Experience with containerization technologies (Docker, Kubernetes) is a plus.
  • Good communication skills and ability to work collaboratively in a cross-functional team.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
Report this job