Epicareer Might not Working Properly
Learn More

Data Ingestion Engineer

Salary undisclosed

Apply on

Availability Status

This job is expected to be in high demand and may close soon. We’ll remove this job ad once it's closed.


Original
Simplified

Data Ingestion Engineer

Location: Remote

Duration: 12+ Months Contract

Role Overview:

This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of:

  • Data Lake ingestion processes and best practices
  • ETL/ELT implementation
  • CI/CD
  • System integration tools
  • Data pipeline management

About the role:

  • Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis
  • Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform
  • Optimize data pipelines for scalability and speed
  • Document ingestion and integration flows and pipelines
  • Use Airflow to schedule and automate ingestion jobs
  • Manage metadata and master data in technical data catalog
  • Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion
  • Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies
  • Ensure adherence to SDLC best practices

About you:

  • Bring 8+ years of experience in data engineering, ingestion pipelining, and ETL/ELT
  • Hold a bachelor s degree in computer science, engineering, statistics, or related field
  • Have hands-on experience with and understanding of the following:
    • Spark/Scala
    • SQL
    • Python/PySpark or similar programming language
    • Databricks
    • Unity Catalog
    • ETL/ELT development, monitoring and pipelining using tools such as Apache Airflow
    • Ingestion tools such as Dell Boomi
    • Data quality guidelines
    • CI/CD pipelines
    • Agile
    • Git and version control
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
Report this job