Data Engineer

Job Title: Data Engineer Location: Remote
Job Type: Full-time
Pay Rate: Up to $60/hr

Job Responsibilities:

• Design, build, and maintain efficient data pipelines to collect, process, and store data from various sources, including user interactions, financial details, and external data feeds.
• Develop data models that facilitate data analysis and ensure quality, consistency, and accuracy.
• Build scalable data pipelines using SparkSQL and Scala, utilizing the Airflow scheduler/executor framework.
• Collaborate with Data Scientists, Product Managers, and Software Engineers to define data requirements and deliver solutions that enhance merchandising and sales.
• Contribute to the Data Engineering community by influencing tooling and standards for improved productivity.
• Improve code and data quality by leveraging internal tools for detecting and resolving issues.

Qualifications:

• 5-9+ years of relevant industry experience with a BS/Masters, or 2+ years with a PhD
• Expertise in distributed processing technologies and frameworks (Hadoop, Spark, Kafka)
• Experience with distributed storage systems (HDFS, S3)
• Proficiency in analyzing large datasets to identify gaps and provide data insights
• Hands-on experience with ETL schedulers such as Apache Airflow, Luigi, Oozie, AWS Glue
• Strong knowledge of data warehousing concepts and relational databases (PostgreSQL, MySQL)
• Experience with columnar databases (Redshift, BigQuery, HBase, ClickHouse)
• Excellent written and verbal communication skills

Key Skills:

• Python (Expert)
• SQL (Expert)
• Spark (Intermediate)
• Scala (Intermediate)

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.