Big Data Engineer

Title:Big Data Engineer
Type:Contract 18 months
Location:Rockville, MD (Hybrid 2 days a week) Experience required :: 8+

Job Requirements:

• Extensive Experience with cloud based Big Data technologies like Hadoop, Hive, Spark, AWS EMR and Airflow.

• Experience in one or more programming languages like Java, Scala, and Python.

• Experience in solving complex problems using SQL.

• Experience with developing enterprise grade solutions in an iterative or Agile environment.

• Clear, effective communication with strong interpersonal skills.

• Ability to push the frontier of technology and independently pursue better alternatives.

• Ability to maintain focus and develop proficiency in new skills rapidly.

• Ability to utilize problem solving skills in a fast-paced environment

• Experience in AWS Cloud is required.

Intake Notes:
Java, Scala, and Python - very different languages, where do we hone in or are they language

• PySpark (much easier, more bang for your buck) vs SparkScala

Big data processing vs streaming uses:

• Mainly for batch processes

• Lambdas that support the processes, but the modernization is on batch

Anything outside of AWS emr that they need: Hadoop

Team: last year, ETL project moved from to Market Regulation

• Need this person to be strong because they are operating in tight constraint on time and resources so this needs to be a producer and can't be affecting the productivity of the team

• 10-member team so there is support but once trained, expectation is they ramp up and can work indecently

Main technologies:
Scala, Python, SQL, AWS
PySpark is OK
Big Data
Hadoop, Hive, Spark, AWS EMR and Airflow.

Main responsibility is to understand existing process, learn the new design and make the code changes to implement that. Not a major rewrite but modernizing a part

• Have clear guidelines and process defined to approach the modernization

Hadoop has own, Cloudera - and Hadoop distribution
Data bricks - same as Hadoop, it's fine

Must Have Skills/ Requirements:

1. SQL

1. Prep data

Excel
This is how they communicate with users, so if they are strong in SQL, Excel can copy paste
Python - OK if they don't have

Nice to Have Skills/ Requirements:
Market knowledge is a plus to help with learning curve
Know the behaviors of the market Project:

• ML algorithms

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Title:Big Data Engineer
Type:Contract 18 months
Location:Rockville, MD (Hybrid 2 days a week) Experience required :: 8+

Job Requirements:

• Extensive Experience with cloud based Big Data technologies like Hadoop, Hive, Spark, AWS EMR and Airflow.

• Experience in one or more programming languages like Java, Scala, and Python.

• Experience in solving complex problems using SQL.

• Experience with developing enterprise grade solutions in an iterative or Agile environment.

• Clear, effective communication with strong interpersonal skills.

• Ability to push the frontier of technology and independently pursue better alternatives.

• Ability to maintain focus and develop proficiency in new skills rapidly.

• Ability to utilize problem solving skills in a fast-paced environment

• Experience in AWS Cloud is required.

Intake Notes:
Java, Scala, and Python - very different languages, where do we hone in or are they language

• PySpark (much easier, more bang for your buck) vs SparkScala

Big data processing vs streaming uses:

• Mainly for batch processes

• Lambdas that support the processes, but the modernization is on batch

Anything outside of AWS emr that they need: Hadoop

Team: last year, ETL project moved from to Market Regulation

• Need this person to be strong because they are operating in tight constraint on time and resources so this needs to be a producer and can't be affecting the productivity of the team

• 10-member team so there is support but once trained, expectation is they ramp up and can work indecently

Main technologies:
Scala, Python, SQL, AWS
PySpark is OK
Big Data
Hadoop, Hive, Spark, AWS EMR and Airflow.

Main responsibility is to understand existing process, learn the new design and make the code changes to implement that. Not a major rewrite but modernizing a part

• Have clear guidelines and process defined to approach the modernization

Hadoop has own, Cloudera - and Hadoop distribution
Data bricks - same as Hadoop, it's fine

Must Have Skills/ Requirements:

1. SQL

1. Prep data

Excel
This is how they communicate with users, so if they are strong in SQL, Excel can copy paste
Python - OK if they don't have

Nice to Have Skills/ Requirements:
Market knowledge is a plus to help with learning curve
Know the behaviors of the market Project:

• ML algorithms

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.