Spark/Scala Developer
Apply on
Spark/Scala Developer
Location: NYC, NY
Duration: 18 Months
Job Description
10+ years of Develop and maintain data pipelines using Apache Spark and Scala to process large volumes of data efficiently
Ability to write and optimize code in Scala for Spark applications.
Designing, creating, and maintaining Scala-based applications.
Create real-time data pipelines using Apache Spark
Build and maintain SQL databases using Spark SQL
Build and maintain a codebase for Apache Spark data processes
Develop distributed code using the Scala programming language
Flow Control in Scala, Functions in Scala, Data Structures in Scala
Build, deploy, and run Spark scripts
Transform structured data using SparkSQL, DataSets, and DataFrames
Optimize Spark jobs through partitioning, caching, and other techniques
Process continual streams of data with Spark Streaming
Should have understanding on Spark SQL, DataSets, Running Spark on Clusters, Scala Libraries
Experience with streaming data architectures and technologies for real-time and low-latency data processing.