Big Data Production Support

Big Data Production Support

Location- Iselin, NJ (Hybrid)
Long term Contract

Job Description:

• The position will form part of Data Engineering team working with global technology team, largely based in the US.
• The candidate will be expected to provide production support for Data Engineering team.
• Candidate will coordinate with various teams like Infrastructure, Network Security, Support teams across Market Surveillance technology and will be responsible for deployment of code, automation, monitoring batch jobs and ensure availability of application.
• Scope will include support, deployment, automation and migration to strategic platforms like cloud and well versed with ITIL standards like Incident, Change and Problem Management.
• The candidate will be required to ensure that the project complies with all company policies regarding the management of third-party solutions, Information Security and the Software Development Lifecycle.
• In this contingent resource assignment, you may:
• Consult on or participate in moderately complex initiatives and deliverables within Software Engineering and contribute to large-scale planning related to Software Engineering deliverables.
• Review and analyze moderately complex Software Engineering challenges that require an in-depth evaluation of variable factors.
• Contribute to the resolution of moderately complex issues and consult with others to meet Software Engineering deliverables while leveraging solid understanding of the function, policies, procedures, and compliance requirements.
• Collaborate with client personnel in Software Engineering.

Required Qualifications:

• 4 plus years of Software Engineering experience, or equivalent demonstrated through one or a combination of the following: work or consulting experience, training, military experience, education.
• 5 plus years of experience and deep understanding of Software Development Life cycle, data ingestion, storage, querying, processing and analysis.
• Expertise on scheduling tools like Autosys, Yoda and other standard industry scheduling tools
• Strong Unix/Linux skills and experience in writing shell scripts
• Oracle & SQL DB expertise and strong SQL skills and writing complex queries,
• Expertise in Data load management, importing & exporting data using SQL, ETL and Cloud Platforms.
• Experience with strategy, planning and execution of new infrastructure setup and Data Center/Cloud Migration
• Experience in configuring CS/CD pipelines, configure manage Kubernetes
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs/Data frames, Scala.
• Experience developing, installing, and configuring Hadoop ecosystem components.
• Good knowledge on ETL procedures to transform the data in the intermediate tables according to the business rules and functionality requirements.

Intake:

• Supporting trade and market surveillance team.
• Will need to have Advanced SQL," meaning that they are a true expert in SQL queries and are able to find the gaps.
• Will do root cause analysis on production job failures, will run queries, find issues, and fix the issues.
• Looking to improve upon the current members in this position who rely on others to write the queries for them.
• Hadoop is where the data is stored, Hive is used to query it, and Spark is the language used for programming.
• Needs to be able to communicate so that they can coordinate with other teams to identify problems and work on solutions.
• Agile is good but not required.
• Finance industry experience is highly preferred.
• 3 plus years of production application support using technologies Advanced SQL, Autosys, Big Data (Spark, Hadoop, Hive, Python, Scala)
• Advanced SQL and Stored Procs expertise is MUST

Advanced capabilities' with:

• SQL
• Hadoop
• Hive
• Spark

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.