Data Engineer

Job Description

Role and Responsibilities:

We received approval to hire a Data Engineer. This is a contract based with an intention to extend once we have the budget visibility. They will be a part of a team with Tech Lead, Business Analyst and a Data Engineer. Working in the Software Engineering team, they will have the following responsibilities:

Technical:

• Operationalizing and maintaining data pipelines deployed to cultivate a business relationship that assists in meeting business demands
• Translating requirements into solutions
• Deploying ETL/ELT pipelines to achieve a high level of reliability, scalability and security
• Sourcing, transforming and delivering structured and unstructured assets for analytical use
• Optimizing performance of ETL pipelines and data sources
• Participating in solution build, delivery, support and troubleshooting
• Provisioning and managing Azure resources, including security and cost
• Monitoring pipelines. Improving reliability. Automating audit
• Defining and executing technical test plans, performing unit, performance and integration tests and automating regression tests

Organizational:

• Operating in an Agile development environment
• Communicating effectively with other engineers and various other partners such as product managers, data & business analysts, IT infrastructure specialists, security specialists, enterprise architects and members of senior management
• Leading change and communicating impacts to partners and fellow team members
• Identifying, defining and implementing opportunities for improving existing processes
• Exhibiting the ability to work on multiple projects simultaneously and ensuring timely delivery
• Participating in establishing sound data management practices for the team and the organization
• Helping shape and scale the future of our development and production infrastructure

Qualifications

Professional:

• Experience. 3-5 years solving sophisticated technical data projects using the Microsoft Azure data stack or equivalent work experience. Financial industry application is preferred.
• A bachelor s degree in a scientific or engineering field of study or equivalent work experience
• Microsoft Data engineer associate level certificate will be considered as an asset
• Industry Knowledge of best practices. Previous experience in developing and deploying operational procedures, tuning guides and best practices documentation
• Relationship Building. Consistent track record of building deep technical relationships. Experience in aligning expectations across various partners
• Problem-Solving. The ability to trace data lineage / workflows and resolve technical issues with minimal direction
• Teamwork. Motivated and keen to work in a collaborative environment with a focus on team success

Preferred Skills:

• Enterprise-scale technical experience with cloud and hybrid infrastructures, architecture designs, and technology management
• Experience in handling data in Python using libraries like Pandas, Numpy
• Proficient with SQL
• Experience with data architecture, modeling, data warehousing
• Breadth of technical experience and knowledge across the Azure Data Platform, with depth / Subject Matter Expertise in two or more of the following Data Platform resources:

1. Azure Databricks, experience with Python
2. Azure Functions and/or Web Apps using C#.Net
3. Azure Managed SQL Server
4. Azure Data Factory

• Proven experience automating CI/CD pipelines using Azure DevOps or equivalent work experience
• Experience with containerization and container deployment to Kubernetes
• Experience developing star schema reporting pipelines using Kimball or Data Vault v2.0 methodologies