Cloud Data Engineer
Salary undisclosed
Apply on
Original
Simplified
Job Description
Job Description
Job Description: Cloud Data Engineer
Role: Cloud Data Engineer
Experience Level: 8+ years
Location: Montreal, QC or Mississauga, ON (Local candidates only)
Priority Level: High
Overview:
We are seeking an experienced Cloud Data Engineer with a strong background in Python, Pyspark, SQL, and data warehousing for enterprise-level systems. The ideal candidate is proficient in working with business users and possesses a combination of technical expertise and business analysis skills.
- Data Pipeline Development: Build and optimize data pipelines for efficient ingestion, transformation, and loading from various data sources, ensuring data quality and integrity.
- Spark Development: Design, develop, and deploy Spark programs in Databricks to process and analyze large data volumes.
- Data Integration & Modeling: Leverage Delta Lake, Data Warehouse (DWH), and other data integration techniques, with expertise in data design and dimensional modeling.
- Python & SQL Proficiency: Develop programs using Python and SQL, particularly for large datasets.
- Data Management: Handle structured, semi-structured, and unstructured data, and work with event-based/streaming technologies.
- Optimization: Optimize Databricks jobs for performance and scalability to handle big data workloads.
- Monitoring & Troubleshooting: Monitor and troubleshoot Databricks jobs, resolving issues or bottlenecks as they arise.
- Data Management & Governance: Implement best practices for data management, security, and governance within the Databricks environment.
- SQL Development: Write SQL queries, including stored procedures, and perform reverse engineering on existing processes.
- Code Review: Conduct code reviews to ensure alignment with requirements, optimal performance, and adherence to standards.
- Technical Proficiency:
- Python: 5+ years of coding experience.
- SQL: 5+ years of SQL Server development experience with large datasets.
- ETL Pipeline Development: 5+ years in Databricks Pyspark ETL pipeline development and deployment.
- Data Warehousing: Expertise in cloud-based data warehousing solutions (e.g., Synapse, BigQuery, Redshift, Snowflake) and data modeling concepts (OLTP, OLAP, Dimensions, Facts).
- Architectural Experience: Prior experience leading an enterprise-wide Cloud Data Platform migration, with strong skills in architecture and design.
- Cloud Architectures: Familiarity with cloud-based data architectures, messaging, and analytics frameworks.
- Additional Knowledge: Experience with Apache Airflow is a plus.
- Bachelor's Degree in Engineering, Computer Science, or related field (minimum requirement).
- Master's Degree is strongly preferred.
Cloud Certifications: Relevant cloud certifications are highly valued for this role.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
Report this job Similar Jobs