Lead Data Engineer

About Acerta:

Acerta helps manufacturers optimize the quality, safety, and reliability of the products they make. Our machine-learning solutions use industrial data to deliver valuable insights that improve the entire product lifecycle resulting in greater efficiency and satisfied customers.

Job Summary:

We are seeking a talented and experienced Lead Data Engineer to join our growing team. The Lead Data Engineer will play a key role in designing, building, and maintaining our data infrastructure, ensuring scalability, reliability, and performance with a team of data engineers around them. This position offers an exciting opportunity to work with large-scale datasets, cutting-edge technologies, and a passionate team of data professionals.

Key Responsibilities:

• Architect and Develop Data Ingestion to Acerta Platform: Architect, develop, and maintain scalable data pipelines and ETL processes to ingest, process, and analyze large volumes of structured and unstructured data from diverse sources.

• Collaborate with Cross-Functional Teams: Collaborate with cross-functional teams to understand data requirements, identify opportunities for data-driven insights, and develop solutions to address business challenges.

• Optimize Data Pipelines: Optimize and tune data pipelines and database queries for performance, efficiency, and reliability.

• Implement Data Governance: Implement data governance and security best practices to ensure data integrity, privacy, and compliance with regulatory requirements.

• Support Machine Learning Initiatives: Work closely with data scientists to support their data needs, provide access to relevant datasets, and enable advanced analytics and machine learning initiatives.

• Stay Abreast of Industry Trends: Stay abreast of industry trends, emerging technologies, and best practices in data engineering and analytics, and apply them to improve our data infrastructure and processes.

Skills and Qualifications:

• Previous experience in ETL/ELT Architecture: Expertise in designing and optimizing ETL/ELT pipelines to handle various data formats (CSV, JSON, Parquet, etc.) and integrating data from multiple sources (e.g., APIs, cloud storage, client Databricks instances).
• Strong understanding of REST API principles, experience with high-volume API requests, and ability to optimize API calls and data ingestion strategies.
• Automation and Orchestration: Proficiency in using orchestration tools like Apache Airflow, or similar tools to automate and manage data workflows.
• Proficiency using Unity Catalog to establish a consistent security and governance layer, manage access control, and ensure regulatory compliance.
• Expertise in building efficient ETL/ELT workflows to enable scalable feature engineering.
• Previous experience working closely with ML/DS teams, understanding of data modeling required for various teams and products.
• Previous experience in performance testing and optimization (data load testing/performance tuning/monitoring) for various databases, and ETL pipelines.
• Technical Mentorship: Leadership in coaching junior engineers, setting best practices for coding standards, and fostering a collaborative environment for knowledge sharing and troubleshooting.

Technical stack:

1. Programming Languages: Python, JavaScript/Node.js, Spark
2. Big Data experience: Databricks
3. Data Orchestration and Automation: K8, Apache Airflow or similar.
4. APIs and Data Ingestion: Strong understanding of REST API principles, experience with high-volume API requests, and ability to optimize API calls and data ingestion strategies.
5. Large data workflow optimization: Optimizations of
6. Bonus - experience with Vector Databases.

Why Join Acerta?

• Competitive Salary: Competitive salary and performance-based incentives.

• Comprehensive Benefits: Comprehensive benefits package, including health insurance, and more.

• Flexible Work Hours: Flexible work hours and remote work options.

• Professional Growth: Opportunities for professional growth and career advancement.

• Collaborative Environment: Collaborative and inclusive work environment with a diverse team of professionals.

If you are passionate about data engineering and eager to contribute to Acerta s mission, we invite you to apply for this exciting opportunity. Apply now and be part of Acerta s mission to shape the future of automotive intelligence!

Acerta Analytics Solutions Inc. is an equal-opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.

www.acerta.ai | LinkedIn | Twitter