DATA ENGINEER
Salary undisclosed
Apply on
Availability Status
This job is expected to be in high demand and may close soon. We’ll remove this job ad once it's closed.
Original
Simplified
Role: Data Engineer
Duration: 6-month contract to hire
Location: Remote
Visa: W2 role
Required Skills:
1. ClickHouse- 2+ Years 2. Kubernetes - understanding infra. 3. AWS
Job Description:
Data Pipeline and Storage: Detailed the process of collecting logs from various sources, structuring them through a pipeline using vectored dev for rate limiting and security, and storing them in ClickHouse as the database.
Querying and Visualization: They plan to offer Grafana for standard querying but will support other tools like Databricks, Power BI, and Excel. An open-source tool called Keep will serve as the middle layer for event management and rules engine.
Event Management: Keep will query the lake for triggering events in ServiceNow, acting as an intermediary due to ServiceNow's inability to handle millions of rules directly.
Scale and Talent Needs: The platform aims to handle 100 petabytes a month initially, requiring sharp talent, especially in ClickHouse and vectored dev. They are seeking vendor support for architecture and operational expertise.
Challenges and Solutions: The discussion also covered the challenges of scaling, the need for a multi-tenant solution, and the importance of optimizing SQL queries for efficiency. The goal is to consolidate various data sources into a single, queryable database to facilitate correlation and analysis.
--
Thanks & Regards,
Kalyan Sadanala| Talent acquisition specialist
Thoughtwave Software and Solutions
314 N. Lake St, Suite 6, Aurora IL 60506
Desk: , EXTN: 148
Text Message:
Email:
Website:
Linked in:
Duration: 6-month contract to hire
Location: Remote
Visa: W2 role
Required Skills:
1. ClickHouse- 2+ Years 2. Kubernetes - understanding infra. 3. AWS
Job Description:
Data Pipeline and Storage: Detailed the process of collecting logs from various sources, structuring them through a pipeline using vectored dev for rate limiting and security, and storing them in ClickHouse as the database.
Querying and Visualization: They plan to offer Grafana for standard querying but will support other tools like Databricks, Power BI, and Excel. An open-source tool called Keep will serve as the middle layer for event management and rules engine.
Event Management: Keep will query the lake for triggering events in ServiceNow, acting as an intermediary due to ServiceNow's inability to handle millions of rules directly.
Scale and Talent Needs: The platform aims to handle 100 petabytes a month initially, requiring sharp talent, especially in ClickHouse and vectored dev. They are seeking vendor support for architecture and operational expertise.
Challenges and Solutions: The discussion also covered the challenges of scaling, the need for a multi-tenant solution, and the importance of optimizing SQL queries for efficiency. The goal is to consolidate various data sources into a single, queryable database to facilitate correlation and analysis.
--
Thanks & Regards,
Kalyan Sadanala| Talent acquisition specialist
Thoughtwave Software and Solutions
314 N. Lake St, Suite 6, Aurora IL 60506
Desk: , EXTN: 148
Text Message:
Email:
Website:
Linked in:
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
Report this job Similar Jobs
