Data Engineer/Architect (Databriks) (вакансія неактивна)

Sphere Partners with Clients to transform their organizations, embed technology and process into everything they do, and enable lasting competitive advantage. We combine global expertise and local insight to help people and companies turn their ambitious goals into reality. At Sphere we put people first and strive to be a changemaker by building a better future through innovation and technology. Sphere is helping a known multinational company to innovate and bring new platforms to market and is looking for a Data Engineer/Architect (Databriks) who will join our team.

Location: Remote
Type: Contract
Start Date: ASAP

Responsibilities include but are not limited to:

• Collaborating with data scientists, analysts, and other engineers to design and develop complex data ingestion pipelines using various data sources;
• Creating and managing Databricks jobs/workflows and managing access to sensitive data;
• Developing services that provide accurate and relevant information depending on user context using vast datasets and different LLMs, including OpenAI integration;
• Improving the quality of responses depending on user context and using LLM frameworks like LlamaIndex and Langchain;
• Writing clean, efficient, and well-documented code that can be easily maintained and extended by the team;
• Provide support for critical production systems as needed.

Requirements:

• Strong knowledge of data pipeline design and implementation, including data transformation techniques, data storage optimization, and data security best practices;
• Experience with Databricks (Databricks Notebooks, workflows, Unity Catalog);
• Advanced SQL skills and extensive experience with relational databases, including expertise in writing complex queries and familiarity with various database systems;
• Experience with AI tools like (Llama, ChatGTP, DBRX), Spark and Python;
• Databricks certification (preferred);
• Fluent English.

Sphere offers a competitive and rewarding salary and benefits package, as well as an intellectually and creatively stimulating work environment, flexibility, and unique international travel opportunities.