Senior Data Engineer (230009AA) (вакансія неактивна)

Ciklum is looking for a Senior Data Engineer to join our team full-time in Ukraine.

We are a leading global product engineering and digital services company that unites 4000+ seasoned professionals globally on various projects in healthcare, fintech, travel, sportswear, entertainment, and security. Ciklum delivers high-impact technology solutions to Fortune 500 and fast-growing organisations worldwide.

About the role:

As a Senior Data Engineer, become a part of a cross-functional development team working for our Client.

About the project:

This is a migration project from client’s data platform from Cloudera (On premises) to Databricks (Azure). Among the things to be migrated they have many PySpark scripts that read data from relational databases and save the processed data to Hive, Impala, Kudu and Solr.

There are also some NiFi workflows that read different files from many sources and persist them into the data lake, which is built on top of HDFS (inside Cloudera stack). These workflows need to be migrated as well.

There is a third processing engine in Python that also reads data from different sources and saves the results to the data lake (Hive, HDFS, KUDU).They will use ADLS gen2 as their data lake and the processing will be done in spark inside the Databricks stack.
Currently the job coordination is done using NiFi and Oozie, but the client wants to migrate to either Databricks workflows with Azure Data Factory or Airflow (hosted on Azure and offered as a service). This last point is still open.

The use cases are all related to analytics of payment data and many operational reports that the payment ecosystem and the different clients need.

Responsibilities:

• Contributes design, code, configurations, and documentation for components that manage data ingestion, batch processing, data extraction, transformation, and loading across multiple data storages
• Owns one or more key components of the infrastructure and works to continually improve it, identifying gaps and improving the platform’s quality, robustness, maintainability, and speed
• Interacts with engineering teams and ensures that solutions meet customer requirements in terms of functionality, performance, availability, scalability, and reliability
• Performs development, QA, and dev-ops roles as needed to ensure total end-to-end

Requirements:

• 2+ years of experience using Spark/PySpark (in the recent projects)
• 3+ years of experience working with SQL
• 3+ years of experience in Azure
• Experience with Rest services
• Experience with ORM / SqlAlchemy
• Knowledge of professional software engineering best practices for the full software
• Knowledge of Data Warehousing, design, implementation and optimization
• Knowledge of BI reports and dashboards design and implementation
• Knowledge of the development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations
• Experience participating in Scrum software development team
• A consistent track record of delivering exceptionally high-quality software on large, complex, cross-functional projects
• Demonstrated ability to learn new technologies quickly and independently
• Ability to handle multiple competing priorities in a fast-paced environment
• Undergraduate degree in Computer Science or Engineering from a top CS program is required. Masters preferred
• Experience with supporting data scientists and complex statistical use cases is highly desirable
• English on B2 level

Desirable:

• Experience with Solr
• Experience with Nifi
• Experience with Databricks
• Understanding of cloud infrastructure design and implementation
• Experience in data science and machine learning
• Experience in backend development and deployment
• Experience in CI/CD configuration

What`s in it for you?

• Care: your mental and physical health is our priority. We ensure comprehensive company-paid medical insurance, Mental Health programs, as well as financial and legal consultations
• Tailored education path: boost your skills and knowledge with our regular internal events (meetups, conferences, workshops), Udemy licence, language courses and company-paid certifications
• Growth environment: share your experience and level up your expertise with a community of skilled professionals, locally and globally
• Flexibility: choose how you want to work — a 100% remote or on-site in our offices. Own your schedule — you are the one to decide when to start your working day. Just don’t miss your regular team stand-up
• Opportunities: we value our specialists and always find the best options for them. Our Internal Mobility Program helps change a project if needed to help you grow, excel professionally and fulfil your potential
• Global impact: work on large-scale projects that redefine industries with international and fast-growing clients
• Welcoming environment: feel empowered with a friendly team, open-door policy, informal atmosphere within the company and regular team-building events

About us:

Discover the pros of 20-year global experience! Join one of the largest IT companies in Ukraine, awarded as one of the top 10 employers in Ukraine by Forbes.

Boost your skills and make a difference with cutting-edge projects, skilled colleagues and the latest tech stacks.

Ciklum has been recognised as a veteran-friendly business by the Veteran Hub in Ukraine.
We're committed to creating a welcoming environment for veterans and developing processes that meet their needs. We value the skills and unique perspectives military veterans can bring to our business and are willing to support their transition to civilian life.

Be bold, not bored!

Interested already? We would love to get to know you! Submit your application. We can’t wait to see you at Ciklum.