Data Engineer, Online Marketplace. Sign-on bonus: $1000 (вакансія неактивна)

Hands-on experience in data engineering (e.g., working with large volumes of data, cloud environments, parallelization)
Knowledge of AWS technologies (e.g., S3, Aurora)
Akamai configuration would be a huge plus
Spoken English, good communication and coordination skills

We offer a $1000 sign-on bonus to the specialist that will accept our offer and join DataArt in this position. $1000 will be included in the first salary.

OTHER OPPORTUNITIES:

• Professional Development:
— Experienced colleagues who are ready to share knowledge;
— The ability to switch projects, technology stacks, try yourself in different roles;
— More than 150 workplaces for advanced training;
— Study and practice of English: courses and communication with colleagues and clients from different countries;
— Support of speakers who make presentations at conferences and meetings of technology communities.
• The ability to focus on your work: a lack of bureaucracy and micromanagement, and convenient corporate services;
• Friendly atmosphere, concern for the comfort of specialists;
• Flexible schedule (there are core mandatory hours), the ability to work remotely upon agreement with colleagues;
• The ability to work in any of our development centers.

R&D and solution design phase
Design and implementation of a data migration pipeline
Integration with the existing data platform

Our client is an EU-based marketplace. They developed a website and app that allows users to buy and sell wardrobe items or gadgets.

For the first stage, the project is aimed at re-thinking and redesigning the data store. The client needs to migrate a large image database, apply new technology for the data governance system, and to perform a clean-up of useless data. During the next stage it is expected to conduct some R&D and migrate to another data warehouse. We expect long collaboration with the client as the project is at an early stage of development and new challenges and tasks might appear.

The data engineer will work on the migration of all images to another storage from the current S3 bucket, which takes up around 400 TB of data as well as applying governance over the data, switching from the Amazon internal messaging system to Kafka. The task of restructuring the client’s database also provides an opportunity to learn Akamai in practice as it needs to be reconfigured for serving data from the new bucket.

Tech stack and infrastructure: no hard requirements about language used but Scala is preferable as most of engineers work with that language. AWS technologies — data stored in a S3 bucket, Akamai serves data from a bucket.