Vacancy level: Middle
Team composition: 2 Big Data Engineers from our side, another part of the Big Data team will be on the customer side.
Technology stack: Scala, Spark, Kafka.
Our customer — an American multinational company serving the combined industries of health information technologies and clinical research. Their solutions are used by several top-20 global pharmaceutical companies such as AstraZeneca, Novo Nordisk and Roche, as well as Adhera Therapeutics, Napo Pharmaceuticals, Pierre Fabre Pharmaceuticals, PruGen Pharmaceuticals, Recordati, Theramex, Zentiva, etc.
Would you like to join the development of a Big Data solution that is changing the pharmaceutical industry worldwide? Together with our client from the USA, we started realizing a project to collect and aggregate pharmaceutical data in a Data Lake environment. By becoming part of our team, you will develop a solution that helps rethink the approach to data analysis in the pharmaceutical field.
What you bring along:
- At least 2-3 years of relevant experience as a Scala/Big Data Engineer;
- Good knowledge and hands-on Scala programming;
- Proficiency in Spark Scala for technical development and implementation;
- Experience with AWS components and services, particularly EMR, S3, and Lambda;
- Data pipeline using Spark Scala;
- Load disparate data sets by leveraging Kafka consumers;
- Good understanding of Big Data technologies like Hadoop, Spark, Scala, Hive, HBase, Pig, Cascading;
- Excellent in writing shell scripts;
- Knowledge of Cassandra Architecture and CQL is desirable;
- Ability to utilize Hive, Spark, Cassandra, Mesos and Kafka;
- Good knowledge of Hive and HiveQL;
- Experience with messaging and complex event-processing systems such as Kafka;
- Hands-on in Hadoop stack (MapReduce, Hive, Sqoop, Oozie);
- Good understanding of file formats including JSON, Parquet, Avro, and others;
- Extensive experience with Data frame and Dataset operations of Spark;
- Proficient in AWS technologies or equivalent cloud stack as a hands-on engineer to influence all architecture decisions;
- Experience with messaging and complex event-processing systems such as Kafka and Storm;
- English — Upper-Intermediate or higher.
Nice to have:
- Experience with application architecture in a big data environment;
- Build reusable code, with the ability to scale with very large data volumes;
- Data Migration/ETL knowledge.
This is your mission:
- Program, test, and document systems in accordance with programming standards and validation procedures;
- Plan and coordinate the complex design, development, implementation, maintenance, and level 4 support of associated systems;
- Provide software, system and programming solutions across projects and technologies to meet internal and external needs;
- Engineer systems and/or software, and/or transfer data for internal and external clients;
- Plan and coordinate peer review of work products such as code, designs, and test plans produced by other team members;
- Interact with corporate and project team members to negotiate timelines, responsibilities, and deliverables;
- Provide expert technical advice and solutions for internal and external clients;
- Provide an expert review of database development work with regard to programming standards and validation procedures;
- Provide long term evaluations of systems, company wide;
- Meet personal and professional milestones as assigned and perform other duties upon request.
- Flexible remote policy;
- Reimbursement of education and certification;
- Medical and sport services compensation;
- Paid out of reach, sick-days and public holidays;
- Language school on site;
- Accounting and legal support.