Our team is building a high load Data Platform with over 300 GB/day real-time data and up to 40 GB/day of new aggregated data, which also includes next directions: Integration with third-party services/solutions; Supporting for operations data-centric services and Data validation and storage.
— AWS (S3 / Athena / Kinesis / Glue/ RedShift / SageMaker);
— Docker and Kubernetes;
— Gitlab pipelines / Flux / Helm;
— ML Flow;
— Artifacts builds by: Maven, Python tools;
— Kafka Confluent stack, ELK, Grafana, Prometheus;
— Design and maintain High-load and High-availability infrastructure;
— Analyze service infrastructure needs and their justification;
— Support integration with AWS/Google cloud solutions;
— Creating and support pipelines for Continuous delivery/deployment in teams of the full development cycle;
— Performance monitoring;
— Performance tuning via auto-scaling;
— Write and update automation scripts;
— Dockerizing Applications;
— Container-orchestration system: Kubernetes.
— Good knowledge of Unix systems;
— Experience with AWS/Google cloud providers;
— Writing pipelines for Continuous delivery/deployment in teams of the full development cycle;
— Working with large amounts of data (Databases, storages, Hadoop like projects);
— Good knowledge of k8s maintenance
— Experience with large databases Maintenance (Postgresql/etc)
— Good practice in Monitoring Tools (Prometheus/etc)
— Experience with Ansible and Terraform as with automation tools.
— Good knowledge of virtualisation systems (Vmware / Hyper-V / KVM);
— Working experience with hardware (servers, storages, fc hardware).