Senior Data Engineer

HiringWise, a recruitment agency, is seeking a Senior Data Engineer to join the development team of a software product company renowned as the world’s premier community for engaged investors. The company serves as the ultimate hub for investors seeking actionable stock market insights, timely market analysis, and distinctive financial perspectives.

In this role, you will be responsible for design, build, and maintain the infrastructure required to analyze large data sets. As an expert in data management, ETL (extract, transform, load) processes, and data warehousing, you will work with various big data technologies such as Spark, Hadoop, and NoSQL databases. You will work closely with data and analytics team to identify and prioritize data engineering projects, ensuring the data infrastructure aligns with overall business goals and objectives.

What You’ll Do:

• Work with business analysts and stakeholders to identify and prioritize data engineering projects, regularly evaluate the data infrastructure, and make necessary changes to meet evolving business needs
• Help drive data-driven decision-making to improve user engagement and increase subscriptions
• Handle data engineering tasks in a changing environment, ensuring solutions meet modern standards, remain reusable and maintainable, and are sufficiently documented
• Design, build, and maintain data pipelines that efficiently and cost-effectively handle data from multiple sources, such as external APIs, streams, and data stores
• Optimize SQL queries to manage resource consumption and costs, applying performance tuning across various SQL dialects to ensure efficient and consistent results
• Monitor and improve data infrastructure performance and reliability, addressing scalability, efficiency, and security in line with our engineering standards
• Keep up with data engineering trends and explore new ideas to help the team improve and innovate
• Use data privacy and security measures as needed to meet relevant standards and frameworks

Requirements:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field
• At least 5 years of professional experience in data-focused roles (Data Engineer, Database Developer, BI Developer/Engineer, or DBA), covering object-oriented, functional, and procedural languages
• Strong, hands-on proficiency with Python (including data-oriented frameworks and specific libraries such as Pandas, AWS SDK)
• Strong, hands-on proficiency with SQL across various RDBMS and big-data engines (e.g., Hive, Athena, Spark, Presto, Dremio)
• Solid experience with AWS services (EC2, S3, RDS, EMR, Athena, Glue, Lambda, ECS/EKS)
• Deep expertise in Apache Airflow for data pipeline orchestration
• Advanced understanding of data warehousing, spanning both classic architectures and modern solutions such as Snowflake, Databricks, Redshift, Google BigQuery, Azure Synapse Analytics, ClickHouse, or IBM Db2 Warehouse / IBM Netezza
• Demonstrated active problem-solving ability, backed by a strong personal software engineering culture
• Comfortable in team settings and able to work effectively both collaboratively and independently
• High-level English — a must

Nice To Have:

• Expertise with AWS Athena
• Expertise with AWS Redshift or Vertica Analytics Database
• Expertise with Spark APIs (Scala or PySpark)
• Experience with machine learning and / or data science projects
• Knowledge of data governance and security best practices, including data privacy regulations such as GDPR and CCPA