Natixis in Portugal is currently seeking a Data Engineer to join its BPCE SI (BPCE Solutions Informatiques) team, within the IT (Information Technologies) department, to work in a hybrid model in the Porto or Lisbon office.
**Responsibilities **
Data and Database Migration:
-
Actively participate in the design and execution of the data and ETL process migration strategy from SQL Server to GCP.
-
Develop and optimize scripts and pipelines for efficiently transferring and transforming data into BigQuery.
Design and Development of Data Pipelines on GCP:
-
Design, develop, and maintain robust and high-performing ETL/ELT pipelines on GCP, replacing or rebuilding existing flows.
-
Orchestrate and schedule these pipelines using Google Cloud Composer (or Apache Airflow) to ensure reliable migration and operationalization.
Database Management and Optimization on GCP:
-
Develop and optimize complex SQL queries for BigQuery, taking into account the specificities of migrating from SQL Server to BigQuery (e.g., optimizing partitioning, clustering).
-
Contribute to the design and implementation of optimized data schemas for BigQuery, adapted to new cloud architectures.
Development and Automation:
-
Use Python to develop scripts for automating migration tasks, data transformation, and integration with other GCP services.
-
Implement processes to ensure data quality and integrity during and after migration.
Collaboration and Support:
-
Collaborate closely with current teams (SQL Server administrators, analysts) to understand the existing environment and facilitate the transition.
-
Exchange with end-users (business lines, data scientists) to understand their needs and ensure the new platform meets their expectations.
-
Participate in code reviews, development best practices, and testing strategies for the new platform.
-
Ensure monitoring, debugging, and incident resolution related to data pipelines during the transition phase and upon production deployment.
Technological Watch and Optimization: