Data Engineer GCP (Google Cloud Plataform) Porto

Data Engineer GCP (Google Cloud Plataform)
Porto
Porto, Porto District, Portugal

Job Description

We are looking for a skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for understanding business and technological challenges, developing data pipelines to address those challenges, and ensuring their smooth deployment. You will also be responsible for applying standard industry and company best practices, as well as evolving our various patterns. This is a remote position.

MAIN RESPONSIBILITIES

1. Project Understanding and Communication

• Understand problems from a user perspective and communicate clearly to grasp the issues.
• Ensure the architecture provided by the Data Architect is well understood.
• Communicate with the Data Architect, project managers, and peers regarding technical solutions and project updates.

• Write and communicate interface contracts for new or updated components.
• Develop data pipelines based on the defined architecture.
• Apply best practices regularly.
• Deploy infrastructure using tools like Terraform.
• Participate in peer reviews and review peers' code during merges.

• Define tests with your project manager based on functional and technical requirements.
• Perform tests and report results regularly.
• Summarize test results in documentation.

• Present deployment work during review sessions with Data Architect and Data
  Ops Lead.
• Monitor and communicate any errors
  - deployment.
• Ensure proper deployment processes, logging, and monitoring are followed.

REQUIRED SKILLS

• Google Cloud Platform: General knowledge and at least 1 year of experience.
• Azure Cloud Platform: Knowledge is a plus.
• Apache Airflow: At least 2 years of experience; Google Composer experience is a plus.
• Google Big
  Query: Extensive experience (≥2 years), including query optimization and database design.
• Terraform: At least 1 year of experience.
• Apache Spark: Optional, but highly valued; ability to maintain and evolve Py
  Spark pipelines.
• Additional skills include Pub/Sub, Kafka, Google Cloud Storage, and Dataflow or Apache Beam, which are a plus.