Data Engineer Leiria

Overview

Industry: E-commerce (Fashion & Home)

Location: Portugal

Work Model: 100% Remote

Start Date: ASAP

Project Language: English

Project Overview: Join a leading
- commerce company in fashion and home décor, as they undertake critical data engineering projects to support their evolving business needs. This is an exciting opportunity to contribute to
- impact data solutions for an innovative,
- focused brand, leveraging the latest in data engineering best practices.

Responsibilities

• Project Understanding and Communication: Analyze business and technical challenges from a user perspective. Collaborate with Data Architects and Project Managers to ensure solutions align with the client’s data architecture.
• Data Pipeline Development: Design, build, and deploy efficient data pipelines according to project requirements. Apply best practices for performance, scalability, and maintainability. Use Terraform to deploy and manage infrastructure efficiently.
• Testing and Deployment: Define test cases and conduct testing in collaboration with the Project Manager. Present completed developments to Data Architects and Lead Data
  Ops, ensuring smooth deployment and active monitoring
  - deployment.
• Documentation and Peer Review: Document processes, tests, and results thoroughly. Conduct peer reviews and participate in code reviews for quality assurance.

Requirements

Hard Skills

• Proficiency in Py
  Spark and Spark SQL for data processing.
• Experience with Databricks and Delta Live Tables for ETL and workflow orchestration.
• Familiarity with Azure Data Lake Storage for data storage and management.
• At least 1 year of experience with Terraform and Git
  Ops practices for infrastructure deployment.
• Strong understanding of ETL/ELT processes, data warehousing, data lakes, and data modeling.
• Knowledge of orchestration tools (e. g. , Apache Airflow) for pipeline scheduling and management.
• Experience with data partitioning and lifecycle management in cloud storage.

Optional: Experience with Databricks Asset Bundles, Kubernetes, Apache Kafka, and Vault is a plus.

Soft Skills

• English Fluency: Strong written and verbal English skills in a working environment.
• Communication: Ability to convey technical concepts effectively and understand user needs.
• Organizational Skills: Detail-oriented with the ability to maintain structured documentation.
• Problem-Solving: Proactive approach to understanding and addressing data challenges.