Senior Big Data Python Engineer Let's make things Fly! Porto, Lisboa Porto

Senior Big Data Python Engineer Let's make things Fly! Porto, Lisboa
Porto
Porto, Porto District, Portugal

Let's make things Fly!

Your Role & Responsibility:

• Data Transformation and Processing: Extract, transform, and load (ETL) Flight Data Monitoring data using Spark and Py
  Spark, applying relevant algorithms based on engineering requirements to ensure accuracy and consistency.


• Algorithm Implementation: Develop, code, and test Py
  Spark algorithms that meet specific definitions and requirements provided by aircraft engineers, addressing use cases such as anomaly detection, event recognition, and trend analysis in flight data.


• Timeseries Data Management: Structure and optimize large
  - series datasets in Postgre
  SQL Timeseries, with a focus on efficient storage, retrieval, and processing for
  - time insights.


• Platform Management: Maintain data processing pipelines within Azure, ensuring scalability, performance, and data security for big data workloads.


• API and Data Delivery: Publish processed data via API or other methods to ensure seamless integration with frontend applications, enabling
  - time data access.


• Monitoring and Optimization: Continuously monitor data pipelines and workflows for performance and reliability, optimizing code and infrastructure to meet SLAs and operational goals.


• Collaboration with Aircraft Engineers: Work directly with aircraft engineers to translate their data analysis needs and specifications into Py
  Spark algorithms, ensuring that data processing outputs align with technical and operational requirements.


• Documentation and Best Practices: Document data transformation processes, maintain code quality standards, and contribute to best practices in big data engineering.