Senior Data Engineer (Python)

Description

Who are we?

OceanStream is a cloud-based data platform for marine and oceanographic research. It is built by Pineview Labs – an independent software development consultancy based in Oslo, Norway and Iași, Romania. We specialise in fullstack web development and data engineering, building end-to-end data solutions using IoT technologies. 

We are a 100% bootstrapped company and we have the freedom to work on projects which we believe in, while also forming strong connections with the organisations with whom we collaborate. One of our most successful projects is the automated testing library Nightwatch.js – which we’ve published in 2014 and which has been acquired by BrowserStack in 2021.

What are we working on? 

OceanStream which aims to facilitate the collection and analysis of real-time data from sensors and hydroacoustic instruments. Hydroacoustic data is crucial in a variety of domains but our main focus is on:

• marine ecology and conservation: by analysing hydroacoustic data, researchers study the distribution, behaviour, and abundance of marine organisms, including fish, mammals, and plankton, which informs conservation strategies, habitat protection, and sustainable management practices for maintaining the health of marine ecosystems.
• oceanography: hydroacoustic data provides insights into the ocean floor’s topography, water column properties, and underwater habitats, helping scientists understand ocean dynamics and processes.
• environmental monitoring: the study of hydroacoustic data helps with detecting and tracking changes in marine ecosystems, identify pollution sources, and monitor the impact of climate change on underwater environments.

More info about water column sonar data can be found on this article.

The tech stack

The platform consists of a container-based architecture which enables development and deployment of containerised modules across a varied landscape of IoT Edge devices, with AI at Edge capabilities (training models in the cloud and deployment on the edge devices for local inference using real-time data).

We are using Azure IoT Edge as the underlying framework and for individual module development we are using Docker, Node.js, Python, and/or R.

Requirements

If you’re interested in working with us send your CV and tell us a bit about yourself and which are your areas of interest.

Your role involves

• develop and maintain open-source libraries for data processing and analysis (published on our Github page)
• work with data from various types of sensors and hydroacoustic instruments, such as echosounders, multibeam sonars, and ADCPs (Acoustic Doppler Current Profilers) and build scalable data analysis pipelines and workflows using Python, Dask, and Azure services
• develop data visualisation tools and collaborate with UI/UX to integrate them into our web apps
• implement containerisation strategies using Docker and Kubernetes
• collaborate with data scientists and platform architects to integrate ML models into our data platform, including LLMs for Gen AI using the Azure OpenAI platform

Qualifications

To be successful in this role, you should have:

• a degree in Computer Science, Data Science, Engineering, or alternatively extensive work experience in these fields

• proven experience as a Data Engineer, with expertise in Python and advanced parallel data processing tools like Dask
• extensive working knowledge of NoSQL databases and REST apis
• strong background in Docker, with familiarity in Kubernetes
• experience in cloud-based environments, preferably Azure
• experience with Git and working with Github
• experience with CI/CD pipelines in a Linux environment
• an interest in oceanography, marine research, and ecological sustainability
• strong communication and teamwork abilities.

Benefits

Instead of a job with empty promises of being part of “something bigger”, trapping you in endless planning meetings and retrospectives, we only give you a space where you can experiment, discover, and create. We offer a unique opportunity to work on open-source software and on projects that can deliver value to scientific research and marine ecology studies.