Data Engineering Specialist

ABOUT THE OPPORTUNITY

The Data Engineering Specialist, Media Viability Accelerator (MVA) will head up an important function for a cutting-edge, online platform that will set a new standard for the industry — developing, testing, and deploying data components for the MVA.

Reporting to the Media Business Unit’s Head of Technology, the Data Engineering Specialist will oversee Azure Synapse, notebooks, pipelines, and related resources. This role entails managing data pipelines, databases, and visualizations for the MVA, including Power BI embedded analytics. Additionally, the role involves implementing data scraping for enriched datasets and liaising with the Back-End Web Development Specialist.

ABOUT THE MVA

Only by staying in business can independent news and information providers keep people safe and able to make informed decisions that affect their lives. With the rise of digital media, social and entertainment platforms, and disinformation, these independent media struggle to survive. To help meet this challenge, Internews has secured a 3-year program from USAID which will be known as the Media Viability Accelerator (“MVA”) Forward. It will be a highly visible initiative (announced at the 2021 and 2023 Summits for Democracy) and could have substantial impact on the media sector worldwide.

MVA Forward is contributing to the development of a new online platform supporting independent media in strengthening their business performance. Build on Azure and using ASP.NET Core, Angular, and Bootstrap with embedded Power BI data visualizations, the platform offers a growing array of tailored insights and opportunities that can transform media businesses. By making these opportunities intuitive and easy to act on, independent media at scale will engage to create a thriving new ecosystem that unlocks new paths to long-term financial security.

Brought to Minimum Viable Product in January 2024 after seven months of development, the platform is now ready for phased launch through public launch by August 2024. Over the coming years, the development team will add features, functionality, and languages so that independent media around the world can use the platform. In collaboration with Microsoft, the integration of AI to create powerful insights and recommendations is a priority. To ensure deep understanding of the MVA’s subject matter, purpose, and datasets, a small and dedicated team will lead further development, supplemented by contractors for any specific deliverables beyond the team’s capacity.

LOGISTICS

Internews has Headquarters offices in Washington, DC and London, UK which support a hybrid workforce with staff working remotely, fully in-office, or partially remote. This is a remote-based role, and candidates based anywhere in the United States or United Kingdom will be considered.

Applications will be reviewed on a rolling basis, so candidates are encouraged to apply soon.

This is a limited-term position with an expected end date of September 24, 2026.

OUR COMMITMENT TO FOSTERING A CULTURE OF BELONGING

We are an organization of dynamic, mission-driven individuals who are passionate about our core values and about supporting positive change in the world. We pride ourselves on our commitment to innovation and flexibility. We believe that diverse teams are strong teams and work to support an ethic of belonging, dignity, and justice for all people. Our current team includes a mix of genders, parents and non-parents, and people of multiple races, nationalities, ages, sexual orientations, and socioeconomic backgrounds. We are an EEO employer and encourage candidates of all races, genders, ages, orientations, ethnicities, and national origins to apply, and welcome those with alternative backgrounds and experiences.

DAY-TO-DAY TASKS will include:

General

• Commitment to understanding the MVA’s subject matter, purpose, datasets, and core technologies to ensure meaningful contribution to the program’s overall objectives.
• Contribute to defining the scope, objectives, and activities of the MVA.
• In all duties, upholding Internews’ Core Values and demonstrating a commitment to fostering a culture of Belonging, Dignity, and Justice.

Collaboration

• Collaborating with the MVA Forward program team, which leads the rollout of the platform globally, ensuring partner and stakeholder feedback is reflected in the MVA.
• Working closely with colleagues to improve application usability and feature integration.
• Working in Scrum and contributing to sprint planning, goal setting, and product design.
• Communicating with, coordinating with, and/or managing the work of external contractors and solutions providers.

Data Engineering

• Data Pipelines: Maintain and extend analytics pipelines and reports with Power BI, Power Query, Azure Data Factory, and Azure Synapse. Focus on integrating real-time data processing and automated data quality checks to support dynamic business needs.
• Database Systems: Build and manage database systems for efficient indexing, querying, pre-fetching and caching with active management of Azure SQL and Synapse Analytics. Implement best practices for database versioning and schema management to ensure system robustness and data integrity.
• Optimizing Performance: Optimize connectors, memory management, and queues for maximum accessibility, speed, scalability, and future compatibility. Monitor and identify bottlenecks in Power BI and data pipelines, with adjustments grounded in testing and user feedback.
• Data Storage: Manage, maintain, and rotate encrypted data stores — including data warehousing, transformation, staging, and quality assurance. Employ advanced encryption methods and disaster recovery strategies to safeguard data.
• Automated Task Management: Oversee scheduled tasks for user registration and data scraping operations to maintain efficient data processing workflows.
• API Integration, Management and Python Scripting: Develop and maintain integrations with platforms such as Google Analytics, Meta, YouTube, MailChimp, WhatsApp, and Telegram for data retrieval and analysis. Additionally, develop, improve, and maintain Python scripts for web scraping, extracting data on a scheduled basis from public directories and user-specified sources such as partner web properties.
• Integration with Back End: Work closely with the Back-End Web Development Specialist to ensure seamless collaboration to integrate the data-gathering process with backend services.
• Data Modeling: Contribute to the ongoing evaluation of the MVA’s data models and reports.
• Documentation: Maintain and extend documentation of processes and components. Ensure documentation is accessible and up to date to support team collaboration and knowledge sharing.
• Code Maintenance and Improvement: Maintain and enhance the existing codebase for optimal performance and scalability, with a focus on future compatibility.
• Testing: Implement continuous logging and testing strategies. Adopt Test-Driven Development (TDD) or Behavior-Driven Development (BDD) methodologies to ensure high-quality outputs and efficient bug tracking.
• Actions and Automation: Ensure the smooth scheduling and execution of pipeline tasks. Manage cycles of updates, release and deployment for components. Track and maintain automated maintenance and environment workflows.
• Project Management Tools: Issues and docs in Azure DevOps and Jira are key to project management, tracking tasks, and collaborating with team members.

QUALIFICATIONS WE’RE LOOKING FOR

Required

• A minimum of 10 years of prior relevant experience, including at least three years of professional experience as a Data Engineer, with a proven track record of successful projects.
• Microsoft Azure Data Engineer certification, indicating in-depth knowledge of Azure services and best practices.
• Familiarity with Azure cloud services including ADLs, Azure Key Vault, and Azure Batch.
• Advanced experience or certifications in Azure cloud services and solutions strongly preferred.
• Database and data warehousing expertise, including experience with Azure SQL Database and Azure Synapse Analytics for data storage and warehousing.
• Ability to write and maintain Python scripts for data scraping tasks, highlighting versatility in programming language and frameworks.
• Integration experience, including OAuth authentication and working with RESTful APIs for data retrieval, manipulation, and third-party service integration.
• Proficiency with continuous integration and continuous deployment (CI/CD) practices and tools, along with experience in using version control systems like Git, ensuring best practices in code management and deployment workflows.
• Excellent written and spoken English skills.
• Proven interpersonal skills, including excellent team-building and communication skills.
• Experience with agile software development in a team.
• Ability to work effectively both independently and in close collaboration with colleagues, across multiple locations.
• Demonstrated organizational skills, ability to multi-task and prioritize workload.
• Self-reliance, resourcefulness, flexibility, and problem-solving attitude.
• Optimism, a sense of humor, and excitement regarding this work.

Preferred

Note: Candidates who do not have the preferred qualifications below, but who are interested and willing to learn, are encouraged to apply.

• Experience working with performance metrics for data visualization.
• Ability to troubleshoot complex issues and develop effective solutions.
• Extensive experience with best practices in data security and compliance.
• Experience working with news media.
• Proficiency in any other spoken language.