Senior Data Collection Engineer (Spain)

Job title:

Senior Data Collection Engineer (Spain)

Company:

Centric Software

Job description

LOCATION
Candidates must be legally based in Spain due to employment and compliance regulations.ABOUT US:
In today’s complex retail landscape, characterized by economic fluctuations and supply chain challenges, consumers are more discerning, often comparing prices and seeking compelling products. Centric Pricing™ addresses this by enabling retailers and brands to deeply understand the competitive landscape post-product launch. By leveraging AI-driven insights, businesses can make informed decisions quickly, aligning product development, sourcing, costing, and pricing strategies with real-time market demands.The integration of Centric Pricing™ into Centric Software’s platform provides an end-to-end solution that combines intelligence and execution capabilities. This empowers brands and retailers to optimize product availability, reduce time to market, and enhance product quality, ultimately improving the consumer experience and driving profitability.We are a key innovation partner for iconic and emerging brands across the world. Our Platform can analyze the info of more than 1.000 retailers, processing data from more than 600.000 brands, tracking millions of products.INTRODUCTION:As a Data Collection Engineer, you will be instrumental in building scalable and high-quality data collection systems, collaborating across teams to drive innovation and maintain the robustness of our data pipeline.WHAT YOU’LL DO:Design and Build Robust Web Crawlers:

• Develop and maintain spiders for high-scale data extraction using Scrapy.
• Ensure spiders are modular, reusable, and easy to maintain with components such as loaders, middlewares, and pipelines.
• Apply advanced techniques to bypass anti-bot mechanisms, including rotating proxies, captcha-solving strategies and fingerprinting

Enhance and Maintain Infrastructure:

• Build scalable CI/CD pipelines for automated testing, deployment, and monitoring of spiders.
• Leverage tools like Scrapyd for centralized spider scheduling and lifecycle management.
• Ensure efficient parallelization and cloud deployment for high-throughput crawling.

Code Quality and Consistency:

• Uphold coding standards and implement consistent practices across teams.
• Conduct thorough code reviews and mentor junior engineers on clean code principles.
• Maintain version control and detailed change logs for spider development.

Monitoring, Maintenance & Reliability:

• Integrate performance monitoring systems to ensure real-time alerts and health checks.
• Schedule periodic spider audits to handle structure changes and improve reliability.
• Troubleshoot failures and optimize resource usage (CPU/network) for crawling efficiency.

Data Integrity and Accuracy:

• Build robust data validation mechanisms to guarantee quality outputs.
• Collaborate with internal consumers to ensure data collected aligns with business requirements.
• Continuously track data anomalies and automate recovery strategies.

Collaboration and Knowledge Sharing:

• Work cross-functionally with product, engineering, and other data teams.
• Promote a culture of documentation, onboarding tools, and internal knowledge bases
• Contribute to training initiatives, helping the team stay current on scraping techniques and technologies.

DESIRED TECHNICAL SKILLS AND EXPERIENCE:Core requirements:

• Comfort with Git workflows, code reviews, and CI/CD pipelines
• Experience with cloud infrastructure like AWS
• Experience with monitoring/observability systems like Grafana and Sentry.
• Knowledge of the Web environment (model, standards, DOM, Request-Response, Cookies, JavaScript, Browsers, Headers, XHR, etc.).
• Familiarity with TLS/SSL, TCP/IP stack, and low-level web networking is a strong plus.

Bonus / Senior-Level expectations:

• Proficient in designing fault-tolerant systems and deploying them at scale.
• Familiarity with containerized deployments.
• Proficient in developing scalable web crawlers and data pipelines using Python and Scrapy.
• Experience building resilient scraping systems across diverse web architectures.
• Prior experience mentoring or leading junior developers.

Soft Skills and Work Ethic:

• Excellent communication skills in English, both written and spoken.
• A collaborative mindset with a proactive approach to knowledge sharing.
• Strong analytical thinking and problem-solving abilities.
• Commitment to continuous improvement, mentoring, and agile team dynamics.
• Remain up-to-date with technology trends to keep our software as innovative as possible.

Centric Software provides equal employment opportunities to all qualified applicants without regard to race, sex, sexual orientation, gender identity, national origin, color, age, religion, protected veteran or disability status or genetic information.Powered by JazzHR

Expected salary

Location

España

Job date

Fri, 23 May 2025 02:43:48 GMT

To help us track our recruitment effort, please indicate in your email/cover letter where (vacanciesin.eu) you saw this job posting.