Data Engineer (Web Scraping/Crawling)

Job Title: Web Crawling Engineer Experience:3-5 Years Location: Hyderabad We are looking for a skilled Web Crawling Engineer with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting. Key Responsibilities Design, develop, and maintain scalable and efficient web crawlers for data extraction. Work with Scrapy, Requests, and other Python libraries to implement crawling solutions. Handle anti-scraping measures , including IP blocks, rate limits, and captchas. Implement and manage proxy rotation and session management strategies. Ensure high-quality and structured data extraction, cleaning, and storage. Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency. Collaborate with the data engineering team to integrate crawled data into pipelines. Stay updated with the latest tools, techniques, and best practices in web crawling. Required Skills & Qualifications Strong programming skills in Python . Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required) . Hands-on experience solving blocking issues , handling Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques . Knowledge of HTTP, cookies, headers, sessions, and request/response cycles. Experience with databases ( SQL/NoSQL ) for storing crawled data. Strong debugging, problem-solving, and analytical skills. Good to Have Experience with cloud environments (AWS, GCP, Azure) . Knowledge of data pipelines and ETL processes. Education & Experience Bachelor’s degree in Computer Science, Information Technology, or equivalent practical experience. 3+ years of experience in web crawling or related fields