Scrapy Jobs

I need an automation script to scrape product information. Requirements: - The script should efficiently gather product data from specified sources. - It should be reliable and handle potential changes in website structures. - Data should be organized and exported in a specified format (e.g., CSV, JSON). Ideal Skills and Experience: - Proficiency in scripting languages (e.g., Python, JavaScript). - Experience with web scraping tools/libraries (e.g., Beautiful Soup, Scrapy). - Strong understanding of handling data and APIs. - Problem-solving skills to manage and debug potential issues.

I will supply the full URL of the target site and need an automated crawl that captures every internal page, pulls out all text and every embedded image, and records the direct URLs discovered along the way. Rather than a flat file hand-off, the collected material should be transferred into another website I specify, maintaining the original page structure so that links, headings, and image references remain intact. Please handle the job with a reliable stack such as Python, Scrapy or BeautifulSoup for the crawl, and use whatever bridge—REST, direct database import, or CMS API—you find most efficient for moving the scraped data into the destination site. Deliverables • A reproducible script or crawler project. • Full text and image assets migrated to the destinati...

I need a clean, well-commented script that visits a specific online business directory, extracts every company name together with its publicly listed email address, and then saves the results to a single CSV file. At the moment I’m undecided about whether I’ll reuse the code for other sites later, so please structure it in a way that wouldn’t make future tweaks painful—modules or clear functions will be appreciated, though it doesn’t have to be a full-blown framework. The directory requires no login, but it does paginate the listings. Your solution must handle that pagination automatically, avoid triggering rate-limits, and finish with a CSV that has at least two columns: CompanyName and Email. I’m happy with Python (requests, BeautifulSoup, Selenium, ...

I need a clean, repeatable script that goes through a target classified site, collects full information—titles and descriptions, prices with current availability, and all associated images—and pushes every record straight into my existing SQL database. Product information is the sole focus; other content such as user reviews or contact details is out of scope for now. The script should: • Loop through every product URL on the site (pagination included) • Extract the three data points above with accurate parsing and basic validation • Download or reference images so they can be stored or linked from the SQL tables • Insert or update rows in the database without creating duplicates, using an upsert strategy I am comfortable provisioning a stagin...

I need all text content pulled from a website and compiled into an Excel workbook each week.

I run a mid-sized UK company and want to replace a patchwork of manual tasks with one seamless, self-updating system. Everything starts and ends in Zoho CRM, so the solution must plug straight into that environment while letting me: Automate outreach & follow-ups • Build reusable email templates, marketing sequences and SMS triggers that fire automatically from Zoho when a lead is added, moved or closed. • Track every send, open and click so the CRM timeline stays complete and reporting is effortless. Gather high-value prospect data • Deploy web scrapers (Python, Scrapy, Selenium or a stack you recommend) to pull fresh contact details, direct mobile numbers and behaviour indicators from public sites and niche industry sources. • Clean, deduplicate and en...

Title: Senior Web Scraping & Data Engineering Developer for Content Intelligence Platform Project Overview: I am building a content‑aggregation and insights pipeline that collects updates, posts, and signals from major online platforms and public sources. The output will support a monthly insights publication and a long‑term analytics framework. I am seeking an experienced developer from a cost‑efficient region with strong skills in web scraping, browser automation, data engineering, and API integration. Key Responsibilities: Build and maintain scraping workflows for platforms such as LinkedIn, X/Twitter, Reddit, Substack, Facebook, Instagram, and other public sources Implement browser-based automation using tools like Playwright, Puppeteer, or Selenium Use lightweight scraper...

I need a script that visits the California State License Board site, drills down into every contractor record, and captures every publicly visible field—license number, classifications, bonding, personnel, status history, contact info, everything. The scraper must handle pagination, search-form quirks, and any soft-rate limiting the site imposes so the run finishes without manual stops or captchas blocking the flow. Once gathered, the data should be written to a single CSV file, overwriting the previous file on each run so I always have a fresh snapshot. The entire process has to trigger automatically every 24 hours (cron, systemd timer, Cloud Scheduler—whatever you prefer) and run headless on a Linux VPS I’ll provide. I am fine with Python (requests, BeautifulSoup, Sc...

I have a list of niche websites and need every relevant piece of text and its accompanying images extracted, organised, and delivered in a clean, ready-to-use format. I’ll share the URLs and the exact data points once we start, but expect a mix of article-style pages and media galleries. Scope • Capture both written content (headings, paragraphs, metadata) and all on-page images. • Provide the text in CSV or JSON and store images in clearly named folders that map back to the records. • Preserve basic structure—so each text record includes the image file name or path. • Respect and rate limits; the scrape must be discreet and repeatable. What I’d like to see in your proposal Please outline your end-to-end approach: preferred language or f...

AI Enabled Data Scraping Engineer – Junior / Mid Level Experience: 1 to 4 Years Location: Remote (Work from Home) / Bangalore / India Mode of Engagement: Full-time No of Positions: 3 Educational Qualification: B.E / / MCA / Computer Science / IT Industry: AI / Data Engineering / Automation / SaaS Notice Period: Immediate / 15 Days Preferred What We Are Looking For: 1–4 years of experience in Python-based web scraping, browser automation, and data extraction projects. Hands-on experience with Scrapy, Selenium, Playwright, Requests, BeautifulSoup, or similar scraping frameworks. Basic to intermediate understanding of AI/LLM-powered automation workflows using ChatGPT, OpenAI APIs, Claude, Gemini, or LangChain. Experience handling dynamic websites, login sessions, cookies, browse...