Guides
Sites and robots.txt
These are the sites we can read, grouped by type. To watch one, put the entry from its where.sites column in your watcher's where.sites.
If a site's robots.txt disallows the pages with the data, the watcher needs fetch.accept_site_terms: true, which means you take responsibility for fetching it under that site's terms (see robots.txt). If a site walls our server off or isn't connected yet, creating or previewing the watcher fails with site_unavailable.
Dates in attributes.from and attributes.to are YYYY-MM-DD, and both days are included. Sites that take a time zone read attributes.time_zone, an IANA name like America/New_York. These and the other settings below (like party size, ids and selector) tell us what to fetch, and they don't filter the results the way size or color do.
Stores
| Site | where.sites | match | Gives |
|---|---|---|---|
| Any online store | nalgene.com, keychron.com, tandemceramics.com/shop | url, line, category | The platform (Shopify, WooCommerce, Squarespace, BigCommerce or Magento) is found on the first check and read as in its row below. Other stores' product pages are read from schema.org data. Checked at most hourly, like Shopify, since the platform isn't known before the first check |
| Any Shopify store | a /products/ link, as a url match | url, line, category | price, stock, quantity, every option |
| Any WooCommerce store | nalgene.com/product-category/bottles | url, line, category | price (exact), stock, low-stock count, every variation option |
| Any Squarespace store | tandemceramics.com/shop, <name>.squarespace.com | url, line, category | price (exact), stock and count per variant, every option. A listing reads 8 products in full per run |
| Any BigCommerce store | store.com/shop-all (a category page), <name>.mybigcommerce.com | url, line, category | url: price and stock per option combination (up to 24). Listing: price and Add to Cart per product. A line watch needs a category page, since robots.txt disallows BigCommerce's search |
| Any Magento / Adobe Commerce store | www.vitamix.com, or a store view like bulk.com/uk | url, line, category | price (exact) and stock per configurable variant, every option, sale and regular price |
| Any page with schema.org product data | sweetwater.com | url | price, stock |
| Nike | nike.com | url, sku | price, stock per size, low stock, release date |
| Amazon | amazon.com | url, sku, exact | price, stock |
| Running Warehouse, Zappos, Brooks, Saucony, On, adidas, New Balance | the store's host | url, some sku or exact | price, stock per size |
If a store isn't listed, preview one of its product URLs. Most stores run on one of these platforms or publish schema.org data, which we can read without setup.
Marketplaces
| Site | where.sites | match | Gives |
|---|---|---|---|
| eBay | ebay.com (or ebay.co.uk, ebay.de, ebay.ca, ebay.com.au), a search link ebay.com/sch/i.html?_nkw=leica+m6&_udhi=1500&LH_Auction=1, or ebay.com/itm/<id> | url, line, category, listing | Listings, newest first (auctions ending soonest first), with exact price (the current bid for auctions), listed date, condition, shipping, seller feedback, image, and ends_at and bids for auctions. No seller names. A plain search is mostly Buy It Now, so set "auction": true in target.attributes for auctions only, or false for Buy It Now only. A search link's price range, format, condition and category go to eBay. Fails with site_unavailable until eBay is connected. No page scraping, since robots.txt disallows /sch/ and item pages answered 403 |
| Discogs | discogs.com/release/249504 or discogs.com/seller/<name> | url, listing, line, category | Release: the lowest asking price (USD, exact) and copies for sale. Seller: their newest listings, which need a terms review before use because robots.txt disallows /users/. |
| Reverb | reverb.com, a search link reverb.com/marketplace?query=boss+ce-2&price_max=200&condition=used, or reverb.com/item/<id> | url, line, category, listing | Music gear with make, model, exact price, shop, listed date, condition, finish and year. Searches come in Reverb's order, not newest first (robots.txt disallows sorting), so keep them narrow |
| Etsy | etsy.com, a search link etsy.com/search?q=mechanical+keyboard&min=50&max=200, or etsy.com/listing/<id> | url, line, category, listing | Active listings, newest first, with exact price, quantity, listed date, shop and favorites. Fails with site_unavailable until Etsy is connected |
reddit.com/r/Pen_Swap, or reddit.com/r/pens+fountainpens | url, line, category | Newest posts with title, link, author, posted time, the price from the title or text, and whether it's sold. New posts show up within about 15 minutes. A new subreddit's first check takes about 5 minutes | |
| Craigslist | A search link, www.craigslist.org/search/area/sfbay?cat=apa&max_price=3800&min_bedrooms=2, or the older sfbay.craigslist.org/search/apa?... | listing, line | One listing per first-page result, with title, link, exact price and neighborhood, plus beds and baths when stated. Set fetch.accept_site_terms: Craigslist's terms forbid automated searching, and the flag means you take that on. A new listing is items added. One that drops off the first page is removed, which doesn't mean it was taken down. Checked at most hourly, and the server fetches at most 100 Craigslist pages a day |
This watcher looks for new Craigslist apartments under $3,800:
{
"target": { "query": "www.craigslist.org/search/area/sfbay?cat=apa&max_price=3800&min_bedrooms=2", "match": "listing" },
"conditions": { "all": [{ "field": "items", "op": "added" }] },
"where": { "sites": ["www.craigslist.org/search/area/sfbay?cat=apa&max_price=3800&min_bedrooms=2"] },
"schedule": { "every": "1h", "until": "+30d" },
"fetch": { "accept_site_terms": true }
}Auctions
| Site | where.sites | match | Gives |
|---|---|---|---|
| HiBid | hibid.com/lots?q=rolex, or hibid.com/lot/<id> | url, line, category, listing | The first page of lots (25), with high bid (or the minimum bid when nobody has bid), bid count, close time, auction house and image. Flagged list_truncated when there are more. Live catalog, webcast and account pages aren't read |
| Spectrum Wine | spectrumwine.com, with a query | url, line, category, listing | The current auction, through the search API robots.txt allows, with current bid (USD), vintage, size, score, estimate and lot link. At most 100 lots per check |
| Bidsquare | A catalog or lot link, hakes.com/auctions/... or bidsquare.com/auction-house/... | url, line, category, listing | One listing per lot on the page, with the current bid when labeled (otherwise the starting bid, only when nobody has bid), bid count, estimate and image. The item link is the lot's page. /search isn't read. Other pages are read as a store |
| Hart Davis Hart | hdhauctions.com, or a link to its .xlsx lot list | url, line, category, listing | Every row of the sale's spreadsheet: wine, producer, bottle size, quantity, low and high estimates. No current bid, and the bid site isn't read |
| WineBid | winebid.com | url, line, category, listing | Current lots from the public sitemap, with title, vintage and lot link. No bids, because lot pages are behind a firewall we don't open. Set fetch.accept_site_terms. WineBid asks for a licensing email, which is up to you to send |
We don't read Auction Mobility (Hart Davis Hart's bid site), because it loads lots in the browser and has no verified public list.
Cars
| Site | where.sites | match | Gives |
|---|---|---|---|
| Dealer.com inventory | A dealer's inventory page (hendrickcars.com/new-inventory/index.htm) or one vehicle's page (/new/BMW/2026-BMW-2-Series-abc.htm) | url, listing | One product per car on the page, with exact price, VIN, stock number, year, color, new or used, and dealer name. The next page isn't followed (flagged list_truncated), so watch a filtered inventory URL or one vehicle. Other pages are read as a store. No key needed |
| Xtime | consumer.xtime.com | url, listing | Not supported: the scheduler disallows every path and its homepage is a 404. Refused with site_unavailable |
| myKaarma | app.mykaarma.com | url, listing | Not supported: the app disallows every path and only has a login page. Refused with site_unavailable |
Apartments
| Site | where.sites | match | Gives |
|---|---|---|---|
| Apartment listings (AppFolio) | sdppm.appfolio.com, cbpm.appfolio.com/listings, or a manager's own page with AppFolio's embed (sdppm.com/available.html) | url, listing | One listing per unit for rent, with exact monthly rent, beds, baths and sqft as numbers, move-in day as date (a unit available now gets the check's day, Pacific time), and address. attributes.property_group limits it to one AppFolio property group, and an unknown group reads as empty, so check the name. A unit with no rent gets price_missing. Craigslist is under Marketplaces |
| Apartment unit maps (SightMap) | A community page with a sightmap.com unit map (oakrow.com/floorplans), or sightmap.com/embed/<id> | url, listing | One listing per available unit, with exact rent (plus total_monthly when fees are added), available_date, beds, baths, sqft, unit number, floor plan, building, floor, lease term and leasing system (yardi_rentcafe, realpage_v2, ...) |
| RentCafe | rentcafe.com, securecafe.com | url, listing | Not supported: Cloudflare blocks us, robots.txt disallows the unit pages, and the API needs a Yardi key. Watch the community's own floor plans page instead if it has a SightMap unit map. Refused with site_unavailable |
| Entrata | <property>.prospectportal.com | url, listing | Not supported: Cloudflare blocks every request, even robots.txt. Refused with site_unavailable |
If a page's path is about units for rent (like /floorplans, /available.html or /apartments/...), we read it as an apartment page, or with a learned recipe if it shows neither AppFolio nor SightMap.
Booking and appointments
Booking sites give one slot per open time. To match one, use { "field": "availability", "op": "=", "value": "open_slot" }.
| Site | where.sites | match | Gives |
|---|---|---|---|
| Cal.com | cal.com/peer/15, cal.com/team/cal/intro, acme.cal.com/jane/30min | listing, url | One slot per open time from from to to (default 14 days) in time_zone (default UTC), with start and offset, duration, host, price, and seats left on seated events. attributes.duration picks a length on events with several |
| Calendly | calendly.com/<user>/<event> | listing, url | One slot per open start time (default 14 days, up to 93), with seats left for group events, length, location and host. attributes.timezone reads the days in another time zone. Set fetch.accept_site_terms |
| Acuity Scheduling | app.acuityscheduling.com/schedule.php?owner=26704703, <name>.as.me | url, line, category, listing | One item per appointment type, with name, exact price, duration_minutes, category and service_id. Open times need a listing match. Set fetch.accept_site_terms, for service menus too |
| Square Appointments | book.squareup.com/appointments/<id>/location/<id>/services | url, line, category, listing | One item per service, with fixed price, duration_minutes, category, staff and place. Open times need a listing match (service or service_id, staff or staff_id, from, to, up to 62 days) and the browser. Set fetch.accept_site_terms, for service menus too |
| Jane App clinics | versamc.janeapp.com, versamc.janeapp.com/locations/versa-movement-collective/book, .../book#/staff_member/2/treatment/1, <clinic>.janeapp.co.uk | listing, url | One slot per open time for one visit type at one location, from to to (default about 14 days, at most 28), with start and the clinic's offset, length, practitioner, visit type, location, and price when shown. Set the visit type with treatment_id or treatment, the practitioner with staff_member_id or practitioner (omit for all), and location when there are several. Without a visit type, the check fails and lists the clinic's visit types |
| SimplePractice (therapy, counseling, small clinics) | example-practice.clientsecure.me | listing, line, category | One slot per open time for the next 7 days (at most 14), with exact start and end and their offset, clinician, service, length, office or video, and listed rate. Narrow it with clinician, service, video and time_zone. Without time_zone, days are picked in New York time (time_zone_default). Only public booking times are read, never client data. Call-to-book services are skipped |
| Vetstoria vet booking | us.vetstoria.com/booking/<clinic> | listing, line | Not read: open times only show after starting a booking, and the terms forbid bots |
| NexHealth | app.nexhealth.com/appts/<practice> | listing, url | Once reviewed, one slot per open time per provider at one location (location_id, optional provider_id, slot_length). Set fetch.accept_site_terms |
| Zocdoc | zocdoc.com/doctor/<name>, zocdoc.com/dentist/<practice>, zocdoc.com/neurologists/phoenix-220419pm | listing, url | One slot per open time from the availability grids on a public profile or a specialty and city page, with provider, specialty, place and exact start with offset. from and to filter by the appointment's local day. Fails with site_unavailable until Zocdoc is connected. Each request counts against a monthly Bright Data budget we share with Reddit (Server keys). Search, booking and specialty index pages aren't fetched, and only times already in the page are read |
| Epic MyChart open scheduling | <health system>/mychart/SignupAndSchedule/EmbeddedSchedule?id=..&dept=..&vt=.. | listing, url | Not supported yet: the pages need a browser session, and we haven't read Epic's and the health system's terms. Refused with site_not_reviewed |
| Fresha | fresha.com/a/perfect-color-salon-waukegan-1206-golf-road-xkd1xi4g | url, line, category | One item per service variant, with exact price (a "from" price goes in price_from with price_type: starts_at), duration_minutes, category, restriction (like "Female only") and place. attributes.service picks one service. Group appointments the page doesn't list are left out, and the check flags partial_data. No open times: Fresha only shows them in a booking cart on the salon's account (and may ask for a reCAPTCHA), so they're off until someone decides carts are acceptable |
| Booksy | booksy.com/en-us/388730_barber-shop-nyc_barber-shop_30067_new-york-city | url, line, category, listing | One item per service variant, with exact price ("$65.00+" goes in price_from), duration_minutes, category and staff. attributes.staff (a name) or staff_id keeps one person's services. When only schema.org offers can be read (partial_data), the number goes in price_from with price_type: unknown and price is empty. Open times need a listing match with service and fetch.accept_site_terms. The region API can still answer with a captcha (blocked) |
| Vagaro | vagaro.com/<business> | url, line, category, listing | Services with name, price and duration_minutes. Open times need a listing match with service or service_id, up to 7 days. Set fetch.accept_site_terms. time_zone adds an offset to the times |
| Boulevard | joinblvd.com/b/<business>/widget, blvd.app/@<business> | url, line, category, listing | Not supported yet: the widget creates a cart on the salon's account as soon as it opens, before showing services or times. Needs a review first |
| Trusted Traveler interviews | ttp.cbp.dhs.gov/5140 (a location id) | listing | Global Entry, NEXUS and SENTRI interview slots with open spots, in the center's local time |
| Recreation.gov permits | recreation.gov/permits/233262 | listing, url | One slot per entry point per day, with permits left and walk-up flag. attributes.division picks one trailhead. Set fetch.accept_site_terms |
| Recreation.gov campsites | recreation.gov | listing | Open campsite nights. Set fetch.accept_site_terms |
Restaurants
Set attributes.party_size (2 by default) and either date or from/to in the restaurant's local time, at most 14 days per check. time_from and time_to (HH:MM) narrow the hours. Each slot has party_size, seating, local_date, local_time and time_zone.
| Site | where.sites | match | Gives |
|---|---|---|---|
| Resy | resy.com/cities/ny/venues/lilia | listing, url | Every open time for your party, with seating (Dining Room, Bar, Patio), length, deposit and cancellation fee |
| OpenTable | opentable.com/r/carbone-new-york | listing, url | Open times near the time you ask for, with seating and table type. Needs fetch.accept_site_terms: true. Fails with site_unavailable until OpenTable is connected |
| Tock | exploretock.com/alinea | listing, url | Each time and experience for your party, with tickets left and the prepaid price. Cloudflare blocks us for now |
Tickets and events
| Site | where.sites | match | Gives |
|---|---|---|---|
| Eventbrite | eventbrite.com/e/snakehips-twofaced-tickets-1993478754618 | url, listing | Lowest and highest price, on sale, sold out or not yet on sale, sales start (opens_at) and end, start time, venue. Ticket classes when Eventbrite is connected |
| Luma | luma.com/u9l1w54q (an event), luma.com/genai-sf (a calendar), luma.com/sf (a city) | url, listing, line | Events with exact price, open, low, sold out or waitlist, spots left, start time, place. Event pages add each ticket type |
| Ticketmaster | ticketmaster.com | url, line, category, listing (with attributes.venue_id or attraction_id) | Face-value price range, on-sale state and time (opens_at), start time, venue. Fails with site_unavailable until Ticketmaster is connected |
| SeatGeek | seatgeek.com | url, line, category, listing (with attributes.performer or venue_id) | Lowest resale price, listing count, average and median price. Needs fetch.accept_site_terms (its API's robots.txt disallows everything). Fails with site_unavailable until SeatGeek is connected |
| DICE | dice.fm/event/6a0b2ac3c6b6f90001b98470 (an event), dice.fm/browse/london (a city), dice.fm/venue/earth-76a7 (a venue) | url, listing, line | Events with exact price (or the lowest, as price_from, when there are several ticket types), on sale, sold out or announced, waiting list, sale and presale start (opens_at), start time, venue, promoter. Ticket types aren't read (ticket_types_not_read) |
| Meetup | meetup.com/torontojs (a group's upcoming events), meetup.com/torontojs/events/317276003 (an event) | url, listing, line | Events with the fee and how it's paid, spots left against the limit, going and waitlist counts, full or RSVPs closed, RSVP opening (opens_at, event pages only), start time, venue, group |
| Tito | ti.to/clearleft/ux-london-2027 (an event), ti.to/clearleft (an organizer's events) | url, listing, line | An event gives each ticket type with exact price and state (available, low, sold out, waiting list, not yet on sale), tickets left when shown, sale start (opens_at) and end, tax note and event date. A time without a zone keeps only the date, with the clock time in local_start. An organizer gives its upcoming events, without prices. In conditions, give GBP and EUR prices as a bare number |
| Partiful | partiful.com/e/AWx1DaE9DfUoBmqFTaVU | url | Public events only, with open, low, full or closed, spots left, capacity, waitlist, RSVP deadline, start time, place, guest counts (no names). Partiful's own status decides, so an event cancelled only in its description reads as open |
| Sessionize | sessionize.com/ndc-oslo-2027 (a call for speakers), sessionize.com/api/v2/jl4ktls0 (a published schedule) | url, listing, line | A call gives open, upcoming or closed, exact closing time, opening time (opens_at), event days, venue, travel and fee offers. A schedule gives one listing per session with start, end, room, speakers, track |
| start.gg | start.gg/tournament/genesis-x2, or listing with attributes.videogame_id | url, listing, line | Registration open or closed, closing time, attendees, start time, venue, and each event's entrants and game. Fails with site_unavailable until start.gg is connected |
| AXS | axs.com | url, listing | Not available. AXS sends our server a bot check, and its API is partners only |
| See Tickets | seetickets.com, seetickets.us | url, listing | Not available. Both send our server a bot check (seetickets.us is now eventim.us) |
| SignUpGenius | signupgenius.com/go/<id>, or signupgenius.com for the account's active signups | url, listing | Open slots on a signup you own, with spots left (myqty), item, start time. A bare signupgenius.com lists your active signups, not every slot. Fails with site_unavailable until SignUpGenius is connected. Names, emails and the public /go/ page aren't read |
| Challonge | challonge.com | url, listing | Not available. Cloudflare's human check blocks our server, and the API needs a key |
| Battlefy | battlefy.com | url, listing | Not available. The API refuses direct access, and the site is an empty app shell |
Classes, courts and tee times
| Site | where.sites | match | Gives |
|---|---|---|---|
| ActiveNet (city recreation) | activecommunities.com/denver | url, line, category, listing | One slot per program, with spots left, class size, fee, first session time, place, and opens_at before registration opens. Full, closed and cancelled count as out of stock |
| CivicRec (Rec1) city recreation | secure.rec1.com/CA/temecula-ca | listing, line, category | One slot per program session, with exact start and the agency's offset, fee, days, ages, place, and opens_at before registration opens. Full, closed and cancelled count as out of stock. Rec1 doesn't publish spots left, so quantity is empty. Pick tabs with tab ("Swim Lessons"). kind: facilities gives one slot per facility and day (from to to, 14 days by default, 31 at most): available, limited or unavailable, with rates. Facility days have the date, not the hours (hours_not_read) |
| Amilia program registration | app.amilia.com/store/en/example-org | url, listing | One slot per activity, with spots left as quantity, class size, start, schedule, ages, waitlist, and price when you set currency. drop_in: true adds a slot per drop-in session with its own spots left. Watch by activity link (url) or activity_ids (listing). Fails with site_unavailable until Amilia is connected (it takes a partner agreement). Not yet checked against a live answer |
| Mindbody | mindbodyonline.com/explore/locations/dear-pilates | url, line, category, listing | One slot per class from from to to (7 days by default, 14 at most), with spots open, capacity, waitlist, instructor, start time, and opens_at before booking opens |
| foreUP golf tee times | foreupsoftware.com/index.php/booking/19765/2431 | listing, url | One slot per open tee time, with spots left, holes, green fee, cart fee. booking_class, holes and party are optional. Set fetch.accept_site_terms |
| GolfNow tee times | golfnow.com/tee-times/facility/1225-audubon-park-golf-club | listing, url | One slot per open tee time from from to to (3 days by default, 7 at most), with per-player rate, holes, rate name, Hot Deal, transaction fee, and the largest group as quantity. holes (9 or 18), party (1 to 4) and time_zone are optional. Without time_zone, times use the course's offset on the check day, so times past a clock change in the window are an hour off |
| Chronogolf tee times | chronogolf.com/club/miami-beach-golf-club | listing, url | One slot per open tee time at every course of the club (3 days by default, 7 at most), with spots left, holes, starting hole, public green fee, deals. A held time is out_of_stock. holes and party are optional. Chronogolf's firewall blocks our tee-time requests for now, so checks report blocked |
| Mariana Tek class schedules | barrysbootcamp.marianatek.com?location=10462 | url, line, category, listing | One slot per class (7 days by default, 14 at most), with spots left (when shown), class size, waitlist, instructors, room, and opens_at before booking opens. Set fetch.accept_site_terms |
| Playtomic courts | playtomic.com/clubs/02-padel-sport-academy | listing, url | One slot per free court time (3 days by default, 7 at most), with court, length, price, indoor or outdoor. sport (PADEL or TENNIS), duration, court and indoor are optional. Set fetch.accept_site_terms |
| CourtReserve court bookings | app.courtreserve.com/Online/Reservations/Bookings/12345 | listing, url | Not read yet. The court grid's API isn't verified, and most clubs need a member login |
| Zen Planner class schedules | <gym>.sites.zenplanner.com/calendar.cfm | listing, url | One slot per class on the public list, with time, spots left, enrolled/capacity. Set fetch.accept_site_terms. time_zone adds an offset to the times |
| ClassPass studios | classpass.com/studios/<studio> | listing, url | Not available. ClassPass challenges every request from our server, so watch the studio's own booking page (Mindbody, Mariana Tek) |
| CampMinder | system.campminder.com | url, listing | Not available. The portals return an error page and publish no session list (campminder.com is the marketing site) |
| PetDesk | api.petdesk.com | url, listing | Not available. The API answers 403, and no public booking host resolved |
Jobs
Each open job is a listing, and a closed one drops out. Use match: line with words from the title.
| Site | where.sites | match | Gives |
|---|---|---|---|
| Greenhouse | boards.greenhouse.io/discord | listing, line, category | title, link, date posted, location, department, team, remote, req id |
| Lever | jobs.lever.co/spotify | listing, line, category | title, link, date, location, department, team, employment type, remote, salary when published |
| Ashby | jobs.ashbyhq.com/notion | listing, line, category | title, link, date, location, department, team, employment type, remote, salary when published |
| Workday | nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite | listing, line, category | title, link, posting date (from "Posted N days ago"), location, remote, req id. The first 100 jobs per run |
| SmartRecruiters | careers.smartrecruiters.com/BoschGroup | listing, line, category | title, link, date, location, department, employment type, remote. Set fetch.accept_site_terms |
Software, games and status pages
| Site | where.sites | match | Gives |
|---|---|---|---|
| GitHub releases | github.com/nodejs/node | url, line | version (major), status, date |
| GitHub commits | github.com/nodejs/node/commits/main, github.com/nodejs/node/blob/main/doc/api/fs.md, github.com/nodejs/node/tree/main/lib | listing, line, url | New commits on a branch, or only those touching a file or folder, with sha, title, date, author login, message, verified, parents. Put a branch name with a slash in attributes.branch. A path no commit has touched reads as no commits, so check the path |
| GitHub repository stats | github.com/nodejs/node/stargazers | url, listing | One record with stars, forks, watchers, open_issues and size_kb as numbers (so stars >= 10000 or compare_to works), plus default branch, language, license, archived, pushed_at |
| GitHub issues and pull requests | github.com/nodejs/node/issues, github.com/nodejs/node/pulls | listing, line, url | The newest 100 open issues or PRs (closed with attributes.state), with number, author, labels, comments, reactions, thumbs_up, draft, merged_at. Words match the title or labels. attributes.label is sent to GitHub |
| GitHub security advisories | github.com/advisories/npm/lodash, github.com/advisories/pip/jinja2 | listing, line, url | Reviewed advisories for one package, newest first, with GHSA and CVE ids, severity, cvss_score, vulnerable range, patched version, withdrawn. attributes.affects_version and severity are sent to GitHub. A misspelled package reads as one with no advisories |
| OSV.dev | osv.dev/PyPI/jinja2, osv.dev/Go/github.com/gin-gonic/gin, osv.dev/vulnerability/GHSA-35jh-r3h4-6jhm | listing, line, url | Every known vulnerability for a package in any OSV ecosystem, or one by id, with aliases (CVE, GHSA), fixed versions, severity, cvss_score (from CVSS 3.x), modified. attributes.affects_version keeps those affecting that version. Only the first page is read (partial_results says so). No key needed |
| NVD (National Vulnerability Database) | nvd.nist.gov plus words, nvd.nist.gov plus attributes.cpe, nvd.nist.gov/vuln/detail/CVE-2021-44228 | listing, line, url | CVEs published in the last 119 days matching the words or CPE (newest 200), or one CVE, with cvss_score, cvss_version, severity, status, CWEs, kev (on CISA's exploited list). Checked every 2 hours at most |
| npm | npmjs.com/package/react, npmjs.com/package/@types/node | url, line | Versions, with the major as version, semver, prereleases (announced), dist-tags, deprecated, publish date |
| crates.io | crates.io/crates/serde | url, line | The 30 newest versions, with semver, yanked, downloads, publish date |
| Docker Hub | hub.docker.com/_/node, hub.docker.com/r/grafana/grafana | url, line | The 100 most recently pushed tags (or those in attributes.tag), with digest (watch it with changed), version, size, platforms, push time |
| PyPI | pypi.org/project/requests | url, line | Versions, prereleases, yanked, upload date. Set fetch.accept_site_terms (robots.txt disallows the JSON API) |
| Steam | store.steampowered.com/app/1245620 | url, sku | Exact price in the location.country currency, price before discount, discount_percent, coming soon, release date |
| Statuspage (and incident.io) | statuspage:www.cloudflarestatus.com, statuspage:status.clerk.com | listing, url, line, category | One record per component (status: operational, degraded_performance, partial_outage, major_outage, under_maintenance), plus overall state, open incidents and scheduled maintenances. Many Statuspage-hosted pages (*.statuspage.io, githubstatus.com) disallow /api/ in robots.txt and fail with robots_disallowed |
| Instatus | instatus:status.zed.dev, railway.instatus.com | listing, url, line, category | Same records and status values as Statuspage |
Feeds and posts
| Site | where.sites | match | Gives |
|---|---|---|---|
| Any RSS, Atom or JSON feed | blog.rust-lang.org/feed.xml, example.substack.com, or feed:blog.rust-lang.org for a page that links its feed | url, listing, line, category | The newest 100 items, with title, link, published time, author, categories, plain-text summary, and podcast audio file and length |
| Hacker News | news.ycombinator.com, news.ycombinator.com/show, /ask, /jobs, /newest, /user?id=pg | listing, line, category | Stories with title, discussion link, posted time, author, points, comments, and the link and its domain |
| Bluesky | bsky.app/profile/bsky.app, or bsky.app with words in target.query | listing, line, category | The newest 50 posts, with text, link, time, author, likes, reposts, replies |
| Wikipedia page edits | en.wikipedia.org/wiki/Earth (any language) | url, listing | The newest 20 revisions, with editor, time, summary, size, bytes changed |
| YouTube | youtube.com/@GoogleDevelopers, youtube.com/playlist?list=... | url, listing, line, category | The latest 15 videos. Set fetch.accept_site_terms (robots.txt disallows the feed) |
| Stack Exchange questions | The question link, https://stackoverflow.com/questions/<id>/... (any Stack Exchange site) | url, listing | A post with score, answer_count, view_count, accepted, accepted_answer_id, closed, closed_reason, tags |
Books and streams
| Site | where.sites | match | Gives |
|---|---|---|---|
| Open Library | openlibrary.org/authors/OL7234434A (an author), openlibrary.org/works/OL82563W (a book's editions), openlibrary.org/books/OL62447619M or openlibrary.org/isbn/9781551927282 (one edition) | url, listing, line | One release per work or edition, with title, publish date (exact day when known, else publish_year), publishers, format, pages, languages, ISBN as GTIN, scan id. Use { "field": "items", "op": "added" } for new works or editions. Ids only, since robots.txt disallows search. Reads the first 100 works or 50 editions |
| Open Library loans | archive.org/loans, with an archive.org/details/<id> link as the url query | url | Whether a scan can be borrowed now, copies free, waitlist size. Set fetch.accept_site_terms |
| Twitch | twitch.tv/somechannel, or category with a game name | url, category | One profile per channel, with live or offline (in_stock while live, so back_in_stock means it went live), stream title, game, viewers, start time. Needs fetch.accept_site_terms (its API's robots.txt disallows everything). Fails with site_unavailable until Twitch is connected |
Markets
Quotes are exact, so $0.000002808 is compared as that, not rounded to a cent.
| Site | where.sites | match | Gives |
|---|---|---|---|
| DexScreener | dexscreener.com, with a pair or token link https://dexscreener.com/solana/<address> as a url match, or a symbol for line | url, sku (solana:<address>), line | A quote with USD price, price_change_5m/1h/6h/24h, volumes, liquidity_usd, fdv, market_cap. A symbol search can rank a lookalike token first, so use the address when you can |
| OpenSea | opensea.io/collection/pudgypenguins | url, sku (the slug), line | The collection floor as a quote in its currency, plus volume, sales, owners. Fails with site_unavailable until OpenSea is connected |
| Magic Eden (Solana NFTs) | magiceden.io/marketplace/okay_bears | url, listing; sku only for a symbol without underscores | The collection floor as an exact quote in SOL, with listed_count, avg_price_24h and volume_7d (SOL, as numbers), plus the 20 cheapest listings as listing items (exact SOL price, rarity ranks, traits as trait_<type>). listings: false reads only the floor. An unknown symbol reads as an empty collection (nothing_listed), so check it on magiceden.io first. No key needed. Set fetch.accept_site_terms (Magic Eden's API terms limit passing its data on) |
| Etherscan (wallets and gas) | etherscan.io/address/0x..., polygonscan.com/address/0x..., arbiscan.io/address/0x..., etherscan.io/gastracker | listing, url | An address gives a record per recent transaction and ERC-20 transfer (50 of each, newest first), with direction (in, out, self), value in the coin or token as a number (value_wei exact), fee, block, method, failed, token symbol, contract and reputation. direction, token and include narrow it. A gas tracker gives a quote in gwei with safe_gwei, propose_gwei, fast_gwei. Free-tier chains only (Ethereum, Polygon, Arbitrum, Linea and others). Base, BNB, Optimism and Avalanche fail with paid_plan_required. A wallet can belong to a private person, so the address you pick is your responsibility. Needs fetch.accept_site_terms, and anything you show must say Powered by Etherscan.io APIs. Fails with site_unavailable until Etherscan is connected |
| CoinGecko | coingecko.com/en/coins/bitcoin | url, sku (the coin id), line | Coin price in attributes.vs_currency (usd by default), market cap, 24h volume and change. Set fetch.accept_site_terms (robots.txt disallows /api/v3) |
| Kalshi | kalshi.com (a market ticker as the sku query), kalshi.com/markets/kxfed/fed-meeting/kxfed-27apr (an event), kalshi.com/markets/kxfed (a series) | sku, url, listing | One quote per market. price is the last Yes trade in dollars (0.56 = 56%), empty if never traded. yes_bid, yes_ask, no_bid, no_ask, volume, volume_24h, open_interest and strikes are numbers. No key needed |
| Polymarket | polymarket.com/event/<event-slug>, polymarket.com/event/<event>/<market>, or polymarket.com with a market slug (sku) or words (line) | url, sku, line | One quote per outcome (just Yes for a Yes/No market), at the exact price in dollars. best_bid, best_ask, spread, last_trade_price, volume, volume_24h, liquidity, price_change_1d and price_change_1w are numbers. No key needed. Checked every 5 minutes at most, since Polymarket caches for 5 minutes |
| Stock quotes (Finnhub) | finnhub.io with a symbol as the sku query (AAPL) | sku | One quote per U.S. symbol, with USD price, change, change_percent, open, high, low, previous_close, updated_at. Needs fetch.accept_site_terms and a licence that allows passing quotes on. Fails with site_unavailable until Finnhub is connected |
| Crypto exchange listings (Coinbase, Kraken) | coinbase.com/listings, kraken.com/listings, with currencies as the query ("SOL", "usd") | listing | A record per trading pair, with pair, base, quote, status (online, auction, post_only, limit_only, cancel_only, disabled, delisted) and trading. A new listing is items added. No prices |
Rates and economic data
| Site | where.sites | match | Gives |
|---|---|---|---|
| U.S. Treasury Fiscal Data | fiscaldata.treasury.gov/v2/accounting/od/debt_to_penny, or any Fiscal Data API endpoint, with the API's ?filter= if you want one | url, line | One record per row of the newest release (date is its record_date). Numeric columns are numbers under their API names (tot_pub_debt_out_amt, avg_interest_rate_amt, exchange_rate). Pin text columns with target attributes. Releases over one page are read in full, up to 10,000 rows. No key needed |
| FRED | fred.stlouisfed.org/series/UNRATE, or fred.stlouisfed.org with the series id as the sku query | url, sku | One record per series, with value, previous_value, change, change_percent, period, units, frequency, and date as the newest observation. Fails with site_unavailable until FRED is connected |
| BLS | data.bls.gov/timeseries/CUUR0000SA0, or bls.gov with the series id as the sku query | url, sku | One record per series, with value, previous_value, change, change_percent, period_name, and date as the first day of the newest period. Set fetch.accept_site_terms (api.bls.gov's robots.txt disallows everything) |
| European Central Bank | ecb.europa.eu with a currency code (line: USD), data.ecb.europa.eu/data/datasets/EXR/EXR.D.USD.EUR.SP00.A, or FLOW.KEY as the sku query | line, url, sku | A euro reference rate is a quote with the exact price of one euro in that currency. Other series (deposit rate, €STR) are records. Both have value, previous_value, change, change_percent, unit. A + in the key gives several series at once. No key needed |
| U.S. economic release calendars (BLS, BEA) | bls.gov/schedule, bea.gov/schedule, with words as the query ("consumer price index", "GDP") | listing | A release for each scheduled release from the past week onward, with date (Eastern time offset), release_date, release_status (announced, then available once the time passes), agency, release, period. A rescheduled release keeps its key, so date changed catches it. Dates only, not the numbers |
Government
| Site | where.sites | match | Gives |
|---|---|---|---|
| SEC EDGAR filings | sec.gov/AAPL, sec.gov/cik/320193 | listing, url | The newest 100 filings, with form, filing date, archive link and 8-K items |
| Companies House (UK companies) | find-and-update.company-information.service.gov.uk/company/00445790 (add /filing-history or /officers to read only that part) | listing, url | A record for the company (status, accounts and confirmation statement due dates, accounts_overdue, has_insolvency_history, SIC codes, filings_total, active_officers), one per filing (form like CS01 or AA, category, description, processed date and PDF link), and one per officer appointment (name, role, appointed_on, resigned_on, active). form: ["AA"] or category: ["accounts"] narrows the filings. Officers' birth dates and addresses are never read. Watching officers means watching private people, which is your call. Fails with site_unavailable until Companies House is connected (Server keys). Not yet checked against a live answer |
| NHTSA vehicle recalls | nhtsa.gov/recalls/honda/accord/2018 | listing, line, category | Each recall campaign, with component, summary, consequence, remedy and park-it warnings |
| CPSC product recalls | cpsc.gov/recalls, with words in target.query | listing, line, category, url | Recalls from the last 60 days that have every word, with hazard, remedy, units and where they were sold |
| openFDA drug shortages | open.fda.gov/drug/shortages, with the drug in target.query | listing, line, category, sku | Shortage entries per package, with availability, status and reason |
| openFDA recalls | open.fda.gov/drug/enforcement, /food/enforcement, /device/enforcement | listing, line, category | Enforcement reports, with class, status, reason, firm and lots |
| Federal Register | federalregister.gov, federalregister.gov/agencies/<slug> | listing, line, category | Rules, proposed rules and notices, with type, comment deadline and effective date |
| Grants.gov | grants.gov, grants.gov/agencies/HHS-NIH11 | listing, line, category | Open and forecast funding opportunities, with open and close dates |
| City and county meetings (Legistar) | legistar:seattle, oakland.legistar.com | listing, line, category | Upcoming meetings as records (start time with offset, agenda and minutes status, cancelled, agenda PDF), or with words, matching agenda items from the next 12 agendas (matter file, type, status, action). For a city with no known time zone, set attributes.time_zone, or you get the day only and time_zone_unknown. An unknown client name answers 500 and reads as unreachable |
| US wildfires (NIFC) | wildfire:CA, wildfire:2026-CALPF-002271, wildfire with a location | listing, line, category | Current incidents, with name, acres, percent_contained, personnel, county, cause, discovered, contained and out dates, and point of origin. attributes.include_rx adds prescribed burns |
| InciWeb | inciweb | listing, line, category | Incident team updates. Set fetch.accept_site_terms. wildfire:<state> reads NIFC's open incident data without it |
| Road work and closures (WZDx) | roadwork:WA, roadwork:MN, wzdx:https://<feed> | listing, line, category | Work zones and detours, with road, direction, vehicle_impact, start and end, active and mileposts. Narrow with attributes.closed_only, attributes.road or a location radius (distance_miles). On a feed that marks every impact unknown (WSDOT), closed_only adds impact_unknown. A feed that stopped updating adds feed_stale |
| AirNow air quality | airnow:94110, or airnow with a location | listing | Current AQI per pollutant plus an overall record, and forecasts with category and action_day. Fails with site_unavailable until AirNow is connected. Not yet checked against a live answer |
| USGS earthquakes | earthquake.usgs.gov/m4.5, earthquake.usgs.gov/near/34.05,-118.24/200km | listing, line, category | Events in the last 7 days, with magnitude, depth, alert level and tsunami flag |
| CourtListener | courtlistener.com/opinions/scotus, courtlistener.com/dockets/cand | listing, line, category | New opinions or dockets. Fails with site_unavailable until CourtListener is connected |
| NWS weather alerts | weather.gov/alerts/area/KS, weather.gov/alerts/zone/KSZ009 | listing, line, category | Active alerts, with event, severity, urgency, onset and expiry. Set fetch.accept_site_terms |
| USAspending.gov | usaspending.gov, https://www.usaspending.gov/award/CONT_AWD_.../ | listing, line, category, url | Federal contract, IDV or grant awards from the last 90 days (days, up to 365), or one award, with the exact amount as the price, recipient, agency, signed date, place of performance, NAICS and PSC. award_type, agency, sub_agency, recipient, state, naics, psc, min_amount and max_amount narrow the search. Defense contracts show up about 90 days late, so use days: 180 or more for DoD. An IDV's amount is often $0 because the money sits on its orders |
| SAM.gov | sam.gov, https://sam.gov/opp/<notice id>/view | listing, line, category, url | Contract opportunities posted in the last 30 days, with type, agency, NAICS, set-aside, the response deadline and its offset, award amount and awardee. Fails with site_unavailable until SAM.gov is connected. Checked every 3 hours at most, because a personal key gets 10 requests a day |
| USAJOBS | usajobs.gov, https://www.usajobs.gov/search/results/?k=nurse&l=Denver | listing, line, category, url | Open federal job announcements, with agency, place, posted and closing dates, salary range and grades. Fails with site_unavailable until USAJOBS is connected. Set fetch.accept_site_terms |
| NOAA space weather (Kp index) | swpc.noaa.gov/kp | listing | A record per 3-hour step from the last 24 hours onward, with kp (0 to 9, where 5 and up is a storm), status (observed, estimated, predicted) and noaa_scale (G1 to G5) |
| Hospital price files | uchealth.org/cms-hpt.txt (the hospital's site plus /cms-hpt.txt), or that link as a url match | listing, url | A record per location from the hospital's CMS index, with mrf_url (the price file), source_page_url, file_type and ein. It catches new locations and moved files, and doesn't read the prices inside them |
| US House periodic transaction reports | disclosures-clerk.house.gov/ptr, with a member's name or seat as the query ("Pelosi", "CA11") | listing | Not available yet, pending a review of the commercial-use limit in 5 U.S.C. §13107(c). Then a record per stock-trade report, with filing date, PDF link, member, state_district, state and district. The trades are inside the PDF |
Bills and regulations
| Site | where.sites | match | Gives |
|---|---|---|---|
| Congress.gov | congress.gov (the current Congress), congress.gov/bill/119th-congress/senate-bill, or a bill congress.gov/bill/119th-congress/house-bill/1 | url, listing, line | One record per bill, with the bill number, the latest action and its date (with Washington's offset when there's a time), and a status read from that action (introduced, reported, passed, to_executive, became_law, vetoed, failed, in_progress). A single bill adds the public law, sponsor, policy area, and action and cosponsor counts. Lists cover the 250 most recently updated bills, and search words keep titles with every word. Watch latest_action with changed. Fails with site_unavailable until Congress.gov is connected (Server keys) |
| Open States (state bills) | openstates.org/ca (one state), openstates.org with search words (every state), or a bill openstates.org/nc/bills/2025/HB1105 | url, listing, line | One record per state bill, with the latest action and its date, latest passage date, status, state, session, chamber and subjects. A single bill adds the latest action's classification, the action count and the primary sponsor. session, chamber and classification narrow the list. 20 bills per check, latest action first. Fails with site_unavailable until Open States is connected. Not yet checked against a live answer |
| Regulations.gov | regulations.gov with search words (documents open for comment), regulations.gov/docket/FAA-2026-10694 (everything filed in a docket), or regulations.gov/document/<id> | url, listing, line | One record per rulemaking document, with type (Proposed Rule, Rule, Notice...), docket, agency, posted time, and the exact times comments open and close. open_for_comment is true while comments are open, and comments_open_ended when there's no deadline. A single document adds the page count and PDF link. agency, document_type and open_for_comment narrow the search (use [true, false] for both). Fails with site_unavailable until Regulations.gov is connected |
status is our reading of the action's wording, not an official field.
Open data
| Site | where.sites | match | Gives |
|---|---|---|---|
| Socrata open data | socrata:data.cityofnewyork.us/erm2-nwe9, data.cityofchicago.org/resource/ydr8-5enu.json | listing, line, category, url | New rows of any dataset on a Socrata portal (like 311 requests, permits or WARN notices), newest first, with every column as an attribute, plus date and place. |
| ArcGIS feature layers | arcgis:https://maps2.dcgis.dc.gov/dcgis/rest/services/FEEDS/DCRA/FeatureServer/4 | listing, line, category, url | New rows of any public FeatureServer or MapServer layer (like permits, zoning cases or code violations), newest first. A day-only date (like an issue date) stays a day, like 2026-11-02, and other times are exact UTC. Set key_field for layers reloaded every night |
Both take these attributes: where (a SoQL or SQL condition), q or search words (Socrata's full-text search), order, date_field, key_field, title_fields, place_field, location_field (Socrata), time_zone (Socrata, to turn the portal's local times into exact ones), and limit (1 to 1000, 200 by default). location.zip with location.radius keeps rows within that distance.
Research
| Site | where.sites | match | Gives |
|---|---|---|---|
| ClinicalTrials.gov | clinicaltrials.gov with words in target.query, clinicaltrials.gov/search?cond=glioblastoma, clinicaltrials.gov/study/NCT00083512 | listing, url, line, category | Studies, most recently updated first, with status (watch it with changed), recruiting as in_stock, phase, sponsor, enrollment, conditions, interventions, and the nearest site when location.zip and radius are set. condition, sponsor and intervention narrow the search. Set fetch.accept_site_terms, because robots.txt disallows /api/ |
| arXiv category feeds | arxiv.org/list/cs.CL, rss.arxiv.org/rss/cs.CL+cs.AI | listing, line, category | Today's announcement, with title, abstract, authors, categories, announce_type (new, cross, replace) and arxiv_version. Words in target.query keep papers whose title or abstract has every word |
| arXiv search | arxiv.org with words (and attributes.category), or https://arxiv.org/abs/2611.07928 as a url match | listing, url, line, category | The newest 100 submissions, with abstract, authors, categories, DOI, journal ref and version. Set fetch.accept_site_terms, because robots.txt disallows the API |
| PubMed | pubmed.ncbi.nlm.nih.gov with a search in target.query, pubmed.ncbi.nlm.nih.gov/?term=semaglutide | listing, url, line, category | Papers added in the last 30 days, with journal, publication date, DOI, PMCID and article types. Set fetch.accept_site_terms, because robots.txt disallows E-utilities. Without it, use Europe PMC |
| Europe PMC | europepmc.org with a search in target.query, or https://europepmc.org/article/MED/42958891 as a url match | listing, url, line, category | Papers indexed in the last 14 days, including PubMed's (attributes.source: "MED" for PubMed only), with journal, DOI, PMID, open access and cited_by |
| Semantic Scholar citations | semanticscholar.org/DOI:10.1056/NEJMoa2032183, semanticscholar.org/paper/<id> | listing, url, line, category | The 100 most recent papers citing it, with venue, year, DOI and citation_count. |
Names and domains
| Site | where.sites | match | Gives |
|---|---|---|---|
| Username and package names (GitHub, Bluesky, npm) | handle:github/<name>, handle:bsky/<handle>, handle:npm/<package> | listing | One record with available (true when the platform has no such account or package), plus account_type and created_at (GitHub), did (Bluesky) or version (npm). Available means the lookup came back empty, which doesn't guarantee you can register it |
| Domain registration (RDAP) | rdap:example.com | listing | available (true when the registry has no record), expires, days_to_expiry, EPP status, registrar, nameservers, dnssec, pending_delete and redemption. Reserved and premium names can have no record too. A host under a registered name (www.example.com) fails with rdap_not_registrable. TLDs with no RDAP server (.de, .io) fail with rdap_unsupported_tld, and .uk and .xyz fail with robots_disallowed because their robots.txt blocks us |
| TLS certificates for a domain (Cert Spotter) | certs:example.com (include_subdomains: false for the domain alone) | listing | A record per unexpired certificate for the domain and its subdomains, with dns_names, issuer, not_before, not_after, wildcard and revoked. A new certificate is items added. Checked hourly at most. Lookalike domains outside yours aren't searched |
Any page
For a page nothing above reads, we learn a recipe for it or watch it for changes. Both are looser than the rows above, so check the preview first.
| Site | where.sites | match | Gives |
|---|---|---|---|
| Any page, read by a learned recipe | the page with its path: books.toscrape.com/catalogue/category/books/travel_2/index.html | url, listing, line, category | Whatever the page lists (like products, slots or posts), with title, link, exact price, stock or open and taken, and date. The first preview or spawn learns the recipe, and later runs reuse it |
| Any page with several pages (learned recipe) | quotes.toscrape.com/page/1/ | listing, line, category, url | Every item across up to 10 pages. Paging stops at an empty page, a repeated page, a "not found" past the last page, or when the site's own "more" flag says it's done |
| Any store page (learned recipe), with variants | shop.example.com/p/trail-runner | url, listing, line | One result per size or color, each with its own stock, price and SKU, so a size restock works on stores we don't support by platform |
| Any page with a form (learned recipe), with steps | shop.example.com/stores/pickup plus target.attributes.steps and location.zip | listing, line, url | What the page shows after typing a ZIP, picking a party size or clicking. The steps are saved in the recipe and replayed with each watcher's values |
| Any page, watched for changes | change:https://www.mozilla.org/en-US/about/legal/terms/mozilla/ | url | One region with the text, text_hash, length, contains, and what changed last time (last_change, previous_hash, paragraph counts). With seo: true it also has the title, meta description, canonical, robots meta, noindex, h1, script hosts and seo_hash. See Conditions |
| Any PDF, watched for changes | change:https://www.irs.gov/pub/irs-pdf/fw9.pdf | url | One document with text (the first 2,000 characters), text_hash, pages, pdf_title, pdf_created, pdf_modified, contains, the same last_change, and change_excerpt (the changed lines with context, which a judge condition reads wherever they are in the file). If a link starts returning a web page, it keeps its key and adds format_changed |
| A list on any page | change:https://www.cisa.gov/news-events/news with target.attributes.list: "article.c-teaser" | url | One record per item (like a press release, a product or a list entry), keyed by its link or its text, with title, href, position, date from the item's <time>, and contains for each item. Use { "field": "items", "op": "added" } or "removed". After the first read, an empty list is ok and flagged list_empty |
| Any page, after form steps (browser) | change:https://www.weather.gov/ with browser: true, steps and location.zip | url | The page the steps led to, read like any change watch. See Steps and location |
| Any page, from a location (browser) | change:https://www.target.com/p/... with browser: true, cookies, locale, timezone and location.zip or lat/lng | url | The page as a visitor there sees it. Many stores go by IP or their own cookie, not the browser's position, so set the site's location cookie |
A page no listed site covers, given with its path (like example.com/book/haircut), is read with a learned recipe.
A new page's recipe is learned only at preview or spawn, and a spawn that can't learn it is refused with recipe_not_learned.
If the page's layout changes and the recipe fails 3 runs in a row, it's learned again, at most once every 6 hours.
For page changes, set what to read in target.attributes: selector (a CSS selector, the page's main text by default), contains (a phrase or a list), ignore (a CSS selector to drop, like a ticker), strip_noise: false to keep times with seconds, and browser: true for a page that builds its text with scripts.
listis a CSS selector for repeated items, one result each (see the row above). It reads at most 500 items, and more is flaggedlist_truncated.pdf: truereads a link as a PDF. Links ending in .pdf are read as PDFs anyway.seo: trueadds the page's tags (see the row above), which is useful for watching your own site.summarize: trueadds a one or two sentence summary of the latest change, written by an LLM.
A selector or list that matches nothing on the first read fails with selector_not_found and names close selectors that do match. To wait for an element to appear, watch a larger region with contains. Each change also has change_excerpt, the changed lines with a line of context, which judged conditions read. The last version is stored, so a deploy doesn't lose what changed. PDFs are always fetched directly (never through the proxy), at most one request a second per host, and robots.txt is checked on every redirect.
A url match with any of these attributes, or a link ending in .pdf, is a page-change watch even without the change: prefix. Timestamps, relative times like "5 minutes ago" and long tokens are dropped before hashing, so they don't count as a change.
Steps and location
With browser: true, target.attributes.steps runs steps in the page before we read it, one per string:
| Step | Does |
|---|---|
fill <selector> => <value> | Types the value into a field |
select <selector> => <value or label> | Picks an option |
click <selector> | Clicks the element |
press <Key>, press <selector> => <Key> | Presses Enter, Tab, Escape, an arrow key and a few others |
wait <selector>, wait network, wait 2s | Waits for an element, for the network to go quiet, or up to 5 seconds |
End a step with ? to make it optional. {{name}} is filled from target.attributes, and {{zip}} from location.zip. You get at most 10 steps, 5 seconds each and 25 seconds in all (less when the page loads quicker). Steps never type into password, file or card fields. A step that can't be done fails the check with step_failed, naming the step, and a bad step is refused at spawn with bad_steps. If a captcha guards the form and can't be solved (Blocks), the run is blocked.
Each account gets 200 browser runs with steps and 100 captcha solves a day (previews count), reset at 00:00 UTC. Past that, a run with steps is skipped (its check says usage_limit with the reset time), and a captcha is left unsolved (the check says usage_limit, not blocked). Steps that would send a contact, sign-up, registration, subscription, checkout or application form, or post a form after filling more than one field, are refused at spawn with steps_submit_refused. If only the live page shows that, the step fails with the same reason. Search forms, a single field followed by Enter, and GET forms are allowed.
For location, the browser gets the position (the ZIP's center without lat/lng), locale (like "en-US") and timezone. cookies (up to 5, as name=value) are set on the page's host before it loads, like "GuestLocation=48104|42.270|-83.730|MI|US" on Target.
Blocks
We fetch pages directly, and use a residential proxy only after a store blocks us. Sites that wall bots off on purpose (booking, restaurant, ticket and job sites, and every API above) never go through the proxy, because a block there is the site saying no.
When a page we load in a browser shows reCAPTCHA (v2 or v3) or Cloudflare Turnstile, we solve it through a captcha-solving service and carry on, when captcha solving is connected (Server keys). After a solve, your account keeps that site's cookies for 6 hours, so later checks there usually get through without solving again. If the site walls a check anyway, the cookies are dropped, or replaced if that check solves the captcha. hCaptcha is reported as blocked. If solving is limited to reviewed sites, a captcha anywhere else stays unsolved and the check says captcha_not_allowed.
We never fetch a page whose address is on a private or local network (private_address). If a site can't be read 3 runs in a row (it's blocked, unreachable, or the page or subreddit is gone), we send a blocked event saying why, and its conditions stay unknown until it answers again.
robots.txt
We read each host's robots.txt once a day and don't fetch paths it disallows. A check that needs a disallowed path fails with robots_disallowed, and the failing event goes out on that first run.
A host whose robots.txt can't be read (a timeout, a server error, or a bot check in its place) counts as fully disallowed until the next read.
Some sites disallow the paths that hold the data, and a few forbid automated access in their terms. Craigslist's terms forbid automated searching, for one. We turn those sites on one at a time, and watchers on them must set:
"fetch": { "accept_site_terms": true }By setting it, you confirm you're allowed to fetch that site under its terms, and that you, not Snag, are responsible for doing so. Snag doesn't read those terms for you.
If preview or spawn can't use a site, it answers 422 with one of these codes:
| Error | Means |
|---|---|
site_unknown | That site can't be read yet |
site_unavailable | The site blocks automated access, or needs a key that isn't connected yet. The message says which, and the watcher isn't created |
site_cant_match | The site doesn't support that match mode. The error lists the ones it does |
site_not_reviewed | Not read yet, because a check would write to someone else's account, a statute restricts it, or support isn't built |
site_terms_not_accepted | The site needs fetch.accept_site_terms |
Server keys
Some sites need a key we connect on our side. If a site's key isn't connected yet, creating the watcher fails with site_unavailable, or for an optional key, the site reads less (usually at a lower rate limit). The table shows which key each site uses, and it's the list to set if you run the server yourself.
| Variable | For |
|---|---|
REDDIT_CLIENT_ID, REDDIT_CLIENT_SECRET, REDDIT_USER_AGENT | Reddit through its own API (REDDIT_USER_AGENT is optional). Without them, Reddit goes through Bright Data, and without either it isn't read |
BRIGHTDATA_API_KEY, BRIGHTDATA_WEB_UNLOCKER_ZONE, BRIGHTDATA_MONTHLY_RECORDS | Zocdoc (through Bright Data's Web Unlocker, with the zone web_unlocker1 by default) and Reddit when it has no keys of its own. Both draw on one monthly cap, 5,000 by default |
RESY_API_KEY | Resy, optional. Without it, we use the key Resy's widget publishes |
BRIGHTDATA_BROWSERAPI_USERNAME, BRIGHTDATA_BROWSERAPI_PASSWORD | OpenTable, read in a real browser (the full username and password of a Bright Data Browser API zone) |
OPENTABLE_AVAILABILITY_HASH | Optional, for OpenTable's legacy direct read. The browser read doesn't need it |
EVENTBRITE_TOKEN | Eventbrite ticket classes |
TICKETMASTER_API_KEY | Ticketmaster |
SEATGEEK_CLIENT_ID | SeatGeek, after its review |
OPENSEA_API_KEY | OpenSea |
COINGECKO_DEMO_API_KEY | CoinGecko's higher limit |
EBAY_CLIENT_ID, EBAY_CLIENT_SECRET | eBay |
DISCOGS_TOKEN | Discogs' higher limit |
ETSY_API_KEY, ETSY_SHARED_SECRET | Etsy |
OPENFDA_API_KEY | openFDA's higher daily limit |
COURTLISTENER_TOKEN | CourtListener |
GITHUB_TOKEN | A higher limit for every GitHub site (5,000 requests an hour instead of 60, shared), and GitHub name lookups |
NVD_API_KEY | NVD's higher limit |
SOCRATA_APP_TOKEN | Socrata's higher limit |
CONGRESS_API_KEY, REGULATIONS_API_KEY | Congress.gov and Regulations.gov (one free api.data.gov key works for both) |
OPENSTATES_API_KEY | Open States |
SAM_API_KEY | SAM.gov |
USAJOBS_API_KEY, USAJOBS_USER_AGENT_EMAIL | USAJOBS, after its review |
FINNHUB_API_KEY | Stock quotes, after their review |
FRED_API_KEY | FRED |
BLS_API_KEY | BLS's higher limit, after its review |
NCBI_API_KEY | PubMed's higher limit, after its review |
SEMANTIC_SCHOLAR_API_KEY | Semantic Scholar's own limit |
STACKEXCHANGE_KEY | Stack Exchange's higher daily limit |
AIRNOW_API_KEY | AirNow air quality |
COMPANIES_HOUSE_API_KEY | Companies House |
STARTGG_TOKEN | start.gg |
TWITCH_CLIENT_ID, TWITCH_CLIENT_SECRET | Twitch, after its review |
ETHERSCAN_API_KEY | Etherscan, after its review |
AMILIA_API_TOKEN | Amilia (a partner token from Amilia) |
SIGNUPGENIUS_API_KEY | SignUpGenius open slots (the account's user key) |
OPENROUTER_API_KEY, OPENROUTER_MODEL, WATCHER_LLM_MONTHLY_USD, WATCHER_LLM_OWNER_MONTHLY_USD | Learning recipes, judged conditions, page-change summaries and suggestions on failed requests. The server's monthly cap is $20 by default, and each account's is $2 |
WATCHER_JUDGE_CALLS_PER_DAY | How many judged checks a watcher may make a day (20 by default) |
CAPTCHA_PROVIDER, CAPTCHA_API_KEY | Solving captchas, with 2captcha, capsolver or anticaptcha |
WATCHER_CAPTCHA_UNREVIEWED_HOSTS | on (the default) solves captchas on any site, and off solves them only on WATCHER_REVIEWED_HOSTS |
WATCHER_REVIEWED_HOSTS | Reviewed hosts, comma-separated. Steps may submit forms there, and with the switch off, only these hosts get captcha solves |
WATCHER_STEP_RUNS_PER_DAY, WATCHER_CAPTCHA_SOLVES_PER_DAY | Each account's daily caps on browser runs with steps (200) and captcha solves (100). 0 turns it off |
WATCHER_SOLVED_SESSION_HOURS | How long an account reuses a site's cookies after solving a captcha there (6 by default). 0 turns it off |