☰

DuckDuckGo URL Collector

Collects URLs from DuckDuckGo with optional site: filtering and rate limiting

作者
Ghosty-Tongue
日安装量
0
总安装量
3
评分
0 0 0
版本
1.3
创建于
2025-01-29
更新于
2025-04-27
大小
6.1 KB
许可证
MIT
适用于

DuckDuckGo URL Collector
A Tampermonkey/Greasemonkey script for comprehensive search result management

🔍 Description

This userscript automatically collects all unique URLs from DuckDuckGo search results, including those hidden behind pagination ("More Results"). Features include:

  • 🌈 RGB-animated collection button
  • ⏱️ Real-time duration timer
  • 🔄 Automatic pagination handling
  • 📋 Duplicate prevention
  • 📥 Automatic TXT file download
  • 📊 Progress logging/notifications
  • 🖥️ Visual feedback during processing
  • 🌐 Optional site-specific filtering
  • ⏸️ Rate limiting breaks

📥 Installation

  1. Install a userscript manager:

  2. Install the script

  3. Confirm installation in your userscript manager

🚀 Usage Guide

  1. Navigate to DuckDuckGo
  2. Perform any search (optional: use site:domain.com for filtering)
  3. Look for the animated 🦆 button (top-right corner)
  4. Click the button to start collection
  5. Wait while the script:
    • Automatically loads all results (clicking "More Results")
    • Collects unique URLs (filtered by site: if specified)
    • Shows progress in console
    • Pauses for 15 seconds every 420 URLs
  6. Automatic download (urls.txt) when complete

⚙️ Technical Features

  • Smart Detection: Uses DOM queries ([data-testid="result"])
  • Efficient Storage: Set object for O(1) lookups
  • Site Filtering: Optional domain restriction with site: searches
  • Rate Limiting: 15-second break after every 420 URLs
  • Visual Feedback:
    • Button animation during processing
    • Desktop notifications
    • Console logging (F12 to view)
    • Break notification banner
  • File Handling: Creates Blob URLs for instant download
  • Performance: 2-second delays between page loads

⚠️ Limitations

  1. Only works on DuckDuckGo search results pages
  2. Dependent on DuckDuckGo's DOM structure
  3. Requires JavaScript enabled
  4. May not work with some privacy extensions
  5. Limited to ~1000 results (DuckDuckGo's pagination limit)

📜 License

MIT Licensed - Feel free to modify/redistribute but please:

  • Maintain author credit
  • Link to original source
  • Share improvements with community