What is IDM Crawler?

A specialized auditing engine designed to detect unauthorized marketing tags and validate data layer integrity across enterprise websites.

Quick Start Guide

System Requirements

Installation

Windows Installer (Recommended)

1. Download IDMCrawler-Setup.exe from our website
2. Run the installer and follow the prompts
3. Launch IDM Crawler from your Start Menu or Desktop shortcut
4. Start crawling immediately - browser is bundled with the application

Portable Version

1. Download IDM_Crawler_Portable.zip
2. Extract to any folder (e.g., C:\IDMCrawler)
3. Run IDM_Crawler_Portable.exe
4. The browsers folder must remain in the same directory as the executable
5. Start crawling immediately - no additional setup required

5-Pillar Governance Framework

The engine evaluates every page against these key technical standards:

Unmanaged Tag Detection

IDM Crawler identifies tags firing outside your approved tag management system by:

• Analyzing network requests against 50+ vendor signatures (Google, Meta, Adobe, Tealium, etc.)
• Checking if requests originate from authorized tag managers (GTM, Tealium iQ, Adobe Launch, Segment)
• Flagging hardcoded pixels, rogue scripts, and shadow IT operating without governance
• Detecting duplicate vendor instances loading from different sources

Performance Scoring Algorithm

The performance score (0-100) is calculated using weighted metrics based on industry benchmarks:

Data Layer Validation

IDM Crawler validates data layer implementations across major platforms:

GTM dataLayer

• Checks for array format and proper structure
• Validates page view events (gtm.js, gtm.dom, gtm.load)
• Detects ecommerce objects and transaction data
• Identifies event duplication and race conditions

Tealium utag.data

• Validates tealium_event presence (view, link, etc.)
• Checks for page_name, dom.title, and site section data
• Detects ecommerce variables (_corder, _ctotal, _cprod)

Adobe digitalData

• Basic structure validation
• Custom validation rules available per implementation

Consent Detection

• CMP Presence - Identifies OneTrust, Cookiebot, Tealium Consent Manager, and other CMPs
• Pre-consent Firings - Flags tags that fire before user consent is given
• Consent Category Validation - Coming in v0.10+

Tag Manager Detection

IDM Crawler can detect and validate the following tag management systems:

Analytics & Tracking Vendors

The engine identifies these analytics platforms and their implementation IDs:

Advertising & Marketing Vendors

IDM Crawler detects these advertising and marketing pixels:

Media & Video Platforms

IDM Crawler detects embedded media players and video analytics:

Detection Methodology

IDM Crawler uses multiple techniques to identify vendors with high accuracy:

Unknown Vendor Handling

When IDM Crawler cannot identify a vendor, it still provides valuable information:

• Domain tracking - The requesting domain is always captured
• Loader identification - Whether the script came from a tag manager or was hardcoded
• Blocking status - Whether the script blocks page rendering
• Execution order - When the script loaded relative to other resources

Overview Tab

The dashboard showing your tag governance health at a glance:

• 5 Pillar Scores - Visual representation with letter grades (A-F)
• Active Stack - All detected vendors organized by category (Tag Managers, Analytics, Marketing, etc.)
• Key Metrics - Pages crawled, total scripts found, unmanaged tags count
• Recent Issues - Top priority findings requiring attention

Use this tab to get a quick health check of your tag implementation.

Network Tab

Complete network request waterfall with vendor detection:

• URL - The full request URL
• Domain - Third-party domain (helps identify external vendors)
• Vendor - Detected vendor name (Google Analytics, Meta Pixel, etc.)
• Loader - How the script was loaded (Tag Manager, Hardcoded, etc.)
• Managed - Whether it's governed by an authorized tag manager
• Blocking - Whether the request blocks page rendering
• Status - HTTP status code (200, 204, 404, etc.)

Scripts Tab

All JavaScript resources with execution order analysis:

• Execution Order - Sequential order in which scripts loaded
• Name/Vendor - Identified vendor or "Unknown"
• Loader Type - How the script was initiated (HTML/Parser, Tag Manager, Hardcoded)
• Issue - Any governance issues detected (Hardcoded, Pre-consent, etc.)
• Severity - Issue severity (0-10) with color coding
• URL - Full script URL for debugging

DataLayer Tab

All data layer events captured during the crawl:

• Event Name - The event type (gtm.js, gtm.dom, page_view, etc.)
• Vendor - Which data layer (GTM dataLayer, Tealium utag.data, etc.)
• Data - The complete event payload with all variables
• Page URL - Which page the event fired on (for multi-page crawls)
• Timestamp - When the event occurred

Issues Tab

Aggregated findings organized by pillar for easy prioritization:

• Severity - Color-coded (Red=Critical, Orange=High, Yellow=Medium, Blue=Low)
• Description - Clear explanation of the issue
• Details - Specific technical details and affected resources
• Pages Count - How many pages are affected by this issue
• Example Pages - Sample URLs where the issue was found

Use this tab to prioritize your remediation efforts based on severity and impact.

Role-Based Permission Matrix

IDM Crawler respects your organization's role-based access control. The following permissions apply to all crawler operations:

Role Descriptions

Sync to Cloud Limitations

The "Sync to Cloud" feature is only available to users with Owner, Admin, or Editor roles. This ensures that audit data is only written to your workspace by authorized personnel.

How to Check or Upgrade Your Role

Check Your Current Role

1. Log in to IDM Crawler
2. Select your organization from the dropdown in the left sidebar
3. Your role is indicated next to the organization name (e.g., "Acme Corp (Professional) - Owner")
4. You can also check by looking for the "Sync to Cloud" button after a crawl:
   • If visible → You have write permissions (Owner/Admin/Editor)
   • If hidden → You have read-only permissions (Viewer/Client)

Request a Role Upgrade

1. Contact your organization's Owner or Admin
2. They can change your role in the IDM Web App under Organization Settings → Members
3. After the role is updated, restart IDM Crawler to see the changes

What Data We Collect

IDM Crawler is designed with privacy-first principles. We collect only what's necessary for functionality and troubleshooting.

Privacy Modes

IDM Crawler offers three privacy modes that control what information is written to logs. You can change this anytime via Help → Privacy Settings.

Telemetry & Crash Reports

Anonymous Usage Statistics (Opt-In)

When enabled, we collect completely anonymous usage data to help improve IDM Crawler:

• Feature usage counts (e.g., "Single Page Audit used 10 times")
• Error frequency and types
• Performance metrics (crawl duration, pages per crawl)
• App version and operating system
• NEVER URLs or domain names
• NEVER page content or personal data
• NEVER browsing history outside IDM Crawler

Crash Reports (Opt-Out by Default, Recommended)

When the application crashes, we may receive:

• Error type and stack trace (no personal data)
• App version and operating system
• Privacy mode setting
• NEVER URLs or crawl data

Crash reports help us identify and fix bugs faster. You can disable this in privacy settings.

Log Files (Stored Locally)

IDM Crawler maintains several log files on your computer for troubleshooting. These are stored in %APPDATA%\IDMCrawler\logs\.

Your Privacy Controls

You have complete control over your data at all times:

Legal Compliance

Single Page Audit

1. Launch IDM Crawler
2. Enter the URL you want to audit (e.g., https://example.com)
3. Select "Single Page Audit" from the dropdown
4. Click START
5. Wait for the audit to complete (typically 10-30 seconds)
6. Review results across the 5 tabs

Full Site Crawl

1. Launch IDM Crawler
2. Enter your website's starting URL (usually the homepage)
3. Select "Full Site Crawl" from the dropdown
4. Click START
5. The crawler will discover and analyze pages based on your plan limit (Starter: 50, Professional: 500, Agency: Unlimited)
6. Monitor progress in the progress bar and console output
7. Review aggregated results across all pages in each tab

Plan Limits & Upgrading

The IDM Crawler is completely free for local audits. However, certain features have limits based on your workspace plan:

How to Check Your Current Plan

1. Log in to IDM Crawler
2. Your plan is displayed next to your organization name (e.g., "Acme Corp - Professional")
3. You can also check in the WebApp under Workspace Settings → Billing

How to Upgrade

1. Go to IDM WebApp
2. Navigate to Workspace Settings → Billing
3. Click "Upgrade to Professional" or contact sales for Agency
4. Choose monthly or annual billing (annual saves 17%)
5. Enter payment details (processed securely by LemonSqueezy)

Export Results

1. Complete a crawl (single page or full site)
2. Log in to your IDM Crawler account (free registration required for export). Export is available on all plans including Starter.
3. Navigate to any tab (Overview, Network, Scripts, etc.)
4. Click the Export button in the tab toolbar
5. Choose your preferred format:
   • CSV - For spreadsheet analysis
   • PDF - For sharing with stakeholders and reports
6. Save the file to your computer

Browser Configuration

IDM Crawler includes a complete Chromium browser engine. No additional setup is required!

Browser Management

• Location: %APPDATA%\IDMCrawler\browsers\
• Size: ~300MB
• First Run: The browser is automatically extracted from the application bundle

Clearing Browser Cache

1. Go to File → Browser → Clear Browser Cache
2. Confirm the action
3. The browser will be re-extracted on your next crawl

Interpret Results

Here's how to read and act on your audit results:

Severity Levels

Pillar Score Interpretation

Common Actions by Issue Type

• Hardcoded Tags → Migrate to tag management system
• Pre-consent Firings → Review consent integration and tag firing rules
• Missing Data Layer → Implement GTM, Tealium, or Adobe data layer
• High Blocking Scripts → Add async/defer attributes or move to footer
• Multiple Analytics → Consolidate to single platform

Browser & Firewall

Crawl Issues

Export Issues

What's New

• 5-Pillar Governance Framework - Comprehensive scoring across Tracking, Consent, Data Layer, Performance, and Governance
• Unmanaged Tag Detection - Identifies 50+ vendors including Google, Meta, Adobe, Tealium
• Bundled Browser - Chromium engine included - no separate installation required!
• Performance Impact Scoring - Weighted algorithm measuring third-party domains, tag bloat, blocking scripts, redundant pixels, and analytics overlap
• Data Layer Validation - Supports GTM dataLayer, Tealium utag.data, and Adobe digitalData
• Consent Detection - Identifies CMP presence and pre-consent tag firings
• Single Page & Full Site Crawl - Starter: 50 pages, Professional: 500 pages, Agency: Unlimited
• Export Capabilities - JSON, CSV, and PDF exports for logged-in users
• Modern UI - 5 tabs for detailed analysis (Overview, Network, Scripts, DataLayer, Issues)
• Guest Mode - Crawl unregistered sites with 50-page limit

Known Limitations (MVP)

• Starter plan limited to 50 pages per crawl (upgrade to Professional or Agency for higher limits)
• Team collaboration requires Professional or Agency plan (Starter: 1 user only)
• Exports require free account login
• Windows 64-bit only (Mac/Linux coming)
• Guest mode results cannot be saved to cloud

Coming in v0.10.0

• Custom vendor signature addition
• Detailed consent category validation
• Enhanced ecommerce funnel validation
• Improved PDF report formatting

Planned for Future Releases

• Enhanced consent management validation (Google Consent Mode, OneTrust, Cookiebot)
• Deeper ecommerce and data layer validation tools
• Custom validation rule builder for enterprise implementations
• White-label reporting for Agency plans
• Client portal access for Agency plan clients