Security News

Cybersecurity news aggregator

📰
INFO News Reddit r/netsec

Crawling the Complete IPv4 Reverse DNS Space

  • What: Researchers crawled the entire IPv4 reverse DNS space
  • Impact: Provides insights into IP geolocation and network infrastructure
Read Full Article →

Published: July 18, 2026 Last Modified: July 19, 2026 Reverse DNS PTR Records Internet-Scale Crawling IP Geolocation Crawling the Complete IPv4 Reverse DNS Space In November 2023, we published a feasibility study that asked a simple question: is it practical to look up the reverse DNS record of every routable IPv4 address? The back-of-the-envelope answer was "probably": roughly 20 small servers querying at full speed for three weeks. That estimate turned out to be pessimistic. The crawler is now in production: every month it sweeps all 3,702,258,432 routable IPv4 addresses, collects 1,039,377,899 PTR records, and finishes in two to four days. The complete result includes every hostname and the exact DNS status of every address that did not resolve, yet compresses into a single artifact of roughly 1 GB . This article explains how the crawl works and, more interestingly, what a billion hostnames are actually good for: hosting detection, VPN and Tor corroboration, connection type inference, and internet-scale geolocation mining. Along the way we survey the current research frontier, from CAIDA's Hoiho to LLM-guided extraction systems like The Aleph , before discussing what is next: faster incremental re-crawls and the IPv6 reverse tree. The raw dataset itself is published as the ipapi.is Reverse DNS Database , and it feeds directly into the ipapi.is IP intelligence API. Download a Real Reverse DNS Sample The curated TSV sample below contains verified PTR records from public resolvers, residential networks, cloud platforms, CDNs and hosting providers. It uses the same two-column ip<TAB>ptr layout as the full database. Download TSV Sample View the Crawler on GitHub 100.34.2.29 pool-100-34-2-29.phlapa.fios.verizon.net 184.27.32.74 a184-27-32-74.deploy.static.akamaitechnologies.com 3.137.190.186 ec2-3-137-190-186.us-east-2.compute.amazonaws.com 88.198.1.1 static.88-198-1-1.clients.your-server.de For the complete monthly dataset, formats and pricing, see the ipapi.is Reverse DNS Database . Table of Contents Why Reverse DNS Matters More in 2026 The Crawl: 3.7 Billion DNS Queries, Politely Throughput, and Why the Next Pass Is Much Faster Storing a Billion Hostnames in One Gigabyte What a Billion PTR Names Actually Look Like From Hostnames to IP Intelligence VPN and Tor: Learning From Ground Truth The Geography Hidden in Hostnames What About IPv6? Limits and Feasibility What We Have Not Built Yet: The rDNS Frontier The Dataset Is a Product Conclusion References Why Reverse DNS Matters More in 2026 A PTR record maps an IP address back to a hostname via the special in-addr.arpa DNS zone. The operator of the address space controls the record, and many operators encode surprisingly rich information in it: $ dig +short -x 100.34.2.29 pool-100-34-2-29.phlapa.fios.verizon.net. This single hostname reveals the ISP (Verizon Fios), the access technology (a residential pool), and even the city ( phlapa = Philadelphia, PA). Multiply that by a billion and you have one of the most information-dense public datasets on the internet. Reverse DNS also stopped being optional for a large class of infrastructure. Since April 2024, Google and Yahoo require bulk email senders to have a valid PTR record with forward-confirmed reverse DNS (FCrDNS: the PTR hostname must resolve back to the same IP), and Microsoft enforced the same for Outlook in May 2025. What used to be a courtesy convention (RFC 1912 already recommended that every internet-reachable host should have a name) is now a hard deliverability requirement, which means PTR data keeps getting better maintained, and more useful to crawl. The Crawl: 3.7 Billion DNS Queries, Politely The open-source crawler is a small Go program deployed on a fleet of 20 cloud servers. The design goals were simple: complete the full sweep in days, be a polite DNS citizen, and record an honest status for every address, not just the ones that resolve. Design Decision Value Why Fleet 20 small cloud VMs (2 vCPUs each) Cheap, disposable, each with its own public IPv4 Resolver Local unbound on every node Public resolvers rate-limit at this volume; recursing directly spreads load across the actual authoritative servers Sharding Interleaved: ip % 20 == shard_id Every shard covers the whole address space evenly, and consecutive IPs of one operator are spread across 20 source IPs, distributing query load by construction Concurrency 800 in-flight lookups per node Proven stable on a 2-vCPU box without stressing the local resolver Timeout / retries 2 seconds / 1 retry Timeouts, not throughput, are the bottleneck of a full sweep Scope All routable IPv4, bogons excluded Reserved ranges ( 10/8 , 127/8 , 100.64/10 , …) are skipped up front Throughput, and Why the Next Pass Is Much Faster In the 2023 feasibility study, a single server pushing everything through 8.8.8.8 managed about 112 lookups per second. The production fleet is far quicker. The full sweep of all 3,702,258,432 routable addresses completes in about three ...

Share this article