Metadata-Version: 1.2
Name: reppy
Version: 0.4.14
Summary: Replacement robots.txt Parser
Home-page: http://github.com/seomoz/reppy
Maintainer: Lindsey Reno
Maintainer-email: lindsey@moz.com
License: MIT
Description: Replaces the built-in robotsparser with a
        RFC-conformant implementation that supports modern robots.txt constructs like
        Sitemaps, Allow, and Crawl-delay. Main features:
        
        - Memoization of fetched robots.txt
        - Expiration taken from the `Expires` header
        - Batch queries
        - Configurable user agent for fetching robots.txt
        - Automatic refetching based on expiration
        
Platform: Posix; MacOS X
Classifier: License :: OSI Approved :: MIT License
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.3
Classifier: Programming Language :: Python :: 3.4
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
