Metadata-Version: 2.4
Name: tidehunter
Version: 1.0.1
Summary: HTTP streaming with accurate flow control
Home-page: https://github.com/woozyking/tidehunter
Author: Runzhou Li (Leo)
Author-email: runzhou.li@gmail.com
License: Copyright (c) 2014 Runzhou Li
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in
        all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
        THE SOFTWARE.
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.2
Classifier: Programming Language :: Python :: 3.3
License-File: LICENSE
Requires-Dist: requests>=2.6.0
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: summary

tidehunter
==========

HTTP streaming with accurate flow control

Master branch: |Build Status|

**NOTE**: Not backward compatible with 0.x since 1.x.

Highlights
----------

-  Consumption limits, total control over your stream quota just on the
   client side.
-  Instant on/off switch and accurate consumption counter. Best used
   with `techies <https://github.com/woozyking/techies>`__.
-  Queue interface for scalable stream data consumption. Best used with
   `techies <https://github.com/woozyking/techies>`__.
-  Core mechanisms based on the solid
   `requests <https://github.com/kennethreitz/requests>`__ library,
   inherits all its goodness.

Installation
------------

.. code:: bash

    $ pip install tidehunter
    $ pip install tidehunter --upgrade

Usage
-----

Example 1 (with limit):
~~~~~~~~~~~~~~~~~~~~~~~

**NOTE**: when no external ``Queue`` or ``StateCounter`` supplied,
``Hunter`` uses the Python standard ``Queue`` and the builtin
``SimpleStateCounter`` respectively, which are usually enough for single
process designs and other simple cases.

.. code:: python

    from tidehunter import Hunter

    # The Hunter!
    h = Hunter(url='https://httpbin.org/stream/20')

    # Start streaming
    h.tide_on(limit=5)

    # Consume the data which should be in the data queue now
    while h.q.qsize():
        print(h.q.get())  # profit x 5

    # You can re-use the same Hunter object, with a difference limit
    r = h.tide_on(limit=1)  # this time we only want one record

    assert h.q.qsize() == 1  # or else there's a bug, create an issue!

    print(h.q.get())  # more profit

    # r is actually just a requests.Response object
    print(r.headers)
    print(r.status_code)
    # ... read up on requests library for more information

Example 2 (without limit):
~~~~~~~~~~~~~~~~~~~~~~~~~~

**NOTE**: this example uses ``techies`` and therefore requires Redis
installed.

Assume you have a process running the following code:

.. code:: python

    from techies import StateCounter, Queue
    from tidehunter import Hunter

    # The data queue
    q = Queue(key='demo_q', host='localhost', port=6379, db=0)

    # The state machine and record counter (state counter)
    sc = StateCounter(key='demo_sc', host='localhost', port=6379, db=0)

    # The Hunter!
    h = Hunter(url='SOME_ENDLESS_STREAM_LIKE_TWITTER_FIREHOSE', q=q, sc=sc)

    # Start streaming, FOREVA
    h.tide_on()

Then you delegate the flow control and data consumption to another/many
other processes such as:

.. code:: python

    from techies import StateCounter, Queue

    # The key is to have the SAME state counter
    sc = StateCounter(key='demo_sc', host='localhost', port=6379, db=0)

    # And the SAME data queue
    q = Queue(key='demo_q', host='localhost', port=6379, db=0)

    while sc.started:
        data = q.get()  # dequeue and
        # ...do something with data

        if SHT_HITS_THE_FAN:
            sc.stop()  # instant off switch
            # end of this loop, as well as the streaming process from above

    # If needed
    q.clear()
    sc.clear()

Example 3 (OAuth with Twitter Sample Firehose):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**NOTE**: this example requires ``requests_oauthlib``

.. code:: python

    import os
    import json
    from requests_oauthlib import OAuth1
    from tidehunter import Hunter

    url = 'https://stream.twitter.com/1.1/statuses/sample.json'
    auth = OAuth1(
        os.environ['TWITTER_CONSUMER_KEY'],
        os.environ['TWITTER_CONSUMER_SECRET'],
        os.environ['TWITTER_TOKEN_KEY'],
        os.environ['TWITTER_TOKEN_SECRET']
    )
    h = Hunter(url=url, q=q, auth=auth)
    r = h.tide_on(5)  # let's just get 5 for now

    print(r.status_code)
    print('')

    while h.q.qsize():
        print(json.loads(h.q.get()))
        print('')

You can find other authentications on `this requests
doc <http://docs.python-requests.org/en/latest/user/authentication/>`__.
In short, all you have to do is to pass the desired ``auth`` parameter
to ``Hunter``, like what you would do with ``requests``.

Test (Unit Tests)
=================

.. code:: bash

    $ pip install -r requirements.txt
    $ pip install -r test_requirements.txt
    $ nosetests --with-coverage --cover-package=tidehunter

License
=======

The MIT License (MIT). See the full
`LICENSE <https://github.com/woozyking/tidehunter/blob/master/LICENSE>`__.

.. |Build Status| image:: https://travis-ci.org/woozyking/tidehunter.png?branch=master
   :target: https://travis-ci.org/woozyking/tidehunter

Contributors
------------

-  `Runzhou Li (Leo) <https://github.com/woozyking>`__

Changelog
---------

1.0.1 (2015-04-17)
~~~~~~~~~~~~~~~~~~

-  **Breaking Change**: ``tidehunter.SimpleStateCounter`` updated to
   reflect ```techies`` <https://github.com/woozyking/techies>`__ 0.2.0
   changes on ``StateCounter``. Consequently ``techies`` must be updated
   to 0.2.0+ if you wish to use ``tidehunter`` 1.0.1+

1.0.0 (2014-01-22)
~~~~~~~~~~~~~~~~~~

-  Moved codebase of ``Queue``, ``StateCounter`` to
   `techies <https://github.com/woozyking/techies>`__. It's recommended
   to use ``techies`` together with ``tidehunter``, but not always
   required, and therefore not a dependency of ``tidehunter``
-  Added ``tidehunter.SimpleStateCounter`` to be used when no other
   state counter provided. It's a pure in-process implementation and
   therefore cannot be accessed by other processes
-  You can now do ``from tidehunter import Hunter`` instead of
   ``from tidehunter.stream import Hunter``
-  Replaced ``PycURL`` with
   `requests <https://github.com/kennethreitz/requests>`__. Some of the
   benefits:

   -  Straight Python 2/3 support
   -  Much cleaner implementation
   -  Further delegation of `various authentications
      support <http://docs.python-requests.org/en/latest/user/authentication/>`__
      to ``requests`` itself

0.1.9 (2013-12-24)
~~~~~~~~~~~~~~~~~~

-  ``PyCurl`` and ``Redis`` Python libraries bumped to the latest
   versions.
-  ``Queue`` now is **almost** Python Queue compatible (in a complaint
   free fashion), with the exception of ``Queue.full`` which always
   returns ``False``; ``Queue.task_done`` and ``Queue.join`` do nothing.
-  NEW: Both ``Queue`` and ``StateCounter`` now have a ``clear`` method
   which performs a Redis ``DEL`` command on the said key and
   reinitialize based on each class's ``initialize`` method.

0.1.8 (2013-10-02)
~~~~~~~~~~~~~~~~~~

-  Added alias methods ``put_nowait()`` and ``get_nowait()`` and other
   place holders to map the Python built-in Queue interfaces.
-  Added ``rstgen`` shell script for Markdown to reStructuredText. To
   use, run ``$ source rstgen`` in the root folder.
-  Credentials involved in unit tests and demo are now using environment
   variables.

0.1.7 (2013-07-22)
~~~~~~~~~~~~~~~~~~

-  Massive update to README.rst
-  Fixed PyPi rendering of long description.

0.1.5 (2013-07-22)
~~~~~~~~~~~~~~~~~~

-  NEW: ``Hunter.tide_on()`` now accepts an optional limit parameter for
   on the fly limit adjustment. The adjustment is not permanent, meaning
   if you want to reuse the same Hunter object, the old limit (or
   default None) is in effect.
-  Fixed a potential issue of Hunter puts in more records than desired
   limit.
-  Added temp Basic Auth test case (no stream, need to find a better
   source).

0.1.3 (2013-07-13)
~~~~~~~~~~~~~~~~~~

-  Use the great httpbin.org (by Kenneth Reitz) for unit test now.
-  Auth (oauth or basic) is no longer required, as long as the target
   stream server supports access without auth.

0.1.2 (2013-07-12)
~~~~~~~~~~~~~~~~~~

-  Include CHANGES (changelog) to be shown on PyPi.
-  Use with statement to open files for setup.py.
-  Added the first
   `demo <https://github.com/amoa/tidehunter/tree/master/demo>`__.

0.1.1 (2013-07-12)
~~~~~~~~~~~~~~~~~~

-  Clean up setup.py to ensure requirements are installed/updated.

0.1.0 (2013-07-12)
~~~~~~~~~~~~~~~~~~

-  Initial release
