PoolManager — urllib3 1.16 documentation

# PoolManager

A pool manager is an abstraction for a collection of ConnectionPools.

If you need to make requests to multiple hosts, then you can use a `PoolManager`, which takes care of maintaining your pools so you don’t have to.

    >>> from urllib3 import PoolManager
    >>> manager = PoolManager(10)
    >>> r = manager.request('GET', 'http://example.com')
    >>> r.headers['server']
    'ECS (iad/182A)'
    >>> r = manager.request('GET', 'http://httpbin.org/')
    >>> r.headers['server']
    'gunicorn/18.0'
    >>> r = manager.request('POST', 'http://httpbin.org/headers')
    >>> r = manager.request('HEAD', 'http://httpbin.org/cookies')
    >>> len(manager.pools)
    2
    >>> conn = manager.connection_from_host('httpbin.org')
    >>> conn.num_requests
    3

A `PoolManager` will create a new ConnectionPool when no ConnectionPools exist with a matching pool key. The pool key is derived using the requested URL and the current values of the `connection_pool_kw` instance variable on `PoolManager`.

The keys in `connection_pool_kw` used when deriving the key are configurable. For example, by default the `my_field` key is not considered.

    >>> from urllib3.poolmanager import PoolManager
    >>> manager = PoolManager(10, my_field='wheat')
    >>> manager.connection_from_url('http://example.com')
    >>> manager.connection_pool_kw['my_field'] = 'barley'
    >>> manager.connection_from_url('http://example.com')
    >>> len(manager.pools)
    1

To make the pool manager create new pools when the value of `my_field` changes, you can define a custom pool key and alter the `key_fn_by_scheme` instance variable on `PoolManager`.

    >>> import functools
    >>> from collections import namedtuple
    >>> from urllib3.poolmanager import PoolManager, HTTPPoolKey
    >>> from urllib3.poolmanager import default_key_normalizer as normalizer
    >>> CustomKey = namedtuple('CustomKey', HTTPPoolKey._fields + ('my_field',))
    >>> manager = PoolManager(10, my_field='wheat')
    >>> manager.key_fn_by_scheme['http'] = functools.partial(normalizer, CustomKey)
    >>> manager.connection_from_url('http://example.com')
    >>> manager.connection_pool_kw['my_field'] = 'barley'
    >>> manager.connection_from_url('http://example.com')
    >>> len(manager.pools)
    2

The API of a `PoolManager` object is similar to that of a ConnectionPool, so they can be passed around interchangeably.

The PoolManager uses a Least Recently Used (LRU) policy for discarding old pools. That is, if you set the PoolManager `num_pools` to 10, then after making requests to 11 or more different hosts, the least recently used pools will be cleaned up eventually.

Cleanup of stale pools does not happen immediately but can be forced when used as a context manager.

    >>> from urllib3 import PoolManager
    >>> with PoolManager(10) as manager:
    ...     r = manager.request('GET', 'http://example.com')
    ...     r = manager.request('GET', 'http://httpbin.org/')
    ...     len(manager.pools)
    ...
    2
    >>> len(manager.pools)
    0

You can read more about the implementation and the various adjustable variables within `RecentlyUsedContainer`.

## API

> class `urllib3.poolmanager.` `PoolManager`(num_pools=10, headers=None, **connection_pool_kw)

>   Allows for arbitrary requests while transparently keeping track of necessary connection pools for you.

>   Parameters: | num_pools – Number of connection pools to cache before discarding the least recently used pool. headers – Headers to include with all requests, unless other headers are given explicitly. **connection_pool_kw – Additional parameters are used to create fresh `urllib3.connectionpool.ConnectionPool` instances.

>   Example:

>       >>> manager = PoolManager(num_pools=2)
>       >>> r = manager.request('GET', 'http://google.com/')
>       >>> r = manager.request('GET', 'http://google.com/mail')
>       >>> r = manager.request('GET', 'http://yahoo.com/')
>       >>> len(manager.pools)
>       2

>   `clear`()

>     Empty our store of pools and direct them all to close.

>     This will not affect in-flight connections, but they will not be re-used after completion.

>   `connection_from_context`(request_context)

>     Get a `ConnectionPool` based on the request context.

>     `request_context` must at least contain the `scheme` key and its value must be a key in `key_fn_by_scheme` instance variable.

>   `connection_from_host`(host, port=None, scheme='http')

>     Get a `ConnectionPool` based on the host, port, and scheme.

>     If `port` isn’t given, it will be derived from the `scheme` using `urllib3.connectionpool.port_by_scheme`.

>   `connection_from_pool_key`(pool_key)

>     Get a `ConnectionPool` based on the provided pool key.

>     `pool_key` should be a namedtuple that only contains immutable objects. At a minimum it must have the `scheme`, `host`, and `port` fields.

>   `connection_from_url`(url)

>     Similar to `urllib3.connectionpool.connection_from_url()` but doesn’t pass any additional parameters to the `urllib3.connectionpool.ConnectionPool` constructor.

>     Additional parameters are taken from the `PoolManager` constructor.

>   `request`(method, url, fields=None, headers=None, **urlopen_kw)

>     Make a request using `urlopen()` with the appropriate encoding of `fields` based on the `method` used.

>     This is a convenience method that requires the least amount of manual effort. It can be used in most situations, while still having the option to drop down to more specific methods when necessary, such as `request_encode_url()`, `request_encode_body()`, or even the lowest level `urlopen()`.

>   `request_encode_body`(method, url, fields=None, headers=None, encode_multipart=True, multipart_boundary=None, **urlopen_kw)

>     Make a request using `urlopen()` with the `fields` encoded in the body. This is useful for request methods like POST, PUT, PATCH, etc.

>     When `encode_multipart=True` (default), then `urllib3.filepost.encode_multipart_formdata()` is used to encode the payload with the appropriate content type. Otherwise `urllib.urlencode()` is used with the ‘application/x-www-form-urlencoded’ content type.

>     Multipart encoding must be used when posting files, and it’s reasonably safe to use it in other times too. However, it may break request signing, such as with OAuth.

>     Supports an optional `fields` parameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:

>         fields = {
>             'foo': 'bar',
>             'fakefile': ('foofile.txt', 'contents of foofile'),
>             'realfile': ('barfile.txt', open('realfile').read()),
>             'typedfile': ('bazfile.bin', open('bazfile').read(),
>                           'image/jpeg'),
>             'nonamefile': 'contents of nonamefile field',
>         }

>     When uploading a file, providing a filename (the first parameter of the tuple) is optional but recommended to best mimick behavior of browsers.

>     Note that if `headers` are supplied, the ‘Content-Type’ header will be overwritten because it depends on the dynamic random boundary string which is used to compose the body of the request. The random boundary string can be explicitly set with the `multipart_boundary` parameter.

>   `request_encode_url`(method, url, fields=None, headers=None, **urlopen_kw)

>     Make a request using `urlopen()` with the `fields` encoded in the url. This is useful for request methods like GET, HEAD, DELETE, etc.

>   `urlopen`(method, url, redirect=True, **kw)

>     Same as `urllib3.connectionpool.HTTPConnectionPool.urlopen()` with custom cross-host redirect logic and only sends the request-uri portion of the `url`.

>     The given `url` parameter must be absolute, such that an appropriate `urllib3.connectionpool.ConnectionPool` can be chosen for it.

> class `urllib3.poolmanager.` `BasePoolKey`(scheme, host, port)
>   `count`(value) → integer -- return number of occurrences of value
>   `host`

>     Alias for field number 1

>   `index`(value[, start[, stop]]) → integer -- return first index of value.

>     Raises ValueError if the value is not present.

>   `port`

>     Alias for field number 2

>   `scheme`

>     Alias for field number 0

> class `urllib3.poolmanager.` `HTTPPoolKey`(scheme, host, port, timeout, retries, strict, block, source_address)
>   `block`

>     Alias for field number 6

>   `count`(value) → integer -- return number of occurrences of value
>   `host`

>     Alias for field number 1

>   `index`(value[, start[, stop]]) → integer -- return first index of value.

>     Raises ValueError if the value is not present.

>   `port`

>     Alias for field number 2

>   `retries`

>     Alias for field number 4

>   `scheme`

>     Alias for field number 0

>   `source_address`

>     Alias for field number 7

>   `strict`

>     Alias for field number 5

>   `timeout`

>     Alias for field number 3

> class `urllib3.poolmanager.` `HTTPSPoolKey`(scheme, host, port, timeout, retries, strict, block, source_address, key_file, cert_file, cert_reqs, ca_certs, ssl_version, ca_cert_dir)
>   `block`

>     Alias for field number 6

>   `ca_cert_dir`

>     Alias for field number 13

>   `ca_certs`

>     Alias for field number 11

>   `cert_file`

>     Alias for field number 9

>   `cert_reqs`

>     Alias for field number 10

>   `count`(value) → integer -- return number of occurrences of value
>   `host`

>     Alias for field number 1

>   `index`(value[, start[, stop]]) → integer -- return first index of value.

>     Raises ValueError if the value is not present.

>   `key_file`

>     Alias for field number 8

>   `port`

>     Alias for field number 2

>   `retries`

>     Alias for field number 4

>   `scheme`

>     Alias for field number 0

>   `source_address`

>     Alias for field number 7

>   `ssl_version`

>     Alias for field number 12

>   `strict`

>     Alias for field number 5

>   `timeout`

>     Alias for field number 3

# ProxyManager

`ProxyManager` is an HTTP proxy-aware subclass of `PoolManager`. It produces a single `HTTPConnectionPool` instance for all HTTP connections and individual per-server:port `HTTPSConnectionPool` instances for tunnelled HTTPS connections.

Example using proxy authentication:

    >>> headers = urllib3.make_headers(proxy_basic_auth='myusername:mypassword')
    >>> proxy = urllib3.ProxyManager('http://localhost:3128', proxy_headers=headers)
    >>> r = proxy.request('GET', 'http://example.com/')
    >>> r.status
    200

## API

> class `urllib3.poolmanager.` `ProxyManager`(proxy_url, num_pools=10, headers=None, proxy_headers=None, **connection_pool_kw)

>   Behaves just like `PoolManager`, but sends all requests through the defined proxy, using the CONNECT method for HTTPS URLs.

>   Parameters: | proxy_url – The URL of the proxy to be used. proxy_headers – A dictionary contaning headers that will be sent to the proxy. In case of HTTP they are being sent with each request, while in the HTTPS/CONNECT case they are sent only once. Could be used for proxy authentication.

>   Example:

>         >>> proxy = urllib3.ProxyManager('http://localhost:3128/')
>         >>> r1 = proxy.request('GET', 'http://google.com/')
>         >>> r2 = proxy.request('GET', 'http://httpbin.org/')
>         >>> len(proxy.pools)
>         1
>         >>> r3 = proxy.request('GET', 'https://httpbin.org/')
>         >>> r4 = proxy.request('GET', 'https://twitter.com/')
>         >>> len(proxy.pools)
>         3