inline_requests package

Submodules

inline_requests.generator module

class inline_requests.generator.RequestGenerator(callback, **kwargs)[source]

Bases: object

This is the core class that wraps the callback and outputs the requests one by one.

__call__(response)[source]

Main response entry point.

This method calls the callback and wraps the returned generator.

inline_requests.utils module

inline_requests.utils.get_args(method_or_func)[source]

Returns method or function arguments.

Module contents

inline_requests.inline_requests(method_or_func)[source]

A decorator to use coroutine-like spider callbacks.

Example:

class MySpider(Spider):

    @inline_callbacks
    def parse(self, response):
        next_url = response.urjoin('?next')
        try:
            next_resp = yield Request(next_url)
        except Exception as e:
            self.logger.exception("An error occurred.")
            return
        else:
            yield {"next_url": next_resp.url}

You must conform with the following conventions:

  • The decorated method must be a spider method.
  • The decorated method must use the yield keyword or return a generator.
  • The decorated method must accept response as the first argument.
  • The decorated method should yield Request objects without neither callback nor errback set.

If your requests don’t come back to the generator try setting the flag to handle all http statuses:

request.meta['handle_httpstatus_all'] = True