Reference

inline_requests.inline_requests(method_or_func)

A decorator to use coroutine-like spider callbacks.

Example:

class MySpider(Spider):

    @inline_callbacks
    def parse(self, response):
        next_url = response.urjoin('?next')
        try:
            next_resp = yield Request(next_url)
        except Exception as e:
            self.logger.exception("An error occurred.")
            return
        else:
            yield {"next_url": next_resp.url}

You must conform with the following conventions:

• The decorated method must be a spider method.
• The decorated method must use the yield keyword or return a generator.
• The decorated method must accept response as the first argument.
• The decorated method should yield Request objects without neither callback nor errback set.

If your requests don’t come back to the generator try setting the flag to handle all http statuses:

request.meta['handle_httpstatus_all'] = True