Auto-pagination: to grab all pages or a subset of pages

[premasagar]

E.g. by gathering the 'next' and 'previous' URLs via the resource's link headers (e.g. GitHub API), or via a selector to the URL in a JSON document (e.g. Lanyrd API) or a DOM selector for a link in an HTML document (e.g. any website with pagination anchor links in the page)

[AaronAcerboni]

What about a paginate: "all" property?

{
  "url": "https://api.github.com/users/dharmafly/repos?per_page=4",
  "type": "json",
  "paginate": "all"
}

[premasagar]

Naming
I'm undecided between paginate and pagination. Leaning towards the latter.

Limiting
Should there be a mechanism to limit the number of pages returned? At the very least, the code should include a check to prevent circular loops by stopping if a duplicate resource is linked - e.g. a website containing page1 as the next parameter of the last page.

Slicing
In the example above, where the url is "https://api.github.com/users/dharmafly/repos?per_page=4", should the first resource be page4, followed by page5 until the end? Or should the whole series, from page1 be loaded? It would be good to have a way to grab a slice of pages - e.g. pages3-5.

Selectors
Is it reasonable to support the use of selectors, as per the examples in the issue's description? If so, how would that look?

Like this?

{
  "url": "https://example.com/things",
  "pagination": "#intro .next",
  "pagination_limit": 3
}

Is there a need to include "prev" resources? I think this is too complicated:

{
  "url": "https://example.com/things",
  "pagination_next": "#intro .next",
  "pagination_prev": "#intro .prev",
  "pagination_limit": 3
}