Merge pull request #443 from bollwyvl/providers

refactor providers

README.md

@@ -1,5 +1,6 @@
 # Jupyter Notebook Viewer
-[![Gitter](https://badges.gitter.im/Join Chat.svg)](https://gitter.im/jupyter/nbviewer?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![Gitter](https://badges.gitter.im/Join Chat.svg)][gitter]
+[gitter]: https://gitter.im/jupyter/nbviewer?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
 
 Jupyter nbviewer is the web application behind [The Jupyter Notebook Viewer](http://nbviewer.ipython.org), which is graciously hosted by [Rackspace](https://developer.rackspace.com/?nbviewer=awesome).
 
@@ -47,32 +48,38 @@ With this configured all GitHub API requests will go to you Enterprise instance
 
 ### With Docker
 
-You can build a docker image that uses your local branch
+You can build a docker image that uses your local branch.
 
 
 #### Build
 
 ```shell
-docker build -t nbviewer .
+$ cd <path to repo>
+$ docker build -t nbviewer .
 ```
 
 
 #### Run
 
 ```shell
-docker run -p 8080:8080 nbviewer
+$ cd <path to repo>
+$ docker run -p 8080:8080 nbviewer
 ```
 
 ### With Docker Compose
 
-The Notebook Viewer uses `memcache` in production. To locally try out this
+The Notebook Viewer uses `memcached` in production. To locally try out this
 setup, a [docker-compose](https://docs.docker.com/compose/) configuration is
-provided to easily start/stop the `nbviewer` and `memcache` containers together:
+provided to easily start/stop the `nbviewer` and `memcached` containers 
+together from a your current branch. You will need to install `docker` prior
+to this.
 
 #### Run
 
 ```shell
-docker-compose up
+$ cd <path to repo>
+$ pip install docker-compose
+$ docker-compose up
 ```
 
 
@@ -83,16 +90,18 @@ The Notebook Viewer requires several binary packages to be installed on your sys
 If they are installed, you can install the required Python packages via pip.
 
 ```shell
-pip install -r requirements.txt`
+$ cd <path to repo>
+$ pip install -r requirements.txt
 ```
 
-
 #### Static Assets
 
-Static assets are maintained with `bower` and `less`.
+Static assets are maintained with `bower` and `less` (which require having
+`npm` installed), and the `invoke` python module.
 
 ```shell
 $ cd <path to repo>
+$ pip install -r dev-requirements.txt
 $ invoke bower
 $ invoke less [-d]
 ```
@@ -110,3 +119,67 @@ $ python -m nbviewer --debug --no-cache
 ```
 
 This will automatically relaunch the server if a change is detected on a python file, and not cache any results. You can then just do the modifications you like to the source code and/or the templates then refresh the pages.
+
+
+#### Running the Tests
+
+`nose` is used to run the test suite. The tests currently make calls to
+external APIs such as GitHub, so it is best to use your Github API Token when 
+running:
+
+```shell
+$ cd <path to repo>
+$ pip install -r dev-requirements.txt
+$ GITHUB_API_TOKEN=<your token> python setup.py test
+```
+
+
+## Extending the Notebook Viewer
+### Providers
+Providers are sources of notebooks and directories of notebooks and directories.
+
+`nbviewer` ships with several providers
+- `url`
+- `gist`
+- `github`
+- `local`
+
+#### Writing a new Provider
+There are several already additional providers
+[proposed/requested](/jupyter/nbviewer/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Aprovider). Some providers are more involved than others, and some,
+such as those which would require user authentication, will take some work to
+support properly.
+
+A provider is implemented as a python module, which can expose a few functions:
+
+##### `uri_rewrites`
+If you just need to rewrite URLs (or URIs) of another site/namespace, implement
+`uri_rewrites`, which will allow the front page to transform an arbitrary string
+(usually an URI fragment), escape it correctly, and turn it into a "canonical"
+nbviewer URL. See the [dropbox provider](./providers/dropbox/handlers.py)
+for a simple example of rewriting URLs without using a custom API client.
+
+##### `default_handlers`
+If you need custom logic, such as connecting to an API, implement
+`default_handlers`. See the [github provider](./providers/github/handlers.py)
+for a complex example of providing multiple handlers.
+
+##### Error Handling
+While you _could_ re-implement upstream HTTP error handling, a small
+convenience method is provided for intercepting HTTP errors.
+On a given URL handler that inherits from `BaseHandler`, overload the
+`client_error_message` and re-call it with your message (or `None`). See the
+[gist provider](./providers/gist/handlers.py) for an example of customizing the
+error message.
+
+### Formats
+Formats are ways to present notebooks to the user.
+
+`nbviewer` ships with two providers:
+- `html`
+- `slides`
+
+#### Writing a new Format
+If you'd like to write a new format, open a ticket, or speak up on [gitter][]! 
+We have some work yet to do to support your next big thing in notebook 
+publishing, and we'd love to here from you.

nbviewer/app.py

@@ -25,16 +25,20 @@
 
 from IPython.config import Config
 
-from .handlers import init_handlers, format_providers, LocalFileHandler
+from .handlers import init_handlers, format_handlers
 from .cache import DummyAsyncCache, AsyncMultipartMemcache, MockCache, pylibmc
 from .index import NoSearch, ElasticSearch
 from .formats import configure_formats
 
+from .providers import default_providers, default_rewrites
+from .providers.local import LocalFileHandler
+
 try:
-    from .client import LoggingCurlAsyncHTTPClient as HTTPClientClass
+    from .providers.url.client import LoggingCurlAsyncHTTPClient as HTTPClientClass
 except ImportError:
-    from .client import LoggingSimpleAsyncHTTPClient as HTTPClientClass
-from .github import AsyncGitHubClient
+    from .providers.url.client import LoggingSimpleAsyncHTTPClient as HTTPClientClass
+
+
 from .log import log_request
 from .utils import git_info, ipython_info
 
@@ -81,6 +85,8 @@ def main():
     define("default_format", default="html", help="format to use for legacy / URLs", type=str)
     define("proxy_host", default="", help="The proxy URL.", type=str)
     define("proxy_port", default="", help="The proxy port.", type=int)
+    define("providers", default=default_providers, help="Full dotted package(s) that provide `default_handlers`", type=str, multiple=True, group="provider")
+    define("provider_rewrites", default=default_rewrites, help="Full dotted package(s) that provide `uri_rewrites`", type=str, multiple=True, group="provider")
     tornado.options.parse_command_line()
 
     # NBConvert config
@@ -168,7 +174,6 @@ def main():
     )
     AsyncHTTPClient.configure(HTTPClientClass)
     client = AsyncHTTPClient()
-    github_client = AsyncGitHubClient(client)
 
     # load frontpage sections
     with io.open(options.frontpage, 'r') as f:
@@ -193,9 +198,10 @@ def main():
         jinja2_env=env,
         static_path=static_path,
         client=client,
-        github_client=github_client,
         formats=formats,
         default_format=options.default_format,
+        providers=options.providers,
+        provider_rewrites=options.provider_rewrites,
         config=config,
         index=indexer,
         cache=cache,
@@ -211,14 +217,15 @@ def main():
     )
 
     # handle handlers
-    handlers = init_handlers(formats)
+    handlers = init_handlers(formats, options.providers)
+
     if options.localfiles:
         log.app_log.warning("Serving local notebooks in %s, this can be a security risk", options.localfiles)
         # use absolute or relative paths:
         local_handlers = [(r'/localfile/(.*)', LocalFileHandler)]
         handlers = (
             local_handlers +
-            format_providers(formats, local_handlers) +
+            format_handlers(formats, local_handlers) +
             handlers
         )