Dynamically generate robots.txt

[apfelbox]

Generate it dynamically in a controller.

Add the following snippet to it:

Sitemap: http://example.org/sitemap.xml.gz

[apfelbox]

Also allow to hook into the creation

[apfelbox]

Also support configuring whether to exclude everything (for staging environments).

For example like here https://github.com/ricbra/robots-txt-bundle