0.90.9.1

Author: jprante

README.rst

@@ -5,27 +5,31 @@ Image by `DaPino <http://www.iconarchive.com/show/fishing-equipment-icons-by-dap
 Elasticsearch Knapsack Plugin
 =============================
 
-Knapsack is an index export/import plugin for `Elasticsearch <http://github.com/elasticsearch/elasticsearch>`_.
+Knapsack is an export/import plugin for `Elasticsearch <http://github.com/elasticsearch/elasticsearch>`_.
 
-It uses tar archive format and gzip compression for input/output.
+It uses archive formats (tar, zip, cpio) and compression algorithms (gzip, bzip2, lzf, xz) for transfer.
+
+A direct copy of indexes or index types, or any search results with stored fields is also supported.
+
+Optionally, you can transfer archives to Amazon S3.
 
 Installation
 ------------
 
-Current version of the plugin is **2.1.5** (Nov 6, 2013)
-
 .. image:: https://travis-ci.org/jprante/elasticsearch-knapsack.png
 
 Prerequisites::
 
-  Elasticsearch 0.90.5+
+  Elasticsearch 0.90+
 
-=============  =========  =================  ===========================================================
-ES version     Plugin     Release date       Command
--------------  ---------  -----------------  -----------------------------------------------------------
-0.90.5         **2.1.4**  Oct 28, 2013       ./bin/plugin --install knapsack --url http://bit.ly/1ipne90
-0.90.6         **2.1.5**  Nov 6, 2013        ./bin/plugin --install knapsack --url http://bit.ly/17cn710
-=============  =========  =================  ===========================================================
+=============  =================  =================  ===========================================================
+ES version     Plugin             Release date       Command
+-------------  -----------------  -----------------  -----------------------------------------------------------
+0.90.9         0.90.9.1           Jan 9, 2013        ./bin/plugin --install knapsack --url http://bit.ly/1e81hwh
+0.90.9         0.90.9.1 (S3)      Jan 9, 2013        ./bin/plugin --install knapsack --url http://bit.ly/K8QwOJ
+=============  =================  =================  ===========================================================
+
+The S3 version includes Amazon AWS API support, it can optionally transfer archives to S3.
 
 Do not forget to restart the node after installation.
 
@@ -37,93 +41,180 @@ The Maven project site is available at `Github <http://jprante.github.io/elastic
 Binaries
 --------
 
-Binaries are available at `Bintray <https://bintray.com/pkg/show/general/jprante/elasticsearch-plugins/elasticsearch-knapsack>`_
+Binaries (also older versions) are available at `Bintray <https://bintray.com/pkg/show/general/jprante/elasticsearch-plugins/elasticsearch-knapsack>`_
+
+Overview
+========
+
+.. image:: ../../../elasticsearch-knapsack/raw/master/src/site/resources/knapsack-diagram.png
 
-Documentation
-=============
 
-Note: you must have the _source field enabled, otherwise the Knapsack export will not work.
+Example
+=======
 
 Let's assume a simple index::
 
    curl -XDELETE localhost:9200/test
    curl -XPUT localhost:9200/test/test/1 -d '{"key":"value 1"}'
    curl -XPUT localhost:9200/test/test/2 -d '{"key":"value 2"}'
 
-Exporting
----------
+Exporting to archive
+--------------------
 
 You can export this Elasticsearch index with::
 
    curl -XPOST localhost:9200/test/test/_export
+   {"running":true,"mode":"export","archive":"tar","path":"file:test_test.tar.gz"}
 
 The result is a file in the Elasticsearch folder::
 
-   -rw-r--r--   1 joerg  staff    296  9 Dez 14:56 test_test.tar.gz
+   -rw-r--r--  1 es  staff  341  8 Jan 22:25 test_test.tar.gz
    
-Check with tar utility, the settings and the mapping is also exported::   
+Check with tar utility, the settings and the mapping is also exported::
 
-   tar ztvf test_test.tar.gz 
-   -rw-r--r--  0 0      0         116  9 Dez 14:56 test/_settings
-   -rw-r--r--  0 0      0          49  9 Dez 14:56 test/test/_mapping
-   -rw-r--r--  0 0      0          17  9 Dez 14:56 test/test/1
-   -rw-r--r--  0 0      0          17  9 Dez 14:56 test/test/2
+    tar ztvf test_test.tar.gz
+    ----------  0 es     0         132  8 Jan 22:25 test/_settings/null/null
+    ----------  0 es     0          49  8 Jan 22:25 test/test/_mapping/null
+    ----------  0 es     0          17  8 Jan 22:25 test/test/2/_source
+    ----------  0 es     0          17  8 Jan 22:25 test/test/1/_source
 
-Also, you can export with::
+Also, you can export a whole index with::
 
    curl -XPOST localhost:9200/test/_export
 
-with the result file test.tar.gz, or even all data with::
+with the result file test.tar.gz, or even all cluster indices with::
 
-   curl -XPOST localhost:9200/_export
+   curl -XPOST 'localhost:9200/_export'
 
-with the result file _all.tar.gz
+to the file _all.tar.gz
 
-Importing
----------
+By default, the archive format is `tar` with compression `gz` (gzip).
 
-You can import the file with::
+You can also export to `zip` or `cpio` archive or use another compression scheme.
+Available are `bz2` (bzip2), `xz` (Xz), or `lzf` (LZF)
 
-   curl -XPOST localhost:9200/test/test/_import
+Export search results
+----------------------
 
-Be sure that the index does not exist. You must delete an index by hand. Knapsack does not delete or overwrite data.
+You can add a query to the `_export` endpoint just like you would do for searching in Elasticsearch::
 
-You can import the file to a new index with renaming your file to test2_test2.tar.gz and executing the import command::
+   curl -XPOST 'localhost:9200/test/test/_export' -d '{
+       "query" : {
+           "match" : {
+               "myfield" : "myvalue"
+           }
+       },
+       "fields" : [ "_parent", "_source" ]
+   }'
 
-   mv test_test.tar.gz test2_test2.tar.gz
-   curl -XPOST localhost:9200/test2/test2/_import
+Export to an archive with a given path name
+-------------------------------------------
 
-and check you have copied the data to a new index with::
+You can configure an archive path with the parameter `path`
 
-   curl -XGET localhost:9200/test2/test2/1
-   {"_index":"test2","_type":"test2","_id":"1","_version":1,"exists":true, "_source" : {"key":"value 1"}}
+    curl -XPOST 'localhost:9200/test/_export?path=/tmp/myarchive.zip'
 
+If ELasticsearch can not write an archive to the path, an error message will appear
+and no export will take place.
 
-State
------
+Renaming indexes and index types
+--------------------------------
 
-While exports or imports or running, you can check the state with::
+You can rename indexes and index types by adding a `map` parameter that contains a JSON
+object with old and new index (and index/type) names::
 
-    curl -XGET localhost:9200/_export/state
+    curl -XPOST 'localhost:9200/test/type/_export?map=\{"test":"testcopy","test/type":"testcopy/typecopy"\}'
 
-or::
+Copy to local or remote cluster
+-------------------------------
 
-    curl -XGET localhost:9200/_import/state
+If your requirement is not saving data to an archive at all, but only copying, Knapsack is your friend.
 
+You can copy an index in the local cluster or to a remote cluster with the `_export/copy` endpoint.
+Preconditions are: you have the same Java JVM version and the same Elasticsearch version.
+
+Example for a local cluster copy of the index `test`::
+
+    curl -XPOST 'localhost:9200/test/_export/copy?map=\{"test":"testcopy"\}'
+
+Example for a remote cluster copy of the index ``test by using the parameters `cluster`, `host`, and `port`::
+
+    curl -XPOST 'localhost:9200/test/_export/copy?&cluster=remote&host=127.0.0.1&port=9201'
+
+This is a complete example that illustrates how to filter an index by timestamp and copy this part to
+another index::
+
+    curl -XDELETE 'localhost:9200/test'
+    curl -XDELETE 'localhost:9200/testcopy'
+    curl -XPUT 'localhost:9200/test/' -d '
+    {
+        "mappings" : {
+            "_default_": {
+                "_timestamp" : { "enabled" : true, "store" : true, "path" : "date" }
+            }
+        }
+    }
+    '
+    curl -XPUT 'localhost:9200/test/doc/1' -d '
+    {
+        "date" : "2014-01-01T00:00:00",
+        "sentence" : "Hi!",
+        "value" : 1
+    }
+    '
+    curl -XPUT 'localhost:9200/test/doc/2' -d '
+    {
+        "date" : "2014-01-02T00:00:00",
+        "sentence" : "Hello World!",
+        "value" : 2
+    }
+    '
+    curl -XPUT 'localhost:9200/test/doc/3' -d '
+    {
+        "date" : "2014-01-03T00:00:00",
+        "sentence" : "Welcome!",
+        "value" : 3
+    }
+    '
+    curl 'localhost:9200/test/_refresh'
+    curl -XPOST 'localhost:9200/test/_export/copy?map=\{"test":"testcopy"\}' -d '
+    {
+        "fields" : [ "_timestamp", "_source" ],
+        "query" : {
+             "filtered" : {
+                 "query" : {
+                     "match_all" : {
+                     }
+                 },
+                 "filter" : {
+                    "range": {
+                       "_timestamp" : {
+                           "from" : "2014-01-02"
+                       }
+                    }
+                 }
+             }
+         }
+    }
+    '
+    curl '0:9200/test/_search?fields=_timestamp&pretty'
+    # wait for bulk flush interval
+    sleep 10
+    curl '0:9200/testcopy/_search?fields=_timestamp&pretty'
 
-Choosing a different location
------------------------------
+Import
+------
 
-With the ``target`` parameter, you can choose a path and alternative name for your tar archive. Example::
+You can import the file with::
 
-   curl -XPOST 'localhost:9200/_export?target=/big/space/archive.tar.gz'
+   curl -XPOST 'localhost:9200/test/test/_import'
 
-Compression
------------
+Knapsack does not delete or overwrite data by default.
+But ou can use the parameter `createIndex` with the value `false` to allow indexing to indexes that exist.
 
-You can select a ``.tar.gz``, ``.tar.bz2``, or ``.tar.xz`` suffix for the corresponding compression algorithm. Example::
+When importing, you can map your indexes or index/types to your favorite ones.
 
-   curl -XPOST 'localhost:9200/_export?target=/my/archive.tar.bz2'
+    curl -XPOST 'localhost:9200/test/_import?map=\{"test":"testcopy"\}'
 
 Modifying settings and mappings
 -------------------------------
@@ -176,15 +267,61 @@ The result is::
     }
   }
 
+Transferring archives to Amazon S3
+----------------------------------
+
+By using special plugin releases including the Amazon AWS S3 API, you can optionally transfer archives
+to S3 or fetch one before importing. You can use the endpoints `_export/s3` and _import/s3` for that.
+
+Export example::
+
+    curl -XPOST 'localhost:9200/test/_export/s3?uri=s3://accesskey:secretkey@awshostname&bucketName=mybucket&key=mykey'
+
+Import example::
+
+    curl -XPOST 'localhost:9200/test/_import/s3?uri=s3://accesskey:secretkey@awshostname&bucketName=mybucket&key=mykey'
+
+Note, the file name which is used for downloading from S3 is `mybucket/mykey` and the directory will be created
+if it does not exist.
+
+
+Check the state of running import/export
+----------------------------------------
+
+While exports or imports or running, you can check the state with::
+
+    curl -XGET 'localhost:9200/_export/state'
+
+or::
+
+    curl -XGET localhost:9200/_import/state
+
 
 Caution
 =======
 
-Knapsack is very simple and works without locking or index snapshots.
-So it is up to you to organize the safe export and import.
-If the index changes while Knapsack is exporting, you may lose data in the export.
-Do not run Knapsack in parallel on the same export.
+Knapsack is very simple and works without locks or snapshots. This means, if Elasticsearch is
+allowed to write to the part of your data in the export while it runs, you may lose data in the export.
+So it is up to you to organize the safe export and import with this plugin.
+
+If you want a snapshot/restore feature, please use the standard napshot/restore in the upcoming
+Elasticsearch 1.0 release.
+
+Credits
+=======
+
+Knapsack contains derived work of Apache Common Compress
+http://commons.apache.org/proper/commons-compress/
+
+The code in this component has many origins:
+The bzip2, tar and zip support came from Avalon's Excalibur, but originally
+from Ant, as far as life in Apache goes. The tar package is originally Tim Endres'
+public domain package. The bzip2 package is based on the work done by Keiron Liddle as
+ well as Julian Seward's libbzip2. It has migrated via:
+Ant -> Avalon-Excalibur -> Commons-IO -> Commons-Compress.
+The cpio package has been contributed by Michael Kuss and the jRPM project.
 
+Thanks to `nicktgr15 <https://github.com/nicktgr15>` for extending Knapsack to support Amazon S3.
 
 License
 =======

pom.xml

@@ -7,13 +7,13 @@
 
     <groupId>org.xbib.elasticsearch.plugin</groupId>
     <artifactId>elasticsearch-knapsack</artifactId>
-    <version>2.1.5</version>
+    <version>0.90.9.1</version>
 
     <packaging>jar</packaging>
 
     <name>elasticsearch-knapsack</name>
     <description>
-       Knapsack is a simple backup/restore tool for Elasticsearch
+       Knapsack is a simple import/export tool for Elasticsearch
     </description>
 
     <url>http://github.com/jprante/elasticsearch-knapsack</url>
@@ -58,7 +58,7 @@
     <properties>
         <github.global.server>github</github.global.server>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
-        <elasticsearch.version>0.90.6</elasticsearch.version>
+        <elasticsearch.version>0.90.9</elasticsearch.version>
     </properties>
 
     <dependencies>
@@ -72,9 +72,17 @@
         </dependency>
 
         <dependency>
-            <groupId>org.apache.commons</groupId>
-            <artifactId>commons-compress</artifactId>
-            <version>1.4.1</version>
+            <groupId>org.xbib.elasticsearch.plugin</groupId>
+            <artifactId>elasticsearch-support</artifactId>
+            <version>0.90.9.2</version>
+            <type>jar</type>
+            <scope>compile</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>com.amazonaws</groupId>
+            <artifactId>aws-java-sdk</artifactId>
+            <version>1.6.11</version>
             <type>jar</type>
             <scope>compile</scope>
         </dependency>
@@ -109,7 +117,7 @@
                     <optimize>true</optimize>
                     <showDeprecation>true</showDeprecation>
                     <showWarnings>true</showWarnings>
-                    <compilerArgument>-Xlint:all,-serial,-path,-rawtypes,-unchecked</compilerArgument>
+                    <compilerArgument>-Xlint:all,-serial,-path,-rawtypes,-unchecked,-fallthrough</compilerArgument>
                 </configuration>
             </plugin>
             <plugin>
@@ -140,10 +148,11 @@
                 <artifactId>maven-assembly-plugin</artifactId>
                 <version>2.3</version>
                 <configuration>
-                    <appendAssemblyId>false</appendAssemblyId>
+                    <appendAssemblyId>true</appendAssemblyId>
                     <outputDirectory>${project.build.directory}/releases/</outputDirectory>
                     <descriptors>
                         <descriptor>${basedir}/src/main/assemblies/plugin.xml</descriptor>
+                        <descriptor>${basedir}/src/main/assemblies/plugin-s3.xml</descriptor>
                     </descriptors>
                 </configuration>
                 <executions>