Cleanup install instructions for optional libs (#2139)

This cleans up the install instructions to specify that optional libs
should be installed using the `kafka-python[opt]` format. This leverages
`setuptools`' `extra_requires` feature to let our users choose to
inherit any versions pins we might apply to our optional dependencies.

This also re-orders the docs slightly to give more visibility to the
`crc32c` install because users are unlikely to realize it exists.

It also cleans up the formatting of the compression libraries slightly,
as they were getting a little unwieldy.

Author: jeffwidman

README.rst

@@ -34,6 +34,7 @@ documentation, please see readthedocs and/or python's inline help.
 
 >>> pip install kafka-python
 
+
 KafkaConsumer
 *************
 
@@ -78,6 +79,7 @@ that expose basic message attributes: topic, partition, offset, key, and value:
 >>> # Get consumer metrics
 >>> metrics = consumer.metrics()
 
+
 KafkaProducer
 *************
 
@@ -124,6 +126,7 @@ for more details.
 >>> # Get producer performance metrics
 >>> metrics = producer.metrics()
 
+
 Thread safety
 *************
 
@@ -133,22 +136,30 @@ KafkaConsumer which cannot.
 While it is possible to use the KafkaConsumer in a thread-local manner,
 multiprocessing is recommended.
 
+
 Compression
 ***********
 
-kafka-python supports gzip compression/decompression natively. To produce or consume lz4
-compressed messages, you should install python-lz4 (pip install lz4).
-To enable snappy compression/decompression install python-snappy (also requires snappy library).
-See <https://kafka-python.readthedocs.io/en/master/install.html#optional-snappy-install>
-for more information.
+kafka-python supports the following compression formats:
+
+- gzip
+- LZ4
+- Snappy
+- Zstandard (zstd)
+
+gzip is supported natively, the others require installing additional libraries.
+See <https://kafka-python.readthedocs.io/en/master/install.html> for more information.
+
 
 Optimized CRC32 Validation
 **************************
 
 Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
 python implementation for compatibility. To improve performance for high-throughput
 applications, kafka-python will use `crc32c` for optimized native code if installed.
-See https://pypi.org/project/crc32c/
+See <https://kafka-python.readthedocs.io/en/master/install.html> for installation instructions.
+See https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.
+
 
 Protocol
 ********

docs/index.rst

@@ -33,6 +33,7 @@ documentation, please see readthedocs and/or python's inline help.
 
 >>> pip install kafka-python
 
+
 KafkaConsumer
 *************
 
@@ -122,12 +123,26 @@ multiprocessing is recommended.
 Compression
 ***********
 
-kafka-python supports multiple compression types:
+kafka-python supports the following compression formats:
+
+ - gzip
+ - LZ4
+ - Snappy
+ - Zstandard (zstd)
+
+gzip is supported natively, the others require installing additional libraries.
+See `Install <install.html>`_ for more information.
+
+
+Optimized CRC32 Validation
+**************************
+
+Kafka uses CRC32 checksums to validate messages. kafka-python includes a pure
+python implementation for compatibility. To improve performance for high-throughput
+applications, kafka-python will use `crc32c` for optimized native code if installed.
+See `Install <install.html>`_ for installation instructions and
+https://pypi.org/project/crc32c/ for details on the underlying crc32c lib.
 
- - gzip : supported natively
- - lz4 : requires `python-lz4 <https://pypi.org/project/lz4/>`_ installed
- - snappy : requires the `python-snappy <https://pypi.org/project/python-snappy/>`_  package (which requires the snappy C library)
- - zstd : requires the `python-zstandard <https://github.com/indygreg/python-zstandard>`_ package installed
 
 Protocol
 ********

docs/install.rst

@@ -23,20 +23,33 @@ Bleeding-Edge
     pip install ./kafka-python
 
 
-Optional LZ4 install
+Optional crc32c install
+***********************
+Highly recommended if you are using Kafka 11+ brokers. For those `kafka-python`
+uses a new message protocol version, that requires calculation of `crc32c`,
+which differs from the `zlib.crc32` hash implementation. By default `kafka-python`
+calculates it in pure python, which is quite slow. To speed it up we optionally
+support https://pypi.python.org/pypi/crc32c package if it's installed.
+
+.. code:: bash
+
+    pip install 'kafka-python[crc32c]'
+
+
+Optional ZSTD install
 ********************
 
-To enable LZ4 compression/decompression, install python-lz4:
+To enable ZSTD compression/decompression, install python-zstandard:
 
->>> pip install lz4
+>>> pip install 'kafka-python[zstd]'
 
 
-Optional crc32c install
+Optional LZ4 install
 ********************
 
-To enable optimized CRC32 checksum validation, install crc32c:
+To enable LZ4 compression/decompression, install python-lz4:
 
->>> pip install crc32c
+>>> pip install 'kafka-python[lz4]'
 
 
 Optional Snappy install
@@ -77,17 +90,4 @@ Install the `python-snappy` module
 
 .. code:: bash
 
-    pip install python-snappy
-
-
-Optional crc32c install
-***********************
-Highly recommended if you are using Kafka 11+ brokers. For those `kafka-python`
-uses a new message protocol version, that requires calculation of `crc32c`,
-which differs from `zlib.crc32` hash implementation. By default `kafka-python`
-calculates it in pure python, which is quite slow. To speed it up we optionally
-support https://pypi.python.org/pypi/crc32c package if it's installed.
-
-.. code:: bash
-
-    pip install crc32c
+    pip install 'kafka-python[snappy]'