Resumable clreader

CHANGES.txt

@@ -1,4 +1,25 @@
-4.0
+4.0-alpha2
+ * Extract an AbstractCompactionController to allow for custom implementations (CASSANDRA-15286)
+ * Move chronicle-core version from snapshot to stable, and include carrotsearch in generated pom.xml (CASSANDRA-15321)
+ * Untangle RepairMessage sub-hierarchy of messages, use new messaging (more) correctly (CASSANDRA-15163)
+ * Add `allocate_tokens_for_local_replication_factor` option for token allocation (CASSANDRA-15260)
+ * Add Alibaba Cloud Platform snitch (CASSANDRA-15092)
+
+
+4.0-alpha1
+ * Inaccurate exception message with nodetool snapshot (CASSANDRA-15287)
+ * Fix InternodeOutboundMetrics overloaded bytes/count mixup (CASSANDRA-15186)
+ * Enhance & reenable RepairTest with compression=off and compression=on (CASSANDRA-15272)
+ * Improve readability of Table metrics Virtual tables units (CASSANDRA-15194)
+ * Fix error with non-existent table for nodetool tablehistograms (CASSANDRA-14410)
+ * Avoid result truncation in decimal operations (CASSANDRA-15232)
+ * Catch non-IOException in FileUtils.close to make sure that all resources are closed (CASSANDRA-15225)
+ * Align load column in nodetool status output (CASSANDRA-14787)
+ * CassandraNetworkAuthorizer uses cached roles info (CASSANDRA-15089)
+ * Introduce optional timeouts for idle client sessions (CASSANDRA-11097)
+ * Fix AlterTableStatement dropped type validation order (CASSANDRA-15203)
+ * Update Netty dependencies to latest, clean up SocketFactory (CASSANDRA-15195)
+ * Native Transport - Apply noSpamLogger to ConnectionLimitHandler (CASSANDRA-15167)
  * Reduce heap pressure during compactions (CASSANDRA-14654)
  * Support building Cassandra with JDK 11 (CASSANDRA-15108)
  * Use quilt to patch cassandra.in.sh in Debian packaging (CASSANDRA-14710)
@@ -358,13 +379,30 @@
  * nodetool clearsnapshot requires --all to clear all snapshots (CASSANDRA-13391)
  * Correctly count range tombstones in traces and tombstone thresholds (CASSANDRA-8527)
  * cqlshrc.sample uses incorrect option for time formatting (CASSANDRA-14243)
+ * Multi-version in-JVM dtests (CASSANDRA-14937)
+ * Allow instance class loaders to be garbage collected for inJVM dtest (CASSANDRA-15170)
 
 
 3.11.5
+ * Make sure user defined compaction transactions are always closed (CASSANDRA-15123)
  * Fix cassandra-env.sh to use $CASSANDRA_CONF to find cassandra-jaas.config (CASSANDRA-14305)
  * Fixed nodetool cfstats printing index name twice (CASSANDRA-14903)
  * Add flag to disable SASI indexes, and warnings on creation (CASSANDRA-14866)
 Merged from 3.0:
+ * Fix LegacyLayout RangeTombstoneList IndexOutOfBoundsException when upgrading and RangeTombstone bounds are asymmetric (CASSANDRA-15172)
+ * Fix NPE when using allocate_tokens_for_keyspace on new DC/rack (CASSANDRA-14952)
+ * Filter sstables earlier when running cleanup (CASSANDRA-15100)
+ * Use mean row count instead of mean column count for index selectivity calculation (CASSANDRA-15259)
+ * Avoid updating unchanged gossip states (CASSANDRA-15097)
+ * Prevent recreation of previously dropped columns with a different kind (CASSANDRA-14948)
+ * Prevent client requests from blocking on executor task queue (CASSANDRA-15013)
+ * Toughen up column drop/recreate type validations (CASSANDRA-15204)
+ * LegacyLayout should handle paging states that cross a collection column (CASSANDRA-15201)
+ * Prevent RuntimeException when username or password is empty/null (CASSANDRA-15198)
+ * Multiget thrift query returns null records after digest mismatch (CASSANDRA-14812)
+ * Handle paging states serialized with a different version than the session's (CASSANDRA-15176)
+ * Throw IOE instead of asserting on unsupporter peer versions (CASSANDRA-15066)
+ * Update token metadata when handling MOVING/REMOVING_TOKEN events (CASSANDRA-15120)
  * Add ability to customize cassandra log directory using $CASSANDRA_LOG_DIR (CASSANDRA-15090)
  * Fix assorted gossip races and add related runtime checks (CASSANDRA-15059)
  * cassandra-stress works with frozen collections: list and set (CASSANDRA-14907)
@@ -375,6 +413,7 @@ Merged from 3.0:
  * Add missing commands to nodetool_completion (CASSANDRA-14916)
  * Anti-compaction temporarily corrupts sstable state for readers (CASSANDRA-15004)
 Merged from 2.2:
+ * Handle exceptions during authentication/authorization (CASSANDRA-15041)
  * Support cross version messaging in in-jvm upgrade dtests (CASSANDRA-15078)
  * Fix index summary redistribution cancellation (CASSANDRA-15045)
  * Refactor Circle CI configuration (CASSANDRA-14806)

build.xml

@@ -25,7 +25,7 @@
     <property name="debuglevel" value="source,lines,vars"/>
 
     <!-- default version and SCM information -->
-    <property name="base.version" value="4.0"/>
+    <property name="base.version" value="4.0-alpha2"/>
     <property name="scm.connection" value="scm:https://gitbox.apache.org/repos/asf/cassandra.git"/>
     <property name="scm.developerConnection" value="scm:https://gitbox.apache.org/repos/asf/cassandra.git"/>
     <property name="scm.url" value="https://gitbox.apache.org/repos/asf?p=cassandra.git;a=tree"/>
@@ -118,7 +118,7 @@
 
     <!-- https://mvnrepository.com/artifact/net.openhft/chronicle-bom/1.16.23 -->
     <property name="chronicle-queue.version" value="4.16.3" />
-    <property name="chronicle-core.version" value="1.16.3-SNAPSHOT" />
+    <property name="chronicle-core.version" value="1.16.4" />
     <property name="chronicle-bytes.version" value="1.16.3" />
     <property name="chronicle-wire.version" value="1.16.1" />
     <property name="chronicle-threads.version" value="1.16.0" />
@@ -548,7 +548,8 @@
           <dependency groupId="com.addthis.metrics" artifactId="reporter-config3" version="3.0.3" />
           <dependency groupId="org.mindrot" artifactId="jbcrypt" version="0.3m" />
           <dependency groupId="io.airlift" artifactId="airline" version="0.8" />
-          <dependency groupId="io.netty" artifactId="netty-all" version="4.1.28.Final" />
+          <dependency groupId="io.netty" artifactId="netty-all" version="4.1.37.Final" />
+          <dependency groupId="io.netty" artifactId="netty-tcnative-boringssl-static" version="2.0.25.Final" />
           <dependency groupId="net.openhft" artifactId="chronicle-queue" version="${chronicle-queue.version}"/>
           <dependency groupId="net.openhft" artifactId="chronicle-core" version="${chronicle-core.version}"/>
           <dependency groupId="net.openhft" artifactId="chronicle-bytes" version="${chronicle-bytes.version}"/>
@@ -771,6 +772,7 @@
         <dependency groupId="com.github.ben-manes.caffeine" artifactId="caffeine" />
         <dependency groupId="org.jctools" artifactId="jctools-core"/>
         <dependency groupId="org.ow2.asm" artifactId="asm" />
+        <dependency groupId="com.carrotsearch" artifactId="hppc" version="0.5.4" />
       </artifact:pom>
       <artifact:pom id="dist-pom"
                     artifactId="apache-cassandra"
@@ -1106,7 +1108,7 @@
       </jar>
     </target>
 
-    <target name="_artifacts-init" depends="jar,javadoc,gen-doc" unless="java.version.8">
+    <target name="_artifacts-init" depends="jar,javadoc,gen-doc">
       <mkdir dir="${dist.dir}"/>
       <!-- fix the control linefeed so that builds on windows works on linux -->
       <fixcrlf srcdir="bin" includes="**/*" excludes="**/*.bat, **/*.ps1" eol="lf" eof="remove" />
@@ -1168,7 +1170,6 @@
 
     <!-- creates release tarballs -->
     <target name="artifacts" depends="_artifacts-init"
-            unless="java.version.8"
             description="Create Cassandra release artifacts">
       <tar compression="gzip" longfile="gnu"
         destfile="${build.dir}/${final.name}-bin.tar.gz">
@@ -1522,6 +1523,7 @@
   <target name="burn-testsome" depends="build-test" description="Execute specific burn unit tests" >
     <testmacro inputdir="${test.burn.src}" timeout="${test.burn.timeout}">
       <test name="${test.name}" methods="${test.methods}"/>
+      <jvmarg value="-Dlogback.configurationFile=test/conf/logback-burntest.xml"/>
     </testmacro>
   </target>
 
@@ -1945,6 +1947,20 @@
     <delete file="${test.distributed.listfile}"/>
   </target>
 
+  <!-- Build a self-contained jar for e.g. remote execution; not currently used for running burn tests with this build script -->
+  <target name="burn-test-jar" depends="build-test, build" description="Create dtest-compatible jar, including all dependencies">
+      <jar jarfile="${build.dir}/burntest.jar">
+          <zipgroupfileset dir="${build.lib}" includes="*.jar" excludes="META-INF/*.SF"/>
+          <fileset dir="${build.classes.main}"/>
+          <fileset dir="${test.classes}"/>
+          <fileset dir="${test.conf}" excludes="logback*.xml"/>
+          <fileset dir="${basedir}/conf" includes="logback*.xml"/>
+          <zipgroupfileset dir="${build.dir.lib}/jars">
+              <include name="junit*.jar"/>
+          </zipgroupfileset>
+      </jar>
+  </target>
+
   <target name="dtest-jar" depends="build-test, build" description="Create dtest-compatible jar, including all dependencies">
       <jar jarfile="${build.dir}/dtest-${base.version}.jar">
           <zipgroupfileset dir="${build.lib}" includes="*.jar" excludes="META-INF/*.SF"/>
@@ -1954,10 +1970,24 @@
       </jar>
   </target>
 
-  <target name="test-jvm-dtest" depends="build-test" description="Execute unit tests">
+  <target name="test-jvm-dtest" depends="build-test" description="Execute in-jvm dtests">
     <testmacro inputdir="${test.distributed.src}" timeout="${test.distributed.timeout}" forkmode="once" showoutput="true" filter="**/test/*Test.java">
       <jvmarg value="-Dlogback.configurationFile=test/conf/logback-dtest.xml"/>
-      <jvmarg value="-Dcassandra.ring_delay_ms=1000"/>
+      <jvmarg value="-Dcassandra.ring_delay_ms=10000"/>
+      <jvmarg value="-Dcassandra.tolerate_sstable_size=true"/>
+      <jvmarg value="-Djava.io.tmpdir=${tmp.dir}"/>
+      <jvmarg value="-Dcassandra.skip_sync=true" />
+    </testmacro>
+  </target>
+
+  <!-- Use this with an FQDN for test class, and a csv list of methods like this:
+      ant test-jvm-dtest-some -Dtest.name=org.apache.cassandra.distributed.test.ResourceLeakTest -Dtest.methods=looperTest
+    -->
+  <target name="test-jvm-dtest-some" depends="build-test" description="Execute some in-jvm dtests">
+    <testmacro inputdir="${test.distributed.src}" timeout="${test.distributed.timeout}" forkmode="once" showoutput="true">
+      <test name="${test.name}" methods="${test.methods}" outfile="build/test/output/TEST-${test.name}-${test.methods}"/>
+      <jvmarg value="-Dlogback.configurationFile=test/conf/logback-dtest.xml"/>
+      <jvmarg value="-Dcassandra.ring_delay_ms=10000"/>
       <jvmarg value="-Dcassandra.tolerate_sstable_size=true"/>
       <jvmarg value="-Djava.io.tmpdir=${tmp.dir}"/>
       <jvmarg value="-Dcassandra.skip_sync=true" />

conf/cassandra.yaml

@@ -26,15 +26,21 @@ num_tokens: 256
 
 # Triggers automatic allocation of num_tokens tokens for this node. The allocation
 # algorithm attempts to choose tokens in a way that optimizes replicated load over
-# the nodes in the datacenter for the replication strategy used by the specified
-# keyspace.
+# the nodes in the datacenter for the replica factor.
 #
 # The load assigned to each node will be close to proportional to its number of
 # vnodes.
 #
 # Only supported with the Murmur3Partitioner.
+
+# Replica factor is determined via the replication strategy used by the specified
+# keyspace.
 # allocate_tokens_for_keyspace: KEYSPACE
 
+# Replica factor is explicitly set, regardless of keyspace or datacenter.
+# This is the replica factor within the datacenter, like NTS.
+# allocate_tokens_for_local_replication_factor: 3
+
 # initial_token allows you to specify tokens manually.  While you can use it with
 # vnodes (num_tokens > 1, above) -- in which case you should provide a 
 # comma-separated list -- it's primarily used when adding nodes to legacy clusters 
@@ -683,6 +689,16 @@ native_transport_port: 9042
 # The default is true, which means all supported protocols will be honored.
 native_transport_allow_older_protocols: true
 
+# Controls when idle client connections are closed. Idle connections are ones that had neither reads
+# nor writes for a time period.
+#
+# Clients may implement heartbeats by sending OPTIONS native protocol message after a timeout, which
+# will reset idle timeout timer on the server side. To close idle client connections, corresponding
+# values for heartbeat intervals have to be set on the client side.
+#
+# Idle connection timeouts are disabled by default.
+# native_transport_idle_timeout_in_ms: 60000
+
 # The address or interface to bind the native transport server to.
 #
 # Set rpc_address OR rpc_interface, not both.
@@ -870,6 +886,32 @@ request_timeout_in_ms: 10000
 # which picks up the OS default and configure the net.ipv4.tcp_retries2 sysctl to be ~8.
 # internode_tcp_user_timeout_in_ms = 30000
 
+# The maximum continuous period a connection may be unwritable in application space
+# internode_application_timeout_in_ms = 30000
+
+# Global, per-endpoint and per-connection limits imposed on messages queued for delivery to other nodes
+# and waiting to be processed on arrival from other nodes in the cluster.  These limits are applied to the on-wire
+# size of the message being sent or received.
+#
+# The basic per-link limit is consumed in isolation before any endpoint or global limit is imposed.
+# Each node-pair has three links: urgent, small and large.  So any given node may have a maximum of
+# N*3*(internode_application_send_queue_capacity_in_bytes+internode_application_receive_queue_capacity_in_bytes)
+# messages queued without any coordination between them although in practice, with token-aware routing, only RF*tokens
+# nodes should need to communicate with significant bandwidth.
+#
+# The per-endpoint limit is imposed on all messages exceeding the per-link limit, simultaneously with the global limit,
+# on all links to or from a single node in the cluster.
+# The global limit is imposed on all messages exceeding the per-link limit, simultaneously with the per-endpoint limit,
+# on all links to or from any node in the cluster.
+#
+# internode_application_send_queue_capacity_in_bytes: 4194304                       #4MiB
+# internode_application_send_queue_reserve_endpoint_capacity_in_bytes: 134217728    #128MiB
+# internode_application_send_queue_reserve_global_capacity_in_bytes: 536870912      #512MiB
+# internode_application_receive_queue_capacity_in_bytes: 4194304                    #4MiB
+# internode_application_receive_queue_reserve_endpoint_capacity_in_bytes: 134217728 #128MiB
+# internode_application_receive_queue_reserve_global_capacity_in_bytes: 536870912   #512MiB
+
+
 # How long before a node logs slow queries. Select queries that take longer than
 # this timeout to execute, will generate an aggregated log message, so that slow queries
 # can be identified. Set this value to zero to disable slow query logging.
@@ -1305,4 +1347,4 @@ enable_sasi_indexes: false
 
 # Enables creation of transiently replicated keyspaces on this node.
 # Transient replication is experimental and is not recommended for production use.
-enable_transient_replication: false
+enable_transient_replication: false

debian/changelog

@@ -1,8 +1,14 @@
-cassandra (4.0) UNRELEASED; urgency=medium
+cassandra (4.0~alpha2) UNRELEASED; urgency=medium
 
   * New release
 
- -- Michael Shuler <michael@pbandjelly.org>  Wed, 15 Feb 2017 18:20:09 -0600
+ --
+
+cassandra (4.0~alpha1) unstable; urgency=medium
+
+  * New release
+
+ -- Michael Shuler <michael@pbandjelly.org>  Tue, 03 Sep 2019 11:51:18 -0500
 
 cassandra (3.10) unstable; urgency=medium
 

debian/patches/cassandra_logdir_fix.diff

@@ -1,14 +1,14 @@
 --- a/bin/cassandra
 +++ b/bin/cassandra
-@@ -171,7 +171,7 @@
-     props="$3"
-     class="$4"
-     cassandra_parms="-Dlogback.configurationFile=logback.xml"
--    cassandra_parms="$cassandra_parms -Dcassandra.logdir=$CASSANDRA_HOME/logs"
-+    cassandra_parms="$cassandra_parms -Dcassandra.logdir=/var/log/cassandra"
-     cassandra_parms="$cassandra_parms -Dcassandra.storagedir=$cassandra_storagedir"
+@@ -109,7 +109,7 @@
+ fi
+
+ if [ -z "$CASSANDRA_LOG_DIR" ]; then
+-  CASSANDRA_LOG_DIR=$CASSANDRA_HOME/logs
++  CASSANDRA_LOG_DIR=/var/log/cassandra
+ fi
 
-     if [ "x$pidpath" != "x" ]; then
+ # Special-case path variables.
 --- a/conf/cassandra-env.sh
 +++ b/conf/cassandra-env.sh
 @@ -93,16 +93,16 @@

doc/native_protocol_v4.spec

@@ -275,6 +275,9 @@ Table of Contents
       mode. This mode will make all Thrift and Compact Tables to be exposed as if
       they were CQL Tables. This is optional; if not specified, the option will
       not be used.
+    - "THROW_ON_OVERLOAD": In case of server overloaded with too many requests, by default the server puts
+            back pressure on the client connection. Instead, the server can send an OverloadedException error message back to
+            the client if this option is set to true.
 
 
 4.1.2. AUTH_RESPONSE
@@ -1185,3 +1188,4 @@ Table of Contents
   * The <paging_state> returned in the v4 protocol is not compatible with the v3
     protocol. In other words, a <paging_state> returned by a node using protocol v4
     should not be used to query a node using protocol v3 (and vice-versa).
+  * Added THROW_ON_OVERLOAD startup option (Section 4.1.1).

doc/source/bugs.rst

@@ -18,7 +18,7 @@ Reporting Bugs
 ==============
 
 If you encounter a problem with Cassandra, the first places to ask for help are the :ref:`user mailing list
-<mailing-lists>` and the ``#cassandra`` :ref:`IRC channel <irc-channels>`.
+<mailing-lists>` and the ``cassandra`` :ref:`Slack room <slack>`.
 
 If, after having asked for help, you suspect that you have found a bug in Cassandra, you should report it by opening a
 ticket through the `Apache Cassandra JIRA <https://issues.apache.org/jira/browse/CASSANDRA>`__. Please provide as much

doc/source/contactus.rst

@@ -17,7 +17,7 @@
 Contact us
 ==========
 
-You can get in touch with the Cassandra community either via the mailing lists or the freenode IRC channels.
+You can get in touch with the Cassandra community either via the mailing lists or :ref:`Slack rooms <slack>`.
 
 .. _mailing-lists:
 
@@ -39,15 +39,12 @@ Subscribe by sending an email to the email address in the Subscribe links above.
 email to confirm your subscription. Make sure to keep the welcome email as it contains instructions on how to
 unsubscribe.
 
-.. _irc-channels:
+.. _slack:
 
-IRC
----
+Slack
+-----
+To chat with developers or users in real-time, join our rooms on `ASF Slack <https://s.apache.org/slack-invite>`__:
 
-To chat with developers or users in real-time, join our channels on `IRC freenode <http://webchat.freenode.net/>`__. The
-following channels are available:
-
-- ``#cassandra`` - for user questions and general discussions.
-- ``#cassandra-dev`` - strictly for questions or discussions related to Cassandra development.
-- ``#cassandra-builds`` - results of automated test builds.
+- ``cassandra`` - for user questions and general discussions.
+- ``cassandra-dev`` - strictly for questions or discussions related to Cassandra development.
 

doc/source/cql/security.rst

@@ -148,6 +148,14 @@ status may ``DROP`` another ``SUPERUSER`` role.
 Attempting to drop a role which does not exist results in an invalid query condition unless the ``IF EXISTS`` option is
 used. If the option is used and the role does not exist the statement is a no-op.
 
+.. note:: DROP ROLE intentionally does not terminate any open user sessions. Currently connected sessions will remain
+   connected and will retain the ability to perform any database actions which do not require :ref:`authorization<authorization>`.
+   However, if authorization is enabled, :ref:`permissions<cql-permissions>` of the dropped role are also revoked,
+   subject to the :ref:`caching options<auth-caching>` configured in :ref:`cassandra.yaml<cassandra-yaml>`.
+   Should a dropped role be subsequently recreated and have new :ref:`permissions<grant-permission-statement>` or
+   :ref:`roles<grant-role-statement>` granted to it, any client sessions still connected will acquire the newly granted
+   permissions and roles.
+
 .. _grant-role-statement:
 
 GRANT ROLE