From 1e7fc135c5cf9641a8d10080591c16c000c9eb5d Mon Sep 17 00:00:00 2001 From: LinZhihao-723 Date: Mon, 27 Oct 2025 15:21:54 -0400 Subject: [PATCH 1/6] Draft docs. --- .../guides-using-object-storage/clp-usage.md | 84 +++++++++++++++---- 1 file changed, 68 insertions(+), 16 deletions(-) diff --git a/docs/src/user-docs/guides-using-object-storage/clp-usage.md b/docs/src/user-docs/guides-using-object-storage/clp-usage.md index e4eec83c76..24c56cfac8 100644 --- a/docs/src/user-docs/guides-using-object-storage/clp-usage.md +++ b/docs/src/user-docs/guides-using-object-storage/clp-usage.md @@ -5,35 +5,87 @@ should be able to use CLP as described in the [clp-json quick-start guide](../qu ## Compressing logs from S3 -To compress logs from S3, use the `sbin/compress.sh` script as follows, replacing the fields in -angle brackets (`<>`) with the appropriate values: +You can use `sbin/compress-from-s3.sh` to compress logs from S3. It has two modes of operation: + +* `s3-object` : Compress S3 objects specified by their full S3 URLs. +* `s3-key-prefix` : Compress all S3 objects under a given S3 key prefix. + +### `s3-object` compression mode + +`s3-object` mode allows you to specify individual S3 objects to compress by their full URLs. To +use this mode, call the `sbin/compress-from-s3.sh` script as follows, replacing the fields in angle +brackets (`<>`) with the appropriate values: ```bash -sbin/compress.sh \ +sbin/compress-from-s3.sh \ --timestamp-key \ - + --dataset \ + s3-object \ + [ ...] ``` -* `` is a URL identifying the logs to compress. It can have one of two formats: - * `https://.s3..amazonaws.com/` - * `https://s3..amazonaws.com//` -* The fields in `` are as follows: +* `` is a URL identifying the S3 object to compress. It can have one of two formats: + * `https://.s3..amazonaws.com/` + * `https://s3..amazonaws.com//` +* The fields in `` are as follows: * `` is the name of the S3 bucket containing your logs. * `` is the AWS region [code][aws-region-codes] for the S3 bucket containing your logs. - * `` is the prefix of all logs you wish to compress and must begin with the - `` value from the [compression IAM policy][compression-iam-policy]. + * `` is the [object key][aws-s3-object-key] of the log file object you wish to + compress. There must be no duplicate object keys across all `` arguments. + +Optionally, you can specify the input object URLs through a text file by using the `--inputs-from` +option as follows: + +```bash +sbin/compress-from-s3.sh \ + --timestamp-key \ + --dataset \ + s3-object \ + --inputs-from +``` + +* `` is a path to a text file containing one S3 object URL **per line**. The URLs must + follow the same format as described above for ``. + :::{note} -Compressing from S3 only supports a single URL but will compress any logs that have the given -prefix. +`s3-object` mode requires the input object keys to share a non-empty common prefix. If the input +object keys do not share a common prefix, they will be rejected and no compression job will be +created. This limitation will be relaxed in a future release. +::: -If you wish to compress a single log file, specify the entire path to the log file. However, if -that log file's path is a prefix of another log file's path, then both log files will be compressed -(e.g., with two files "logs/syslog" and "logs/syslog.1", a prefix like "logs/syslog" will cause -both logs to be compressed). This limitation will be addressed in a future release. +### `s3-key-prefix` compression mode + +`s3-key-prefix` mode allows you to compress all objects under a given S3 key-prefix. To use this +mode, call the `sbin/compress-from-s3.sh` script as follows, replacing the fields in angle brackets +(`<>`) with the appropriate values: + +```bash +sbin/compress-from-s3.sh \ + --timestamp-key \ + --dataset \ + s3-key-prefix \ + +``` + +* `` is a URL identifying the S3 key-prefix to compress. It can have one of two + formats: + * `https://.s3..amazonaws.com/` + * `https://s3..amazonaws.com//` +* The fields in `` are as follows: + * `` is the name of the S3 bucket containing your logs. + * `` is the AWS region [code][aws-region-codes] for the S3 bucket containing your + logs. + * `` is the prefix of all logs you wish to compress and must begin with the + `` value from the [compression IAM policy][compression-iam-policy]. + +:::{note} +`s3-key-prefix` mode only accepts a single `` argument. This limitation will be +relaxed in a future release. ::: [add-iam-policy]: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#embed-inline-policy-console +[aws-s3-object-key]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html [aws-region-codes]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html#Concepts.RegionsAndAvailabilityZones.Availability [compression-iam-policy]: ./object-storage-config.md#configuration-for-compression From f23ab479b3b56d7cc6232f4073bccd00fcb0cd25 Mon Sep 17 00:00:00 2001 From: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com> Date: Wed, 29 Oct 2025 18:48:21 -0400 Subject: [PATCH 2/6] Apply suggestions from code review Co-authored-by: Quinn Taylor Mitchell --- .../guides-using-object-storage/clp-usage.md | 47 ++++++++++--------- 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/docs/src/user-docs/guides-using-object-storage/clp-usage.md b/docs/src/user-docs/guides-using-object-storage/clp-usage.md index 24c56cfac8..09af502281 100644 --- a/docs/src/user-docs/guides-using-object-storage/clp-usage.md +++ b/docs/src/user-docs/guides-using-object-storage/clp-usage.md @@ -5,16 +5,20 @@ should be able to use CLP as described in the [clp-json quick-start guide](../qu ## Compressing logs from S3 -You can use `sbin/compress-from-s3.sh` to compress logs from S3. It has two modes of operation: +To compress logs from S3, use the `sbin/compress-from-s3.sh` script. The script runs in one of two +modes of operation; select which one you'd like to use by using the corresponding argument in the +command: -* `s3-object` : Compress S3 objects specified by their full S3 URLs. -* `s3-key-prefix` : Compress all S3 objects under a given S3 key prefix. +* [**`s3-object` mode**](#s3-object-compression-mode): Compress S3 objects specified by their full + S3 URLs. +* [**`s3-key-prefix` mode**](#s3-key-prefix-compression-mode): Compress all S3 objects under a given + S3 key prefix. ### `s3-object` compression mode `s3-object` mode allows you to specify individual S3 objects to compress by their full URLs. To -use this mode, call the `sbin/compress-from-s3.sh` script as follows, replacing the fields in angle -brackets (`<>`) with the appropriate values: +use this mode, call the `sbin/compress-from-s3.sh` script as follows, and replace the fields in +angle brackets (`<>`) with the appropriate values: ```bash sbin/compress-from-s3.sh \ @@ -24,7 +28,8 @@ sbin/compress-from-s3.sh \ [ ...] ``` -* `` is a URL identifying the S3 object to compress. It can have one of two formats: +* `` is a URL identifying the S3 object to compress. It can be written in either one of + two formats: * `https://.s3..amazonaws.com/` * `https://s3..amazonaws.com//` * The fields in `` are as follows: @@ -34,8 +39,8 @@ sbin/compress-from-s3.sh \ * `` is the [object key][aws-s3-object-key] of the log file object you wish to compress. There must be no duplicate object keys across all `` arguments. -Optionally, you can specify the input object URLs through a text file by using the `--inputs-from` -option as follows: +Instead of typing the input object URLs explicitly in the command, you may specify them in a text +file and then pass the file into the command using the `--inputs-from` flag, like so: ```bash sbin/compress-from-s3.sh \ @@ -52,14 +57,14 @@ sbin/compress-from-s3.sh \ :::{note} `s3-object` mode requires the input object keys to share a non-empty common prefix. If the input object keys do not share a common prefix, they will be rejected and no compression job will be -created. This limitation will be relaxed in a future release. +created. This limitation will be addressed in a future release. ::: ### `s3-key-prefix` compression mode -`s3-key-prefix` mode allows you to compress all objects under a given S3 key-prefix. To use this -mode, call the `sbin/compress-from-s3.sh` script as follows, replacing the fields in angle brackets -(`<>`) with the appropriate values: +`s3-key-prefix` mode allows you to compress all objects under a given S3 key prefix. To use this +mode, call the `sbin/compress-from-s3.sh` script as follows, and replace the fields in angle +brackets (`<>`) with the appropriate values: ```bash sbin/compress-from-s3.sh \ @@ -69,20 +74,20 @@ sbin/compress-from-s3.sh \ ``` -* `` is a URL identifying the S3 key-prefix to compress. It can have one of two - formats: +* `` is a URL identifying the S3 key prefix to compress. It can be written in either + one of two formats: * `https://.s3..amazonaws.com/` * `https://s3..amazonaws.com//` -* The fields in `` are as follows: - * `` is the name of the S3 bucket containing your logs. - * `` is the AWS region [code][aws-region-codes] for the S3 bucket containing your - logs. - * `` is the prefix of all logs you wish to compress and must begin with the - `` value from the [compression IAM policy][compression-iam-policy]. + * The fields in `` are as follows: + * `` is the name of the S3 bucket containing your logs. + * `` is the AWS region [code][aws-region-codes] for the S3 bucket containing your + logs. + * `` is the prefix of all logs you wish to compress and must begin with the + `` value from the [compression IAM policy][compression-iam-policy]. :::{note} `s3-key-prefix` mode only accepts a single `` argument. This limitation will be -relaxed in a future release. +addressed in a future release. ::: [add-iam-policy]: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#embed-inline-policy-console From 7e369870669a22c98e9e8c342503a00a8b110a7e Mon Sep 17 00:00:00 2001 From: LinZhihao-723 Date: Wed, 29 Oct 2025 19:03:48 -0400 Subject: [PATCH 3/6] Address code review comments with nit modification. --- .../guides-using-object-storage/clp-usage.md | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/src/user-docs/guides-using-object-storage/clp-usage.md b/docs/src/user-docs/guides-using-object-storage/clp-usage.md index 09af502281..a3030a1cbf 100644 --- a/docs/src/user-docs/guides-using-object-storage/clp-usage.md +++ b/docs/src/user-docs/guides-using-object-storage/clp-usage.md @@ -9,9 +9,9 @@ To compress logs from S3, use the `sbin/compress-from-s3.sh` script. The script modes of operation; select which one you'd like to use by using the corresponding argument in the command: -* [**`s3-object` mode**](#s3-object-compression-mode): Compress S3 objects specified by their full +* [**s3-object** mode](#s3-object-compression-mode): Compress S3 objects specified by their full S3 URLs. -* [**`s3-key-prefix` mode**](#s3-key-prefix-compression-mode): Compress all S3 objects under a given +* [**s3-key-prefix** mode](#s3-key-prefix-compression-mode): Compress all S3 objects under a given S3 key prefix. ### `s3-object` compression mode @@ -78,19 +78,18 @@ sbin/compress-from-s3.sh \ one of two formats: * `https://.s3..amazonaws.com/` * `https://s3..amazonaws.com//` - * The fields in `` are as follows: - * `` is the name of the S3 bucket containing your logs. - * `` is the AWS region [code][aws-region-codes] for the S3 bucket containing your - logs. - * `` is the prefix of all logs you wish to compress and must begin with the - `` value from the [compression IAM policy][compression-iam-policy]. +* The fields in `` are as follows: + * `` is the name of the S3 bucket containing your logs. + * `` is the AWS region [code][aws-region-codes] for the S3 bucket containing your + logs. + * `` is the prefix of all logs you wish to compress and must begin with the + `` value from the [compression IAM policy][compression-iam-policy]. :::{note} `s3-key-prefix` mode only accepts a single `` argument. This limitation will be addressed in a future release. ::: -[add-iam-policy]: https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage-attach-detach.html#embed-inline-policy-console [aws-s3-object-key]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html [aws-region-codes]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html#Concepts.RegionsAndAvailabilityZones.Availability [compression-iam-policy]: ./object-storage-config.md#configuration-for-compression From bebd04602184e2ee6d43b183f1caac6e2c255195 Mon Sep 17 00:00:00 2001 From: Quinn Taylor Mitchell Date: Thu, 30 Oct 2025 10:22:17 -0400 Subject: [PATCH 4/6] Direct-commit nit comments --- .../guides-using-object-storage/clp-usage.md | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/src/user-docs/guides-using-object-storage/clp-usage.md b/docs/src/user-docs/guides-using-object-storage/clp-usage.md index a3030a1cbf..258c1bb7ab 100644 --- a/docs/src/user-docs/guides-using-object-storage/clp-usage.md +++ b/docs/src/user-docs/guides-using-object-storage/clp-usage.md @@ -5,9 +5,8 @@ should be able to use CLP as described in the [clp-json quick-start guide](../qu ## Compressing logs from S3 -To compress logs from S3, use the `sbin/compress-from-s3.sh` script. The script runs in one of two -modes of operation; select which one you'd like to use by using the corresponding argument in the -command: +To compress logs from S3, use the `sbin/compress-from-s3.sh` script. The script supports two modes +of operation. Select a mode by using the corresponding argument in the command: * [**s3-object** mode](#s3-object-compression-mode): Compress S3 objects specified by their full S3 URLs. @@ -28,8 +27,8 @@ sbin/compress-from-s3.sh \ [ ...] ``` -* `` is a URL identifying the S3 object to compress. It can be written in either one of - two formats: +* `` is a URL identifying the S3 object to compress. It can be written in either of two + formats: * `https://.s3..amazonaws.com/` * `https://s3..amazonaws.com//` * The fields in `` are as follows: @@ -75,7 +74,7 @@ sbin/compress-from-s3.sh \ ``` * `` is a URL identifying the S3 key prefix to compress. It can be written in either - one of two formats: + of two formats: * `https://.s3..amazonaws.com/` * `https://s3..amazonaws.com//` * The fields in `` are as follows: @@ -86,8 +85,8 @@ sbin/compress-from-s3.sh \ `` value from the [compression IAM policy][compression-iam-policy]. :::{note} -`s3-key-prefix` mode only accepts a single `` argument. This limitation will be -addressed in a future release. +`s3-key-prefix` mode accepts a single `` argument. This limitation will be addressed +in a future release. ::: [aws-s3-object-key]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html From 5a33db90b07c2aa3d29029cd41e2d0b458416622 Mon Sep 17 00:00:00 2001 From: Lin Zhihao <59785146+LinZhihao-723@users.noreply.github.com> Date: Sat, 1 Nov 2025 14:57:42 -0400 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com> --- .../guides-using-object-storage/clp-usage.md | 23 ++++++++++++------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/docs/src/user-docs/guides-using-object-storage/clp-usage.md b/docs/src/user-docs/guides-using-object-storage/clp-usage.md index 258c1bb7ab..1ac8a90cb7 100644 --- a/docs/src/user-docs/guides-using-object-storage/clp-usage.md +++ b/docs/src/user-docs/guides-using-object-storage/clp-usage.md @@ -6,7 +6,7 @@ should be able to use CLP as described in the [clp-json quick-start guide](../qu ## Compressing logs from S3 To compress logs from S3, use the `sbin/compress-from-s3.sh` script. The script supports two modes -of operation. Select a mode by using the corresponding argument in the command: +of operation: * [**s3-object** mode](#s3-object-compression-mode): Compress S3 objects specified by their full S3 URLs. @@ -36,9 +36,17 @@ sbin/compress-from-s3.sh \ * `` is the AWS region [code][aws-region-codes] for the S3 bucket containing your logs. * `` is the [object key][aws-s3-object-key] of the log file object you wish to - compress. There must be no duplicate object keys across all `` arguments. + compress. -Instead of typing the input object URLs explicitly in the command, you may specify them in a text + :::{warning} + There must be no duplicate object keys across all `` arguments. + ::: + + +* For a description of other fields, see the [clp-json quick-start + guide](../quick-start/clp-json.md#compressing-json-logs). + +Instead of specifying input object URLs explicitly in the command, you may specify them in a text file and then pass the file into the command using the `--inputs-from` flag, like so: ```bash @@ -52,9 +60,8 @@ sbin/compress-from-s3.sh \ * `` is a path to a text file containing one S3 object URL **per line**. The URLs must follow the same format as described above for ``. - :::{note} -`s3-object` mode requires the input object keys to share a non-empty common prefix. If the input +The `s3-object` mode requires the input object keys to share a non-empty common prefix. If the input object keys do not share a common prefix, they will be rejected and no compression job will be created. This limitation will be addressed in a future release. ::: @@ -85,10 +92,10 @@ sbin/compress-from-s3.sh \ `` value from the [compression IAM policy][compression-iam-policy]. :::{note} -`s3-key-prefix` mode accepts a single `` argument. This limitation will be addressed -in a future release. +`s3-key-prefix` mode only accepts a single `` argument. This limitation will be +addressed in a future release. ::: -[aws-s3-object-key]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html [aws-region-codes]: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html#Concepts.RegionsAndAvailabilityZones.Availability +[aws-s3-object-key]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html [compression-iam-policy]: ./object-storage-config.md#configuration-for-compression From aa941674acb22ab376ecc4f241d2a41d72720612 Mon Sep 17 00:00:00 2001 From: LinZhihao-723 Date: Sat, 1 Nov 2025 15:10:00 -0400 Subject: [PATCH 6/6] Partially apply Kirk's comments with Quinn's suggestions. --- .../user-docs/guides-using-object-storage/clp-usage.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/src/user-docs/guides-using-object-storage/clp-usage.md b/docs/src/user-docs/guides-using-object-storage/clp-usage.md index 1ac8a90cb7..db978e663c 100644 --- a/docs/src/user-docs/guides-using-object-storage/clp-usage.md +++ b/docs/src/user-docs/guides-using-object-storage/clp-usage.md @@ -15,9 +15,9 @@ of operation: ### `s3-object` compression mode -`s3-object` mode allows you to specify individual S3 objects to compress by their full URLs. To -use this mode, call the `sbin/compress-from-s3.sh` script as follows, and replace the fields in -angle brackets (`<>`) with the appropriate values: +The `s3-object` mode allows you to specify individual S3 objects to compress by using their full +URLs. To use this mode, call the `sbin/compress-from-s3.sh` script as follows, and replace the +fields in angle brackets (`<>`) with the appropriate values: ```bash sbin/compress-from-s3.sh \ @@ -68,7 +68,7 @@ created. This limitation will be addressed in a future release. ### `s3-key-prefix` compression mode -`s3-key-prefix` mode allows you to compress all objects under a given S3 key prefix. To use this +The `s3-key-prefix` mode allows you to compress all objects under a given S3 key prefix. To use this mode, call the `sbin/compress-from-s3.sh` script as follows, and replace the fields in angle brackets (`<>`) with the appropriate values: