Skip to content

Latest commit

 

History

History
76 lines (72 loc) · 5.89 KB

s3.md

File metadata and controls

76 lines (72 loc) · 5.89 KB

Amazon S3 plugin for Fluentd

Overview

s3 output plugin buffers event logs in local file and upload it to S3 periodically. This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log '2011-01-02 message B' is reached, and then another log '2011-01-03 message B' is reached in this order, the former one is stored in "20110102.gz" file, and latter one in "20110103.gz" file.

Configuration

Output Config

Variable Name Type Required Default Description
aws_key_id *secret.Secret No - AWS access key id
Secret
aws_sec_key *secret.Secret No - AWS secret key.
Secret
check_apikey_on_start string No - Check AWS key on start
grant_read string No - Allows grantee to read the object data and its metadata
overwrite string No - Overwrite already existing path
path string No - Path prefix of the files on S3
grant_write_acp string No - Allows grantee to write the ACL for the applicable object
check_bucket string No - Check bucket if exists or not
sse_customer_key string No - Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data
sse_customer_key_md5 string No - Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321
compute_checksums string No - AWS SDK uses MD5 for API request/response by default
warn_for_delay string No - Given a threshold to treat events as delay, output warning logs if delayed events were put into s3
use_bundled_cert string No - Use aws-sdk-ruby bundled cert
s3_endpoint string No - Custom S3 endpoint (like minio)
ssekms_key_id string No - Specifies the AWS KMS key ID to use for object encryption
s3_metadata string No - Arbitrary S3 metadata headers to set for the object
force_path_style string No - If true, the bucket name is always left in the request URI and never moved to the host as a sub-domain
auto_create_bucket string No - Create S3 bucket if it does not exists
index_format string No - sprintf format for %{index}
signature_version string No - Signature version for API Request (s3,v4)
enable_transfer_acceleration string No - If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket
ssl_verify_peer string No - If false, the certificate of endpoint will not be verified
proxy_uri string No - URI of proxy environment
grant_read_acp string No - Allows grantee to read the object ACL
check_object string No - Check object before creation
sse_customer_algorithm string No - Specifies the algorithm to use to when encrypting the object
use_server_side_encryption string No - The Server-side encryption algorithm used when storing this object in S3 (AES256, aws:kms)
s3_region string No - S3 region name
acl string No - Permission for the object in S3
grant_full_control string No - Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object
hex_random_length string No - The length of %{hex_random} placeholder(4-16)
s3_object_key_format string No %{path}%{time_slice}_%{index}.%{file_extension} The format of S3 object keys
s3_bucket string Yes - S3 bucket name
store_as string No - Archive format on S3
storage_class string No - The type of storage to use for the object(STANDARD,REDUCED_REDUNDANCY,STANDARD_IA)
aws_iam_retries string No - The number of attempts to load instance profile credentials from the EC2 metadata service using IAM role
buffer *Buffer No - Buffer
format *Format No - Format
assume_role_credentials *S3AssumeRoleCredentials No - Assume Role Credentials
instance_profile_credentials *S3InstanceProfileCredentials No - Instance Profile Credentials
shared_credentials *S3SharedCredentials No - Shared Credentials

Assume Role Credentials

assume_role_credentials

Variable Name Type Required Default Description
role_arn string Yes - The Amazon Resource Name (ARN) of the role to assume
role_session_name string Yes - An identifier for the assumed role session
policy string No - An IAM policy in JSON format
duration_seconds string No - The duration, in seconds, of the role session (900-3600)
external_id string No - A unique identifier that is used by third parties when assuming roles in their customers' accounts.

Instance Profile Credentials

instance_profile_credentials

Variable Name Type Required Default Description
ip_address string No 169.254.169.254 IP address
port string No 80 Port number
http_open_timeout string No - Number of seconds to wait for the connection to open
http_read_timeout string No - Number of seconds to wait for one block to be read
retries string No - Number of times to retry when retrieving credentials

Shared Credentials

shared_credentials

Variable Name Type Required Default Description
profile_name string No - Profile name. Default to 'default' or ENV['AWS_PROFILE']
path string No $HOME/.aws/credentials Path to the shared file.