-
Notifications
You must be signed in to change notification settings - Fork 515
[amazon_bedrock_agentcore] add runtime_application_logs data stream
#15991
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gpop63
wants to merge
22
commits into
elastic:main
Choose a base branch
from
gpop63:runtime-logs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,753
−265
Open
Changes from all commits
Commits
Show all changes
22 commits
Select commit
Hold shift + click to select a range
4cdbe25
Amazon Bedrock AgentCore new integration package
agithomas 21ddd0e
Updated PR link, removed unused configs
agithomas 9182cf0
Merge branch 'main' into issue-obs-integration-670
agithomas 4d06423
Updated dashboard conditions
agithomas c9faea5
Updated the readme with the name of the agentcore components
agithomas a953aa9
add logs data stream
gpop63 c873bed
update dashboard
gpop63 bbe0c10
address comments
gpop63 2fc9ee4
merge
gpop63 39996cd
update
gpop63 0a43cbb
remove sample logo
gpop63 3088a7a
fix policy
gpop63 0ebfadc
add observability category
gpop63 53ee3b9
address comments
gpop63 e7bb9a9
fix pipeline indenting
gpop63 05b69ff
improve field descriptions
gpop63 5c2712d
fix sample event
gpop63 67dce02
update sample event
gpop63 920216e
add aws-s3 input
gpop63 391937f
improve policy templates descriptions
gpop63 a404df2
fix pr id
gpop63 cb1d145
update dashboard img
gpop63 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 change: 1 addition & 0 deletions
1
...re/data_stream/runtime_application_logs/_dev/test/pipeline/test-aws-bedrock-agentcore.log
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| {"resource_arn": "arn:aws:bedrock-agentcore:us-east-1:627286350134:runtime/claudeserver-CdBoW2FLP0", "event_timestamp": 1762144239907, "account_id": "627286350134", "request_id": "136b5865-b303-43a7-b737-3fed1ba17341", "session_id": "090ab333-40ff-49cd-ade1-1527b3a8ede7", "span_id": "3f2b2909093d27f3", "trace_id": "69082fe91a0ed1872de8e57378fe229b", "service_name": "AgentCoreCodeRuntime", "operation": "InvokeAgentRuntime", "request_payload":{"prompt": "What is this agent about, this is claudserver?"}} |
56 changes: 56 additions & 0 deletions
56
.../runtime_application_logs/_dev/test/pipeline/test-aws-bedrock-agentcore.log-expected.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| { | ||
| "expected": [ | ||
| { | ||
| "@timestamp": "2025-11-03T04:30:39.907Z", | ||
| "aws": { | ||
| "bedrock_agentcore": { | ||
| "account_id": "627286350134", | ||
| "request_id": "136b5865-b303-43a7-b737-3fed1ba17341", | ||
| "resource_arn": "arn:aws:bedrock-agentcore:us-east-1:627286350134:runtime/claudeserver-CdBoW2FLP0" | ||
| } | ||
| }, | ||
| "cloud": { | ||
| "account": { | ||
| "id": "627286350134" | ||
| }, | ||
| "provider": "aws", | ||
| "service": { | ||
| "name": "bedrock-agentcore" | ||
| } | ||
| }, | ||
| "ecs": { | ||
| "version": "8.11.0" | ||
| }, | ||
| "event": { | ||
| "original": "{\"resource_arn\": \"arn:aws:bedrock-agentcore:us-east-1:627286350134:runtime/claudeserver-CdBoW2FLP0\", \"event_timestamp\": 1762144239907, \"account_id\": \"627286350134\", \"request_id\": \"136b5865-b303-43a7-b737-3fed1ba17341\", \"session_id\": \"090ab333-40ff-49cd-ade1-1527b3a8ede7\", \"span_id\": \"3f2b2909093d27f3\", \"trace_id\": \"69082fe91a0ed1872de8e57378fe229b\", \"service_name\": \"AgentCoreCodeRuntime\", \"operation\": \"InvokeAgentRuntime\", \"request_payload\":{\"prompt\": \"What is this agent about, this is claudserver?\"}}", | ||
| "outcome": "success" | ||
| }, | ||
| "gen_ai": { | ||
| "conversation": { | ||
| "id": "090ab333-40ff-49cd-ade1-1527b3a8ede7" | ||
| }, | ||
| "operation": { | ||
| "name": "InvokeAgentRuntime" | ||
| }, | ||
| "prompt": "What is this agent about, this is claudserver?", | ||
| "provider": { | ||
| "name": "aws" | ||
| }, | ||
| "system": "aws_bedrock_agentcore" | ||
| }, | ||
| "service": { | ||
| "name": "AgentCoreCodeRuntime" | ||
| }, | ||
| "span": { | ||
| "id": "3f2b2909093d27f3" | ||
| }, | ||
| "tags": [ | ||
| "preserve_original_event", | ||
| "preserve_duplicate_custom_fields" | ||
| ], | ||
| "trace": { | ||
| "id": "69082fe91a0ed1872de8e57378fe229b" | ||
| } | ||
| } | ||
| ] | ||
| } |
5 changes: 5 additions & 0 deletions
5
..._agentcore/data_stream/runtime_application_logs/_dev/test/pipeline/test-common-config.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| fields: | ||
| tags: | ||
| - preserve_original_event | ||
| - preserve_duplicate_custom_fields |
106 changes: 106 additions & 0 deletions
106
...edrock_agentcore/data_stream/runtime_application_logs/agent/stream/aws-cloudwatch.yml.hbs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| {{#unless log_group_name}} | ||
| {{#unless log_group_name_prefix}} | ||
| {{#if log_group_arn }} | ||
| log_group_arn: {{ log_group_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| {{#unless log_group_name}} | ||
| {{#if log_group_name_prefix }} | ||
| log_group_name_prefix: {{ log_group_name_prefix }} | ||
| {{/if}} | ||
| {{#if include_linked_accounts_with_prefix }} | ||
| include_linked_accounts_for_prefix_mode: {{ include_linked_accounts_with_prefix }} | ||
| {{/if}} | ||
| {{#if number_of_workers }} | ||
| number_of_workers: {{ number_of_workers }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| {{#unless log_group_name_prefix}} | ||
| {{#if log_group_name }} | ||
| log_group_name: {{ log_group_name }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_group_arn}} | ||
| region_name: {{ region_name }} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_stream_prefix}} | ||
| {{#if log_streams }} | ||
| log_streams: {{ log_streams }} | ||
| {{/if}} | ||
| {{/unless}} | ||
|
|
||
| {{#unless log_streams}} | ||
| {{#if log_stream_prefix }} | ||
| log_stream_prefix: {{ log_stream_prefix }} | ||
| {{/if}} | ||
| {{/unless}} | ||
|
|
||
| {{#if start_position }} | ||
| start_position: {{ start_position }} | ||
| {{/if}} | ||
|
|
||
| {{#if scan_frequency }} | ||
| scan_frequency: {{ scan_frequency }} | ||
| {{/if}} | ||
|
|
||
| {{#if api_sleep }} | ||
| api_sleep: {{ api_sleep }} | ||
| {{/if}} | ||
|
|
||
| {{#if latency }} | ||
| latency: {{ latency }} | ||
| {{/if}} | ||
|
|
||
| {{#if credential_profile_name}} | ||
| credential_profile_name: {{credential_profile_name}} | ||
| {{/if}} | ||
| {{#if shared_credential_file}} | ||
| shared_credential_file: {{shared_credential_file}} | ||
| {{/if}} | ||
| {{#if api_timeout}} | ||
| api_timeout: {{api_timeout}} | ||
| {{/if}} | ||
| {{#if default_region}} | ||
| default_region: {{default_region}} | ||
| {{/if}} | ||
| {{#if access_key_id}} | ||
| access_key_id: {{access_key_id}} | ||
| {{/if}} | ||
| {{#if secret_access_key}} | ||
| secret_access_key: {{secret_access_key}} | ||
| {{/if}} | ||
| {{#if session_token}} | ||
| session_token: {{session_token}} | ||
| {{/if}} | ||
| {{#if role_arn}} | ||
| role_arn: {{role_arn}} | ||
| {{/if}} | ||
| {{#if proxy_url }} | ||
| proxy_url: {{proxy_url}} | ||
| {{/if}} | ||
| tags: | ||
| {{#if preserve_original_event}} | ||
| - preserve_original_event | ||
| {{/if}} | ||
| {{#if preserve_duplicate_custom_fields}} | ||
| - preserve_duplicate_custom_fields | ||
| {{/if}} | ||
| {{#each tags as |tag|}} | ||
| - {{tag}} | ||
| {{/each}} | ||
| {{#contains "forwarded" tags}} | ||
| publisher_pipeline.disable_host: true | ||
| {{/contains}} | ||
| {{#if processors}} | ||
| processors: | ||
| {{processors}} | ||
| {{/if}} |
171 changes: 171 additions & 0 deletions
171
...es/aws_bedrock_agentcore/data_stream/runtime_application_logs/agent/stream/aws-s3.yml.hbs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,171 @@ | ||
| {{! The aws-s3 input can be configured to read from an SQS queue or an S3 bucket. }} | ||
|
|
||
| {{! start SQS queue }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#unless access_point_arn}} | ||
| {{#if queue_url }} | ||
| queue_url: {{ queue_url }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
| {{! end SQS queue }} | ||
|
|
||
| {{#unless queue_url}}{{! start S3 bucket polling }} | ||
|
|
||
| {{! | ||
| When using an S3 bucket, you can specify only one of the following options: | ||
| - An AWS bucket ARN | ||
| - A non-AWS bucket name | ||
| }} | ||
|
|
||
| {{! shared S3 bucket polling options }} | ||
| {{#if bucket_list_prefix }} | ||
| bucket_list_prefix: {{ bucket_list_prefix }} | ||
| {{/if}} | ||
|
|
||
| {{#if bucket_list_interval }} | ||
| bucket_list_interval: {{ bucket_list_interval }} | ||
| {{/if}} | ||
|
|
||
| {{#if start_timestamp}} | ||
| start_timestamp: {{start_timestamp}} | ||
| {{/if}} | ||
|
|
||
| {{#if ignore_older}} | ||
| ignore_older: {{ignore_older}} | ||
| {{/if}} | ||
|
|
||
| {{! AWS S3 bucket ARN options }} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#unless access_point_arn}} | ||
| {{#if bucket_arn }} | ||
| bucket_arn: {{ bucket_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{! non-AWS S3 bucket ARN options }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless access_point_arn}} | ||
| {{#if non_aws_bucket_name }} | ||
| non_aws_bucket_name: {{ non_aws_bucket_name }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{! AWS S3 Access Point ARN options }} | ||
| {{#unless bucket_arn}} | ||
| {{#unless non_aws_bucket_name}} | ||
| {{#if access_point_arn }} | ||
| access_point_arn: {{ access_point_arn }} | ||
| {{/if}} | ||
| {{/unless}} | ||
| {{/unless}} | ||
|
|
||
| {{/unless}}{{! end S3 bucket polling }} | ||
|
|
||
| {{! allows number of workers to be configured for SQS queue and S3 buckets}} | ||
| {{#if number_of_workers }} | ||
| number_of_workers: {{ number_of_workers }} | ||
| {{/if}} | ||
|
|
||
| {{#if buffer_size }} | ||
| buffer_size: {{ buffer_size }} | ||
| {{/if}} | ||
| {{#if content_type }} | ||
| content_type: {{ content_type }} | ||
| {{/if}} | ||
| {{#if encoding }} | ||
| encoding: {{ encoding }} | ||
| {{/if}} | ||
| {{#if expand_event_list_from_field }} | ||
| expand_event_list_from_field: {{ expand_event_list_from_field }} | ||
| {{/if}} | ||
| {{#if buffer_size }} | ||
| buffer_size: {{ buffer_size }} | ||
| {{/if}} | ||
| {{#if fips_enabled }} | ||
| fips_enabled: {{ fips_enabled }} | ||
| {{/if}} | ||
| {{#if include_s3_metadata }} | ||
| include_s3_metadata: {{ include_s3_metadata }} | ||
| {{/if}} | ||
| {{#if max_bytes }} | ||
| max_bytes: {{ max_bytes }} | ||
| {{/if}} | ||
| {{#if max_number_of_messages }} | ||
| max_number_of_messages: {{ max_number_of_messages }} | ||
| {{/if}} | ||
| {{#if path_style }} | ||
| path_style: {{ path_style }} | ||
| {{/if}} | ||
| {{#if provider }} | ||
| provider: {{ provider }} | ||
| {{/if}} | ||
| {{#if sqs.max_receive_count }} | ||
| sqs.max_receive_count: {{ sqs.max_receive_count }} | ||
| {{/if}} | ||
| {{#if sqs.wait_time }} | ||
| sqs.wait_time: {{ sqs.wait_time }} | ||
| {{/if}} | ||
|
|
||
| {{#if file_selectors}} | ||
| file_selectors: | ||
| {{file_selectors}} | ||
| {{/if}} | ||
|
|
||
| {{#if credential_profile_name}} | ||
| credential_profile_name: {{credential_profile_name}} | ||
| {{/if}} | ||
| {{#if shared_credential_file}} | ||
| shared_credential_file: {{shared_credential_file}} | ||
| {{/if}} | ||
| {{#if visibility_timeout}} | ||
| visibility_timeout: {{visibility_timeout}} | ||
| {{/if}} | ||
| {{#if api_timeout}} | ||
| api_timeout: {{api_timeout}} | ||
| {{/if}} | ||
| {{#if endpoint}} | ||
| endpoint: {{endpoint}} | ||
| {{/if}} | ||
| {{#if default_region}} | ||
| default_region: {{default_region}} | ||
| {{/if}} | ||
| {{#if access_key_id}} | ||
| access_key_id: {{access_key_id}} | ||
| {{/if}} | ||
| {{#if secret_access_key}} | ||
| secret_access_key: {{secret_access_key}} | ||
| {{/if}} | ||
| {{#if session_token}} | ||
| session_token: {{session_token}} | ||
| {{/if}} | ||
| {{#if role_arn}} | ||
| role_arn: {{role_arn}} | ||
| {{/if}} | ||
| {{#if fips_enabled}} | ||
| fips_enabled: {{fips_enabled}} | ||
| {{/if}} | ||
| {{#if proxy_url }} | ||
| proxy_url: {{proxy_url}} | ||
| {{/if}} | ||
| tags: | ||
| {{#if preserve_original_event}} | ||
| - preserve_original_event | ||
| {{/if}} | ||
| {{#if preserve_duplicate_custom_fields}} | ||
| - preserve_duplicate_custom_fields | ||
| {{/if}} | ||
| {{#each tags as |tag|}} | ||
| - {{tag}} | ||
| {{/each}} | ||
| {{#contains "forwarded" tags}} | ||
| publisher_pipeline.disable_host: true | ||
| {{/contains}} | ||
| {{#if processors}} | ||
| processors: | ||
| {{processors}} | ||
| {{/if}} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think, it will be good to mention the steps needed to fetch the log_group_arn, needed for configuring this dataset?
As it is the same steps for all the log-based datasets, maybe a common section (may be under
What do I need to use this integration?) should suffice.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've added a small section to explain how to get the log group arn 👍