[o365] Fixed parsing and indexing errors #15699

StacieClark-Elastic · 2025-10-20T18:37:47Z

Fixes flattening error in Actions list when the list is encoded json string instead of json objects. Adds fields ActorInfoString, OperationCount, TokenObjectId, TokenTenantId. Added fields Messages and Folders as ExchangeAggregatedMessages and ExchangeAggregatedFolder for record type 50: ExchangeItemAggregated and explicitly convert the SizeInBytes values to long. There are other schemas for fields called Messages and Folders so explicit naming is defensive.

Fixes error messages:

field "o365.audit.ActorInfoString" is undefined
field "o365.audit.Folders" is used as array of objects, expected explicit definition with type group or nested
field "o365.audit.OperationCount" is undefined
field "o365.audit.TokenObjectId" is undefined
field "o365.audit.TokenTenantId" is undefined
failed to parse field [o365.audit.Actions] of type [flattened] in document with id
[o365.audit.Folders.FolderItems.SizeInBytes] cannot be changed from type [long] to [float]
[o365.audit.Messages.MessageItems.SizeInBytes] cannot be changed from type [long] to [float]

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Pipeline tests have examples in added test file that will produce errors before the change.
Passing pipeline tests verify the fix.
Note that the missing fields errors do not occur until earlier errors in the pipeline are fixed.

elasticmachine · 2025-10-20T18:41:30Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-10-20T19:10:58Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

efd6 · 2025-10-20T20:21:21Z

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

        if (!(ctx.o365audit.Actions instanceof List)) {
          ctx.o365audit.Actions = [ctx.o365audit.Actions];
        }
+        def regex = /,\"QueryTime\":\"[0-9\/]+\s[0-9]+:[0-9]+:[0-9]+\s[AP]M\"|\"QueryTime\":\"[0-9\/]+\s[0-9]+:[0-9]+:[0-9]+\s[AP]M\",/;


/,"QueryTime\":"[0-9\/]+\s[0-9]+:[0-9]+:[0-9]+\s[AP]M"|"QueryTime":"[0-9\/]+\s[0-9]+:[0-9]+:[0-9]+\s[AP]M",/ since / has literal string quoting behaviour.

Suggest also s/regex/queryTimePattern/g and adding a comment about why we are doing this.

This is replication of logic that we already have for the Data field.

I added the comment but the kept the way I'm using the regex as this appears to be the documented way to do it.

Documented where?

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

packages/o365/changelog.yml

packages/o365/data_stream/audit/fields/fields.yml

packages/o365/changelog.yml

efd6 · 2025-10-20T20:42:53Z

Please also note the origin of the new test cases in the proposed commit message.

StacieClark-Elastic · 2025-10-21T16:26:34Z

From a security standpoint, I don't think we should commenting on where the bug was found. The SDH points to this PR which should be sufficient. The test case is obviously completely fake data.

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

packages/o365/changelog.yml

packages/o365/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

efd6

nits only, then LGTM

efd6 · 2025-10-23T06:28:42Z

packages/o365/changelog.yml

@@ -1,4 +1,20 @@
 # newer versions go on top
+- version: "2.31.1"


Suggested change

- version: "2.31.1"

- version: "2.32.0"

because enhancement

efd6 · 2025-10-23T06:31:51Z

packages/o365/manifest.yml

 name: o365
 title: Microsoft Office 365
-version: "2.31.0"
+version: "2.31.1"


Suggested change

version: "2.31.1"

version: "2.32.0"

Fixes flattening error in Actions list when the list is encoded json string instead of json objects. Adds fields ActorInfoString, OperationCount, TokenObjectId, TokenTenantId. Added fields Messages and Folders as ExchangeMessages and ExchangeFolder for record type 50: ExchangeItemAggregated

…vert SizeInBytes field in ExchangeMessages and ExchangeFolders to long

…eMessages to ExchangeAggregatedMessages and ExchangeFolders to ExchangeAggregatedFolders

…ders field if they have not been renamed

… major version

elasticmachine · 2025-10-24T20:17:31Z

💚 Build Succeeded

Buildkite Build
Commit: 862e0a5

History

💚 Build #33001 succeeded 73b0ae5
💚 Build #32953 succeeded 46e8afc
💚 Build #32917 succeeded ef709b7

cc @StacieClark-Elastic

StacieClark-Elastic added the Integration:o365 Microsoft Office 365 label Oct 20, 2025

StacieClark-Elastic requested a review from a team as a code owner October 20, 2025 18:37

StacieClark-Elastic added the bugfix Pull request that fixes a bug issue label Oct 20, 2025

StacieClark-Elastic force-pushed the fix-parsing-error-due-to-duplicate-fields branch from 73328d2 to ef709b7 Compare October 20, 2025 18:40

StacieClark-Elastic added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Oct 20, 2025

andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Oct 20, 2025

efd6 reviewed Oct 20, 2025

View reviewed changes

narph assigned StacieClark-Elastic Oct 21, 2025

StacieClark-Elastic force-pushed the fix-parsing-error-due-to-duplicate-fields branch from ef709b7 to 874f5a3 Compare October 21, 2025 16:06

StacieClark-Elastic requested a review from efd6 October 21, 2025 18:45

StacieClark-Elastic enabled auto-merge (squash) October 21, 2025 18:46

efd6 requested changes Oct 21, 2025

View reviewed changes

StacieClark-Elastic added the enhancement New feature or request label Oct 22, 2025

StacieClark-Elastic force-pushed the fix-parsing-error-due-to-duplicate-fields branch from 46e8afc to 73b0ae5 Compare October 22, 2025 13:39

StacieClark-Elastic requested a review from efd6 October 22, 2025 13:39

efd6 disabled auto-merge October 23, 2025 06:31

efd6 approved these changes Oct 23, 2025

View reviewed changes

StacieClark-Elastic added 6 commits October 24, 2025 15:48

fix-parsing-error-due-to-duplicate-fields added correct PR number

cc325a5

fix-parsing-error-due-to-duplicate-fields Added code to explcitly con…

61da5f7

…vert SizeInBytes field in ExchangeMessages and ExchangeFolders to long

fix-parsing-error-due-to-duplicate-fields changed Field names Exchang…

7ec4b31

…eMessages to ExchangeAggregatedMessages and ExchangeFolders to ExchangeAggregatedFolders

fix-parsing-error-due-to-duplicate-fields Don't remove Mesages or Fol…

81cbb55

…ders field if they have not been renamed

fix-parsing-error-due-to-duplicate-fields incremented version to next…

862e0a5

… major version

StacieClark-Elastic force-pushed the fix-parsing-error-due-to-duplicate-fields branch from 73b0ae5 to 862e0a5 Compare October 24, 2025 19:54

StacieClark-Elastic requested a review from efd6 October 24, 2025 19:54

		@@ -1,4 +1,20 @@
		# newer versions go on top
		- version: "2.31.1"

[o365] Fixed parsing and indexing errors #15699

Are you sure you want to change the base?

[o365] Fixed parsing and indexing errors #15699

Uh oh!

Conversation

StacieClark-Elastic commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

How to test this PR locally

Uh oh!

elasticmachine commented Oct 20, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Oct 20, 2025

🚀 Benchmarks report

Uh oh!

efd6 Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

StacieClark-Elastic Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

StacieClark-Elastic Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Oct 20, 2025

Uh oh!

StacieClark-Elastic commented Oct 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

efd6 Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Oct 24, 2025

💚 Build Succeeded

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

StacieClark-Elastic commented Oct 20, 2025 •

edited

Loading

StacieClark-Elastic Oct 21, 2025 •

edited

Loading