🐛 Source Datadog : Fix the Pagination in the logs stream

[VigneshSankar01]

What

This PR addresses an issue with the pagination strategy in the logs stream for the Datadog Marketplace Connector in Airbyte. The previous pagination logic did not properly iterate through all pages, causing incomplete log retrieval. This update ensures that all logs are pulled correctly.

Link to the Issue: #54161

How

The pagination strategy in the manifest.yaml file has been updated from DefaultPaginator to CursorPagination. The updated strategy now:

1. Utilizes links.next from the API response as the cursor_value to fetch the next set of logs.
2. Implements a stop_condition to terminate pagination when no further pages exist.
   This ensures that the logs stream correctly fetches all available logs without missing any records.

Review guide

Key changes are in:

1. manifest.yaml – Updates pagination strategy for the Datadog logs stream.

User Impact

Users will now receive all logs from Datadog without missing any entries due to pagination issues.
The connector will function more reliably for log ingestion.

Can this PR be safely reverted and rolled back?

YES 💚 – The changes are limited to the pagination strategy in manifest.yaml. If needed, reverting this PR would restore the previous pagination behavior.

[vercel]

@VigneshSankar01 is attempting to deploy a commit to the Airbyte Growth Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[natikgadzhi]

/bump-version type="patch"

Bump Version job started... Check job output.

✅ Changes applied successfully. (e48fae1)

[natikgadzhi]

Nice! Thank you.

[VigneshSankar01]

Hi Team, Would these checks take a lot of time to process? It says I would need authorization to deploy Vercel.

Please advice

Thanks!

[VigneshSankar01]

@VigneshSankar01 is attempting to deploy a commit to the Airbyte Growth Team on Vercel.

A member of the Team first needs to authorize it.

Hi Team, requesting approval on this, please

[VigneshSankar01]

@natikgadzhi @ChristoGrab for approval on vercel and community CI, please!

[marcosmarxm]

@VigneshSankar01 please be patience.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 20, 2025 10:34pm