Avoiding bot detection on Amazon?

[jbun007]

It was initially working intermittently. Now seem to get consistent error responses.
We aren't running the scraper at scale - rather triggered by a user action.

Wonder if this related to proxy rotation frequency?
Is it possible to force proxy rotation with every request?

Any guidance would be appreciated

Code:

Deno.serve(async (req) => {
  const { url } = await req.json();
  const params = { "limit": 2 };

  const client = new Spider({
    apiKey: Deno.env.get('SPIDER_LOCAL_API_KEY')
  });

  try {
    const result = await client.scrapeUrl(url, params);
    console.log("Crawl result received");

    const cleanedHtml = cleanHtml(result[0].content);

    return new Response(
      JSON.stringify({ cleanedHtml }),
      { headers: { "Content-Type": "application/json" } },
    );
  } catch (error) {
    console.error("Error crawling URL:", error);
    return new Response(
      JSON.stringify({ error: error.message }),
      { status: 500, headers: { "Content-Type": "application/json" } }
    );
  }
})

Response:

{"cleanedHtml":"<!-- To discuss automated access to Amazon data please contact [email protected]. For information about migrating to our APIs refer to our Marketplace APIs at https://developer.amazonservices.com/ref=rm_c_sv, or our Product Advertising API at https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html/ref=rm_c_ac for advertising use cases. --> <!-- Correios.DoNotSend --> <div> <div> <div><i></i></div> <div> <div> <i></i> <h4>Enter the characters you see below</h4> <p>Sorry, we just need to make sure you're not a robot. For best results, please make sure your browser is accepting cookies.</p> ......"}%

[jbun007]

Updated config:

const params = { "limit": 2,
"request": "chrome",
"stealth": true,
"proxy_enabled": true,
"respect_robots": false
};

appears to work now

[jbun007]

@j-mendez
Works on first try usually but when retrying using the same product link - I get the same bot detected response

Do you have any suggestions on client config or request params to improve reliability?
Thx v much