Fastify is not instrumented in combination with node profiling and tracesSampleRate #13662

Xhale1 · 2024-09-10T21:40:50Z

Is there an existing issue for this?

I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
I have reviewed the documentation https://docs.sentry.io/
I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node

SDK Version

8.30.0

Framework Version

nodejs 20.17.0

Link to Sentry event

No response

Reproduction Example/SDK Setup

I made a repo here to demo the issue: https://github.com/Xhale1/sentry-repro

Steps to Reproduce

Clone the repo above
Add your own sentry dsn
Run the dev script
Observe that the error only occurs when both integrations: [nodeProfilingIntegration()] and tracesSampleRate: 1 are set. Removing one or the other seems to make the error go away.

Expected Result

Fastify is instrumented properly.

Actual Result

Sentry prints the following to the console:

[Sentry] fastify is not instrumented. This is likely because you required/imported fastify before calling `Sentry.init()`.

The text was updated successfully, but these errors were encountered:

andreiborza · 2024-09-11T04:52:14Z

Hi @Xhale1, thanks for filing this.

I can confirm this is indeed not working, thank you for the reproduction. We're going to look into this.

One thing to note: The reason you're not seeing the issue when tracesSampleRate is not defined is because we only show this warning when tracing is enabled, but it doesn't mean the app was successfully instrumented.

So definitely some issue with using nodeProfilingIntegration.

cc @JonasBa

andreiborza · 2024-09-11T04:55:58Z

As a workaround, you can leave out the integration inside instrument.ts and set it up inside index.ts

import * as Sentry from "@sentry/node";
import { nodeProfilingIntegration } from "@sentry/profiling-node";
import Fastify from "fastify";

const fastify = Fastify();

Sentry.setupFastifyErrorHandler(fastify);
Sentry.addIntegration(nodeProfilingIntegration())

JonasBa · 2024-09-11T13:34:54Z

@andreiborza @Xhale1 are you seeing any profiling SDK logs if you set debug: true to the SDK init?

Xhale1 · 2024-09-11T16:48:41Z

@JonasBa
With node profiling (reports as not-instrumented)

Sentry Logger [log]: Initializing Sentry: process: 65271, thread: main.
Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: NodeFetch
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariablesAsync
Sentry Logger [log]: Integration installed: Context
Sentry Logger [log]: Integration installed: Modules
Sentry Logger [log]: Integration installed: Express
Sentry Logger [log]: Integration installed: Fastify
Sentry Logger [log]: Integration installed: Graphql
Sentry Logger [log]: Integration installed: Mongo
Sentry Logger [log]: Integration installed: Mongoose
Sentry Logger [log]: Integration installed: Mysql
Sentry Logger [log]: Integration installed: Mysql2
Sentry Logger [log]: Integration installed: Redis
Sentry Logger [log]: Integration installed: Postgres
Sentry Logger [log]: Integration installed: Nest
Sentry Logger [log]: Integration installed: Hapi
Sentry Logger [log]: Integration installed: Koa
Sentry Logger [log]: Integration installed: Connect
Sentry Logger [log]: Integration installed: GenericPool
Sentry Logger [log]: Integration installed: Kafka
Sentry Logger [log]: [Profiling] Profiling integration setup.
Sentry Logger [log]: [Profiling] Continuous profiler mode enabled.
Sentry Logger [log]: Integration installed: ProfilingIntegration
Sentry Logger [log]: Running in CommonJS mode.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for diag v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for trace v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for context v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for propagation v1.9.0.
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'http' }
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'https' }
[Sentry] fastify is not instrumented. This is likely because you required/imported fastify before calling `Sentry.init()`.
Server listening at http://0.0.0.0:3000

Without node profiling (reports as instrumented as far as I'm aware

Sentry Logger [log]: Initializing Sentry: process: 65327, thread: main.
Sentry Logger [log]: Integration installed: InboundFilters
Sentry Logger [log]: Integration installed: FunctionToString
Sentry Logger [log]: Integration installed: LinkedErrors
Sentry Logger [log]: Integration installed: RequestData
Sentry Logger [log]: Integration installed: Console
Sentry Logger [log]: Integration installed: Http
Sentry Logger [log]: Integration installed: NodeFetch
Sentry Logger [log]: Integration installed: OnUncaughtException
Sentry Logger [log]: Integration installed: OnUnhandledRejection
Sentry Logger [log]: Integration installed: ContextLines
Sentry Logger [log]: Integration installed: LocalVariablesAsync
Sentry Logger [log]: Integration installed: Context
Sentry Logger [log]: Integration installed: Express
Sentry Logger [log]: Integration installed: Fastify
Sentry Logger [log]: Integration installed: Graphql
Sentry Logger [log]: Integration installed: Mongo
Sentry Logger [log]: Integration installed: Mongoose
Sentry Logger [log]: Integration installed: Mysql
Sentry Logger [log]: Integration installed: Mysql2
Sentry Logger [log]: Integration installed: Redis
Sentry Logger [log]: Integration installed: Postgres
Sentry Logger [log]: Integration installed: Nest
Sentry Logger [log]: Integration installed: Hapi
Sentry Logger [log]: Integration installed: Koa
Sentry Logger [log]: Integration installed: Connect
Sentry Logger [log]: Integration installed: GenericPool
Sentry Logger [log]: Integration installed: Kafka
Sentry Logger [log]: Running in ESM mode.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for diag v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for trace v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for context v1.9.0.
Sentry Logger [debug]: @opentelemetry/api: Registered a global for propagation v1.9.0.
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'http' }
Sentry Logger [debug]: @opentelemetry/instrumentation-http Applying instrumentation patch for nodejs core module on require hook { module: 'https' }
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Applying instrumentation patch for module on require hook {
  module: 'fastify',
  version: '4.28.1',
  baseDir: '/Users/[...]/Code/sentry-openai/node_modules/.pnpm/[email protected]/node_modules/fastify'
}
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Patching fastify preHandler function
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Patching fastify server.addHook function
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Patching fastify route.handler function
Sentry Logger [debug]: @opentelemetry/instrumentation-fastify Patching fastify route.handler function
Server listening at http://0.0.0.0:3000

It seems that node profiling might force the sdk to run in commonjs mode instead of esm?

JonasBa · 2024-09-11T16:52:19Z

@Xhale1 are you setting profilesSampleRate as well, or is that not defined? I can see the SDK enables continuous profiling mode, which should only be enabled if profilesSampleRate and profilesSampler are not set

Xhale1 · 2024-09-11T16:54:45Z

I was not, but I tried adding profilesSampleRate: 1 to my sdk init and it seems that did not resolve the issue.

Also, I edited my previous reply with additional logs for with-error and without-error

JonasBa · 2024-09-11T17:07:14Z

Let me look at the repo. If you setProfilesSampleRate, then the continuous profiling log should not appear, if it does, then that is an SDK bug. I'm going to investigate

JonasBa · 2024-09-11T17:25:22Z

Ok, so if I set profilesSampleRate to > 0 and curl the endpoint (curl localhost:3000), I can see that the profiler is started and stopped.

// instrument.ts
Sentry.init({
  dsn: "",
  debug: true,
  integrations: [nodeProfilingIntegration()],
  tracesSampleRate: 1,
  profilesSampleRate: 1,
});

Server logs

Server listening at http://0.0.0.0:3000
Sentry Logger [log]: Flushing client reports based on interval.
Sentry Logger [log]: Flushing outcomes...
Sentry Logger [log]: No outcomes to send
Sentry Logger [debug]: @opentelemetry/instrumentation-http http instrumentat
ion incomingRequest
Sentry Logger [log]: [Tracing] Starting sampled root span
  op: < unknown op >
  name: GET
  ID: aacdcc98567c0451
Sentry Logger [log]: [Profiling] started profiling transaction: GET
(node:78766) [FSTDEP017] DeprecationWarning: You are accessing the deprecate
d "request.routerPath" property. Use "request.routeOptions.url" instead. Pro
perty "req.routerPath" will be removed in `fastify@5`.
(Use `node --trace-deprecation ...` to show where the warning was created)
Sentry Logger [log]: [Tracing] Finishing "< unknown op >" root span "GET" wi
th ID aacdcc98567c0451
Sentry Logger [log]: [Profiling] stopped profiling of transaction: GET
Sentry Logger [log]: SpanExporter exported 1 spans, 0 unsent spans remaining
Sentry Logger [log]: [Profiling] max profile duration elapsed, stopping prof
iling for: GET
Sentry Logger [log]: [Profiling] stopped profiling of transaction: GET
Sentry Logger [log]: [Profiling] profiler returned null profile for: GET thi
s may indicate an overlapping span or a call to stopProfiling with a profile
 title that was never started
Sentry Logger [log]: Flushing client reports based on interval.
Sentry Logger [log]: Flushing outcomes...

Did you try sending a request to the server to see if the profiler is started and stopped?

What is happening here in my setup is that the profile is being filtered out by the client (null return log) and not sent to the server, as we require a minimum of 2 samples at ~100hz sampling rate (aka the handler should take 20ms+)

Xhale1 · 2024-09-11T19:10:32Z

I have not tried to verify if profiling or tracing are working as expected or not

Are you saying that the [Sentry] fastify is not instrumented... log is misleading in this case and that the server is being properly instrumented as expected?

andreiborza · 2024-09-12T01:33:35Z

Hm, I don't think it is properly instrumented if you don't see @opentelemetry/instrumentation-fastify Applying instrumentation patch for module on require hook this line.

JonasBa · 2024-09-12T11:49:20Z

@Xhale1 That seems to be the case yes. I dont know exactly what the check relies on, maybe its, because there are no routes defined in the file?

Xhale1 · 2024-09-14T01:12:19Z

I should clarify, I have not confirmed myself whether the warning message is accurate or not.

@JonasBa I experienced this warning message in our internal codebase at trainwell which has a number of routes defined so I don't think that's the case. I did not define routes in my repo to keep things as minimal as possible.

As far as I'm aware my repo follows the Sentry docs for getting started with Fastify closely with the biggest difference being the use of tsup. Are you able to comment on whether my repo seems to be setup appropriately according to the docs?

JonasBa · 2024-09-16T13:53:35Z

@Xhale1 I doubt that node profiling is the issue here tbh, are there any other differences? I can see from the example you shared above that the the successfully instrumented code runs in ESM and not in CJ like the one above it that reports faulty instrumentation.

@andreiborza Any ideas if tsup causes any issues with our instrumentation?

andreiborza · 2024-09-17T01:18:55Z

@Xhale1 hm could be an issue with tsup. I just tried using tsx instead

// package.json
...
scipts: {
   "start": "NODE_OPTIONS='--import ./dist/instrument.js' tsx src/index.ts",
   ...
}

and got successful instrumentation with nodeProfiling. Don't really see why this is the case though, needs further investigation.

Are you able to switch away from tsup for the time being or alternatively add the profiling integration at a later stage as I described before?

ollebergkvist · 2024-09-18T18:04:07Z

Chiming in here as this is not working for us either.

Not working:
"start": "NODE_OPTIONS='--import ./dist/instrument.mjs' node -r source-map-support ./dist/src/index.js" nor
"start": "node --import ./dist/instrument.mjs -r source-map-support ./dist/src/index.js"

When we enable debug mode, we get this from the logs:
[Sentry] fastify is not instrumented. Please make sure to initialize Sentry in a separate file that you --import when running node, see: https://docs.sentry.io/platforms/javascript/guides/fastify/install/esm/.

We compile with tsc (module: "ESNext").

andreiborza · 2024-09-19T02:01:56Z

@ollebergkvist thanks for reporting. Do you have a reproduction repo to investigate this?

spaceemotion · 2024-09-19T12:44:34Z

Just wanted to chime in that I'm seeing the same on a CJS lambda (so it's not just ESM). And I did make sure to have sentry as the very first import in my file. it's all one big .js bundle at the end, not sure if it's related to that.

andreiborza · 2024-09-20T00:16:00Z

Thanks @spaceemotion. Did you try without the profiling integration? Do you still see the warning then?

spaceemotion · 2024-09-20T00:22:36Z

yes. the integration was not bundled and the .node file also not present - and i still got the info.

andreiborza · 2024-09-20T00:31:24Z

@spaceemotion hm, could you provide some more info about your setup? Sounds a bit like a different issue.

You can't just bundle in the profiling because of the profiles. The steps to get that to work are a bit more involved, you can see them here.

We have a task in our backlog to create an profiling layer, but it might take a while before we can tackle that.

getsantry bot added the Waiting for: Product Owner label Sep 10, 2024

github-actions bot added the Package: node Issues related to the Sentry Node SDK label Sep 10, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 11, 2024

getsantry bot added the Waiting for: Product Owner label Sep 11, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 11, 2024

getsantry bot added the Waiting for: Product Owner label Sep 11, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 11, 2024

getsantry bot added the Waiting for: Product Owner label Sep 11, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 12, 2024

getsantry bot added the Waiting for: Product Owner label Sep 14, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 16, 2024

getsantry bot added the Waiting for: Product Owner label Sep 18, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 19, 2024

getsantry bot added the Waiting for: Product Owner label Sep 19, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 20, 2024

getsantry bot added the Waiting for: Product Owner label Sep 20, 2024

getsantry bot removed the Waiting for: Product Owner label Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fastify is not instrumented in combination with node profiling and tracesSampleRate #13662

Fastify is not instrumented in combination with node profiling and tracesSampleRate #13662

Xhale1 commented Sep 10, 2024 •

edited

Loading

andreiborza commented Sep 11, 2024 •

edited

Loading

andreiborza commented Sep 11, 2024

JonasBa commented Sep 11, 2024

Xhale1 commented Sep 11, 2024 •

edited

Loading

JonasBa commented Sep 11, 2024

Xhale1 commented Sep 11, 2024

JonasBa commented Sep 11, 2024

JonasBa commented Sep 11, 2024

Xhale1 commented Sep 11, 2024

andreiborza commented Sep 12, 2024

JonasBa commented Sep 12, 2024

Xhale1 commented Sep 14, 2024 •

edited

Loading

JonasBa commented Sep 16, 2024

andreiborza commented Sep 17, 2024

ollebergkvist commented Sep 18, 2024 •

edited

Loading

andreiborza commented Sep 19, 2024

spaceemotion commented Sep 19, 2024

andreiborza commented Sep 20, 2024

spaceemotion commented Sep 20, 2024

andreiborza commented Sep 20, 2024

Fastify is not instrumented in combination with node profiling and tracesSampleRate #13662

Fastify is not instrumented in combination with node profiling and tracesSampleRate #13662

Comments

Xhale1 commented Sep 10, 2024 • edited Loading

Is there an existing issue for this?

How do you use Sentry?

Which SDK are you using?

SDK Version

Framework Version

Link to Sentry event

Reproduction Example/SDK Setup

Steps to Reproduce

Expected Result

Actual Result

andreiborza commented Sep 11, 2024 • edited Loading

andreiborza commented Sep 11, 2024

JonasBa commented Sep 11, 2024

Xhale1 commented Sep 11, 2024 • edited Loading

JonasBa commented Sep 11, 2024

Xhale1 commented Sep 11, 2024

JonasBa commented Sep 11, 2024

JonasBa commented Sep 11, 2024

Xhale1 commented Sep 11, 2024

andreiborza commented Sep 12, 2024

JonasBa commented Sep 12, 2024

Xhale1 commented Sep 14, 2024 • edited Loading

JonasBa commented Sep 16, 2024

andreiborza commented Sep 17, 2024

ollebergkvist commented Sep 18, 2024 • edited Loading

andreiborza commented Sep 19, 2024

spaceemotion commented Sep 19, 2024

andreiborza commented Sep 20, 2024

spaceemotion commented Sep 20, 2024

andreiborza commented Sep 20, 2024

Xhale1 commented Sep 10, 2024 •

edited

Loading

andreiborza commented Sep 11, 2024 •

edited

Loading

Xhale1 commented Sep 11, 2024 •

edited

Loading

Xhale1 commented Sep 14, 2024 •

edited

Loading

ollebergkvist commented Sep 18, 2024 •

edited

Loading