Sourcemaps were uploaded but Sentry displays only generated JavaScript code

[DMW007]

Since I can't get Sentry/Raven work with TypeScript on NodeJS, I decided to create a small testproject. The progress is now that uploading sourcemaps to Sentry seems to work. At least the Releases tab show me that we have the main js file generated by Webpack and the corresponding sourcemap as you can see here:

But it appears that those sourcemap is never used by Sentry. After provoking an exception, it shows me only the generated js:

According to the ts source code, we see that TypeScript -stuff like the interface or strongly typed parameters are missing. This would be even more clear if the target in tsconfig.json is changed from es6 (default in raven-node TypeScript documentation) to the old es5. Now we see more ugly prototype definitions or pseudo-classes created by functions, which doesn't exist in Typescript:

Strange is, that sentry say the error happens in index.ts as you can see at the headline of the issue:

But in the exception it refers to dist/main.js as you can see in the other screenshot above.

UPDATE Found out that there's another way how we can easily see that sourcemaps doesn't work. Simply run webpack in production mode (In my case, I changed this in package.json). Sentry displays the messy minified code:

What exactly did I do?

For this test-case, I used the official documentations how to use raven-node together with Typescript:

• https://docs.sentry.io/clients/node/config/
• https://docs.sentry.io/clients/node/typescript/
• https://docs.sentry.io/clients/node/sourcemaps/#raven-node-sourcemaps
• https://docs.sentry.io/learn/cli/configuration/#properties-files

According to the docs, a new project was created to make sure, that nothing of the existing stuff in my project cause trouble, which prevent Sentry from work correctly. Altogether, about one day was spend on this topic by me. Now a point is reached, where I think this is a bug, when it doesn't work on a new project as expected.

The test project was launched with the following command:

export RELEASE=0.4 && npm run build && npm start

Environment

I used a local installation of sentry 8.22 on Docker. It runs a main sentry server (default entrypoint), a cron (sentry run cron) and one worker (sentry run worker). All use the same sentry:8.22 image. The database is a postgres-container, like recommended in the installation-guide.

Configuration of my testproject

src/index.ts

import * as Raven from 'raven'
import * as path from 'path'
import { env } from 'process'

declare global {
    namespace NodeJS {
        interface Global {
            __rootdir__: string;
        }
    }
}

global.__rootdir__ = __dirname || process.cwd();
const root = global.__rootdir__;
console.log(`Release: ${process.env.RELEASE}`)

let options = {
    release: process.env.RELEASE,
    dataCallback: function (data) {
        var stacktrace = data.exception && data.exception[0].stacktrace;

        if (stacktrace && stacktrace.frames) {
            stacktrace.frames.forEach(function (frame) {
                let old = frame.filename
                if (frame.filename.startsWith('/')) {
                    frame.filename = "app:///" + path.relative(root, frame.filename);
                }
            });
        }
        return data;
    }
}
let dn = 'http://<xyz>@<host>:9000/2'
Raven.config(dn, options).install();
interface Test {
    name: string
}
class TestClass {
    testVar = 1
    foo(bar: string, test: Test) {
        console.log(test.name)
        throw new Error('foo called with bar = ' + bar)
    }
}
let test = new TestClass()
let testInterface: Test = { name: 'abc' }
test.foo('Test', testInterface)

tsconfig.json

For simplicity, I combined tsconfig.json and tsconfig.production.json, which are divided in the documentation. Shouldn't make a difference since the production config inherit from the base one.

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "allowJs": true,
    "moduleResolution": "node",
    "outDir": "dist",
    "sourceMap": true,
    "inlineSources": true,
    "sourceRoot": "/"
  },
  "include": [
    "./src/**/*"
  ]
}

webpack.config.json

const path = require('path')
const fs = require('fs')
const SentryPlugin = require('@sentry/webpack-plugin')

var nodeModules = {};
fs.readdirSync('node_modules')
    .filter(function (x) {
        return ['.bin'].indexOf(x) === -1;
    })
    .forEach(function (mod) {
        nodeModules[mod] = 'commonjs ' + mod;
    });

module.exports = {
    entry: './src/index.ts',
    target: 'node',
    externals: nodeModules,
    devtool: 'source-map',

    module: {
        rules: [
            {
                test: /\.tsx?$/,
                use: 'ts-loader',
                exclude: /node_modules/
            }
        ]
    },
    resolve: {
        extensions: ['.tsx', '.ts', '.js']
    },
    output: {
        filename: '[name].js',
        path: path.resolve(__dirname, 'dist')
    },

    plugins: [
        // https://github.com/getsentry/sentry-webpack-plugin
        new SentryPlugin({
            organization: '',
            release: process.env.RELEASE,
            mode: 'development',
            configFile: 'sentry.properties',
            include: './dist',
            ignore: ['node_modules', 'webpack.config.js'],
        })
    ]
};

sentry.properties

This is for the sentry-cli, which is used internally by the webpack plugin. auth.token has project.write permission, see Uploading Source Maps to Sentry in the docs.

defaults.url=http://<sentryhost>:9000
defaults.org=test
defaults.project=nodejs-test
# Generated by http://>sentryhost>:9000/api/ 
auth.token=3046b4383db1436ab6c132b16e8b4995be68a40a6f654686937a2b18c1521911

package.json

{
  "name": "sentry-test-app",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "build": "webpack --mode development",
    "start": "node dist/main.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "raven": "^2.4.2",
    "typescript": "^2.7.2"
  },
  "devDependencies": {
    "@sentry/webpack-plugin": "^1.3.3",
    "@types/node": "^9.4.6",
    "@types/raven": "^2.1.5",
    "ts-loader": "^4.0.1",
    "webpack": "^4.0.1",
    "webpack-cli": "^2.0.10"
  }
}

[DMW007]

Since the creation of this issue, I did some troubleshooting by myself. I put it on a comment since the issue itself is already long cause of the config files. Found out that the app-root directory isn't determined correctly:

global.__rootdir__ = __dirname || process.cwd();
const root = global.__rootdir__;

In my testapp, root is equal to /. That's the reason why I get long paths starting from my /home directory. The sentry documentation mentioned that node doesn't provide a robust solution for this. __dirname seems the problem, since process.cwd(); gives me the full path to my project-directory. So I simplified the line above to const root = process.cwd().

Now the paths are relative to the project root. Like app:///dist/main.js instead of app:////home/user/.[...]/dist/main.js. This has made Sentry's mappings in the issue-reports cleaner. And they come closer to the uploaded paths in the release, which seems important for me. I'm not sure if they've to match exactly. According to the documentation this doesn't seem to be the case, since their code in dataCallback generates relative paths to the project root with app:/// as prefix:

frame.filename = "app:///" + path.relative(root, frame.filename);

I also learned that the SentryPlugin has an option called urlPrefix which is set to ~/ per default. This results in paths like ~/dist/main.js, which can be confirmed in the Releases (menu Artifacts). SentryPlugin automatically generates paths with include dir as root. So if we do this:

new SentryPlugin({
        release: process.env.RELEASE,
        include: '.',
        ignore: ['node_modules', 'webpack.config.js'],
})

We get ~/dist/main.js (of course, when target dir is set to dist in webpack). Since it seems that the original ts sourcefiles aren't relly required cause of the maps, it make sense to point this path to dist: include: "./dist". Correspondingly, this results in ~/main.js as target path in the release. I played around with this options, let them match with and without dist folder. And also with/without app:/// prefix (which seems only possible to set in dataCallback event, since SentryPlugins drops additionally slashes at the end). Altough, sourcemaps doesn't work.

Sourcemaps itself seems to work

I also took a look in the generated sourcemap dist/main.js.map to search for any conspicuous things inside. The "sourcesContent" attribute contains the original TypeScript code. It contains the correct code, e.g. my interface

interface Test {\n    name: string\n}

or the correctly typed definition for our foo method:

class TestClass {\n    testVar = 1\n    foo(bar: string, test: Test) { [...]

Since I haven't worked much using sourcefiles yet, I found a good article in the sentry blog: https://blog.sentry.io/2015/10/29/debuggable-javascript-with-source-maps It explains the attributes and what exactly kind of data they contains. There is one thing which can be responsible for the issue: Our sourcemap refers to the file it belongs using "file":"main.js" but sourceRoot itself is empty. Having the suspect that Sentry maybe can't match the source file to the sourcemap, since we have main.js inside the source file, but a path like ~/main.js or even ~/dist/main.js. I additionally tested /main.js to see if the ~ has any effect. Maybe this has to match exactly, so that e.g. ~/dist/main.js file only matches on a sourcemap which contains "file": "~/dist/main.js".

After spending over 10 hours on this issue, I've enough from raven-node for today. Furthermore, it doesn't seem trivial to change this path automatically using webpack. It could be easier to manipulate the .map file by hand and upload it using sentry-cli directly for testing purpose. Maybe I'll try it tomorrow.

[DMW007]

To exclude further sources of error, I created a plain raw js project. The sentry webpack-plugin doesn't work like before, so I wrote a script (release.sh) to push it manually using sentry-cli:

SENTRY_PROPERTIES=$(pwd)/sentry.properties

cli=./node_modules/.bin/sentry-cli
distFile=app

$cli releases new "$RELEASE"
#$cli releases files "$RELEASE" upload dist/${distFile}.js "${distFile}.js"
#$cli releases files "$RELEASE" upload dist/${distFile}.js.map "${distFile}.js.map"
#$cli releases files "$RELEASE" upload dist/${distFile}.js
$cli releases files "$RELEASE" upload-sourcemaps dist

$cli releases finalize "$RELEASE"

I tried different things, also custom names. Some example are in the commented-out lines. The cli tool opens the possibility to set custom aliases for uploading, like without the leading ~ so that e.g. ~/app.js is simply uploaded as app.js. Altouth different combinations doesn't seem to work.

Another thing I have noticed is the entry mapping:

entry: {
        "app": './src/app.js'
    }

The sentry documentation of webpack seems wrong here, since it provides an absolute path without the ./ to make it relative from the current root. Using

entry: {
      "app": 'src/app.js'
    }

from the documentation, we get

ERROR in Entry module not found: Error: Can't resolve 'src/app.js' in '/sentry-js-test'

But we've a mapping from app here, so I suggest sentry use this key to map sourcefiles with leading app:/// relative paths inside. Altouth the tests doesn't show any difference here. Additionally, I tried some other paths in the map file itself. For example, we've webpack prefixes:

"sources":[  
      "webpack:///webpack/bootstrap",
      "webpack:///external \"path\"",
      "webpack:///external \"raven\"",
      "webpack:///./src/index.js",
      "webpack:///./node_modules/@sentry/webpack-plugin/src/sentry-webpack.module.js"
   ],

Also the index.js path was changed to index.js directly instead of ./src/index.js (cause index.js is uploaded directly to sentry). Sentry doesn't care about this, too.

Start the test project

I used

export RELEASE=3.0.49 && ./release.sh && npm run build && node dist/app.js

after every change, RELEASE is incremented to make sure that the new generated js and map file is uploaded (the cli shows an error that the files already exists when publishing to an existing release). Another difference to my typescript setup is that webpack runs in production mode. So my source get minified, which let me clearly see if sourcemaps works or not.

Sadly, again, nothing works. I get messy code without any mapping from the provided sourcemap:

It's really frustrating, that after investing over 2 days in this topic, no progress is visible. Sentry is a great tool, used this before in C# where we don't have the problem to mess with generated js which is different from the original sourcecode. Unfortunately, sourcemaps seems to be very unstable in sentry, so that it's currently unsuitable for serious nodejs projects. Altough I hope that this get fixed, so that it at least work in a plain raw hello world js app according to the docs.

Source files

src/app.js

var Raven = require('raven')
const path = require('path')

console.log(`Release: ${process.env.RELEASE}`)
let options = {
    release: process.env.RELEASE,
    dataCallback: function (data) {
        var stacktrace = data.exception && data.exception[0].stacktrace;

        if (stacktrace && stacktrace.frames) {
            stacktrace.frames.forEach(function (frame) {
                if (frame.filename.startsWith('/')) {
                    frame.filename = 'app:///' + path.basename(frame.filename);
                    console.log(`File: ${frame.filename}`)
                }
            });
        }

        return data;
    }
}
Raven.config('http://<XYZ>@<SentryHost>:9000/3', options).install();
//Raven.captureException(new Error('abc'))

throw new Error('Hello123')

webpack.config.js

const path = require('path')
const fs = require('fs')
const SentryPlugin = require('@sentry/webpack-plugin')

var nodeModules = {};
fs.readdirSync('node_modules')
    .filter(function (x) {
        return ['.bin'].indexOf(x) === -1;
    })
    .forEach(function (mod) {
        nodeModules[mod] = 'commonjs ' + mod;
    });

module.exports = {
    target: 'node',
    devtool: 'source-map',
    entry: {
        "app": './src/app.js'
    },
    output: {
        path: path.join(__dirname, 'dist'),
        filename: '[name].js'
    }
    /*
    plugins: [
        new SentryPlugin({
            release: process.env.RELEASE,
            configFile: 'sentry.properties',
            include: './dist',
            ignore: ['node_modules', 'webpack.config.js'],
        })
    ]*/
};

sentry.properties

Same as from my typescript testproject of the last post, but with wnother project key defaults.project. I created a new one to make sure that we've a clean state.

[DMW007]

Found a post on the Sentry forum where somebody found out that the worker-container needs access to Sentrys file volume. I had the volume only mapped to the main Sentry container. So I tried to map it for the worker as well. Makes no difference.

Here is my docker-compose.yml

version: '2'
networks:
  my-network:
    external: true
volumes:
  sentry-files:
services: 
  sentry:
    image: sentry:8.22
    mem_limit: 2GB
    volumes:
      - sentry-files:/var/lib/sentry/files
    networks:
      - my-network
    env_file: .env
    ports:
      - 9000:9000
    depends_on:
      - postgres
      - redis

  sentry-cron:
    image: sentry:8.22
    entrypoint: 'sentry run cron'
    mem_limit: 512MB
    volumes:
      - sentry-files:/var/lib/sentry/files
    networks:
      - my-network
    env_file: .env
    depends_on:
      - postgres
      - redis

  sentry-worker:
    image: sentry:8.22
    entrypoint: 'sentry run worker --loglevel DEBUG'
    mem_limit: 512MB
    env_file: .env
    volumes:
      - sentry-files:/var/lib/sentry/files
    networks:
      - my-network
    environment:
      - C_FORCE_ROOT=1
    depends_on:
      - postgres
      - redis
      - sentry

  postgres:
    image: postgres
    mem_limit: 256MB
    env_file: .env
    volumes:
      - ./data/pgdata:/var/lib/postgresql/data
    networks:
      - my-network
    environment:
      - "POSTGRES_USER=${SENTRY_DB_USER}"
      - "POSTGRES_PASSWORD=${SENTRY_DB_PASSWORD}"

  redis:
    image: redis
    mem_limit: 256MB
    volumes:
      - ./data/redis:/data
    networks:
      - my-network

Additionally, I also tried the official docker-compose from Sentry on-premise, but both doesn't work as well. To make sure that they're run a clean state after my changes, Sentry was re-installed (deleted all containers and volumes so that the initialization-wizard runs again).

[kamilogorek]

@DMW007 thanks for the report and your patience. If I understand correctly, everything that you described here is only relatable to on-premise instances of Sentry? Have you tried to use our hosted solution to see whether it makes any difference and determine the exact source of the issue?

Would it be possible for you to create a public repo with repro-case that I could use to debug this issue?

[DMW007]

@kamilogorek I'm sorry for the late reply. I was on the go for a workshop and currently working on my exam project. To answer your questions: Yes, I'm using the on-premise version. We (my company and also me personally) don't like cloud solutions for security/customization reasons.

If there is a free trial I could try this as well, but only for isolating the issue. I suggest you deploy the same core engine of sentry in your cloud, maybe with some extensions to optimize it as SaaS solution.

I'll create a clean example project as I find time on github, so could evaluate the issue there.

[kamilogorek]

@DMW007 Sentry is free at the lowest tier, so feel free to use it :) https://sentry.io/pricing/
I'd appreciate isolating the issue, no rush. Thanks!

[DMW007]

@kamilogorek Exam is almost done now, so I was able to create another clean testproject and push it to github: https://github.com/DMW007/sentry-typescript-sourcemaps-bug-poc
The sentry folder contains all docker-relevant files used to setup my test environment where raven-app is a simple node.js/typescript application which provoke an exception in typescript code.

I'll try setting up a cloud account and do some tests later.

[DMW007]

I tried the same example from my git repo on the sentry cloud and still got no match between generated js file and typescript sourcecode:

[DMW007]

Hmm it seems like that Sentry cloud has more issues as the local one. My release was created but no files were uploaded. I tried to make everything by hand and saw that the cli got an error

failure on authentication: not a JSON response

Testcommand:
npx sentry-cli --log-level=debug --url=http://sentry.io --auth-token=mytoken info

Response:

[INFO] sentry_cli::api request GET http://sentry.io/api/0/
[INFO] sentry_cli::api using token authentication
[INFO] sentry_cli::api > GET /api/0/ HTTP/1.1
[INFO] sentry_cli::api > Host: sentry.io
[INFO] sentry_cli::api > Accept: /
[INFO] sentry_cli::api > Connection: TE
[INFO] sentry_cli::api > TE: gzip
[INFO] sentry_cli::api > User-Agent: sentry-cli/1.31.0
[INFO] sentry_cli::api > Authorization: Bearer
[INFO] sentry_cli::api < HTTP/1.1 301 Moved Permanently
[INFO] sentry_cli::api < Server: nginx
[INFO] sentry_cli::api < Date: Thu, 17 May 2018 18:10:33 GMT
[INFO] sentry_cli::api < Content-Type: text/html
[INFO] sentry_cli::api < Content-Length: 178
[INFO] sentry_cli::api < Connection: keep-alive
[INFO] sentry_cli::api < Location: https://sentry.io/api/0/
[INFO] sentry_cli::api < X-Frame-Options: SAMEORIGIN
[INFO] sentry_cli::api < X-Content-Type-Options: nosniff
[INFO] sentry_cli::api < X-XSS-Protection: 1; mode=block
[INFO] sentry_cli::api response: 301
[INFO] sentry_cli::api body:

<title>301 Moved Permanently</title>

301 Moved Permanently

---

nginx

So I tried

npx sentry-cli --log-level=debug --url=http://sentry.io/api/0/ --auth-token=mytoken info

But still no luck:

failure on authentication: request failed because API URL was incorrectly formatted

I'll give it up for now.

[DMW007]

Found out that those strange rediects were caused by using the http url and was fixed using --url=https://sentry.io/

Using the following code, the mapping works on my example project:

export RELEASE=0.13
npm run build

args="--auth-token=mytoken --url=https://sentry.io/"
npx sentry-cli $args releases new ${RELEASE}
npx sentry-cli $args releases files ${RELEASE} upload dist/main.js '~/main.js'
npx sentry-cli $args releases files ${RELEASE} upload-sourcemaps dist

npm start

After dozen of hours, this was the only way it works using Sentry. So I did the same thing on my local installation, and same issue again: Sentry shows me the compiled js files instead of the typescript code using sourcemaps.

I even tried it again with our official getsentry onpremise example docker-compose file and exactly the same bug: Sourcemap files were uploaded and accessable (found an issue that they're not downloadable when volumes aren't mapped correctly to all containers, so I tested to download them manually which works).

Also tried uploading sourcemaps with --rewrite flag: npx sentry-cli $args releases files ${RELEASE} upload-sourcemaps dist --rewrite according to this post: https://forum.sentry.io/t/uploading-sourcemaps-without-js-files-using-cli/1739/2

Nothing works, and I'm running out of ideas what can be done to find/fix the big issue behind those failtures.

[marcelgoya]

Hey @kamilogorek, any news on this? I'm having the same problem. It's been two months since this issue has been opened and @DMW007 even created a repo for you to test, Plus I've spent almost a day myself trying to figure out whats wrong. Cheers

[marcelgoya]

@kamilogorek I've managed to fix the problem, simply by using my sentry.io account. Therefore, could it be possible that this feature is not available in the latest sentry open source release or could it be some problem with the artifact upload? I'm asking because I had problems with the artifacts upload on a previous version and it also didn't generate any errors.

[DMW007]

@marcelgoya That's not a real fix since sentry.io provide only a hosted solution with correspondingly disadvantages. I don't know about such restrictions in the OS version. And I also found multiple posts from other users, who work with things like sourcemaps. The artifacts upload seems to work, at least in general. I could download the files and they seems okay.

So imho our problem is located at the point, where sentry maps the source code from an error report to his artifacts library. I also took a short look in the source code to find out where this happens. But I haven't much experience in Python and also not in the development of sentry. So it seems that it would cost much time to get started with some debugging here for me, which is not suiteable. I have other things to do and already invested a lot of time in trying different things and document all the results in this issue.

In my point of view, it's strange that the cloud version works nearly out of the box, when speaking from the sourcemaps (I had instead other trouble as said in my previous comments, but that's not the main topic here). I would assume that sentry maintains a single codebase, which is the OS version. This is forked to realize special requirements on the cloud version like linking to customers/bills and similar things.

So why do we have such a big problem here? Since I tried a lot of things deeply and couldn't find any workaround, I feel save to say that this seems not to be a small bug.