Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: Error when querying data #483

Open
Skryabind opened this issue Aug 27, 2023 · 7 comments
Open

Bug: Error when querying data #483

Skryabind opened this issue Aug 27, 2023 · 7 comments
Labels
bug Something isn't working

Comments

@Skryabind
Copy link

I tested Parseable and sent full http error object to stream. It led to create many fields in the stream, and after that I can't fetch data from this stream. I see in logs (when run a query from the parseable console):

2023-08-27 21:12:18 thread 'actix-rt|system:0|arbiter:2' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ComputeError("concat requires input of at least one array"))', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/datafusion-common-27.0.0/src/scalar.rs:2835:18

I can't append full stream schema to due to limit of issue body, but it's realy huge.
@nitisht
Copy link
Member

nitisht commented Aug 28, 2023

Thanks for reporting @Skryabind let us check and get back to you.

@nitisht nitisht added the triage label Aug 28, 2023
@trueleo
Copy link
Contributor

trueleo commented Aug 28, 2023

@Skryabind I recognize this error because we have ran into this couple of times. I created an issue for this while back over here. apache/datafusion#5706

I need to check though if this is the same issue. The issue mostly stems from null array fields in data or very nested struct type.

Can i know few things about your setup.

  • full http error object - Is this an object directly dumped from a program?

@Skryabind
Copy link
Author

Skryabind commented Aug 28, 2023 •
edited by theteachr
Loading

@trueleo yes, I just sent full error object to log. The code was like this (in nodejs):

try {
  axios.get(....)
} catch (e) {
  log.error(e)
}

So log was:

{
  "address": "127.0.0.1",
  "code": "ECONNREFUSED",
  "config": {
    "baseURL": "http://127.0.0.1:4522/v1",
    "env": {},
    "headers": {
      "Accept": "application/json, text/plain, */*",
      "User-Agent": "axios/0.27.2"
    },
    "maxBodyLength": 1000000000,
    "maxContentLength": 100000000,
    "method": "get",
    "params": {
      "integrationIds": []
    },
    "timeout": 0,
    "transformRequest": [
      null
    ],
    "transformResponse": [
      null
    ],
    "transitional": {
      "clarifyTimeoutError": false,
      "forcedJSONParsing": true,
      "silentJSONParsing": true
    },
    "url": "/company/53/messages/unread",
    "xsrfCookieName": "XSRF-TOKEN",
    "xsrfHeaderName": "X-XSRF-TOKEN"
  },
  "errno": -61,
  "level": "error",
  "message": "connect ECONNREFUSED 127.0.0.1:4522",
  "name": "Error",
  "port": 4522,
  "request": {
    "_currentRequest": {
      "_closed": false,
      "_contentLength": 0,
      "_defaultKeepAlive": true,
      "_ended": false,
      "_events": {},
      "_eventsCount": 7,
      "_hasBody": true,
      "_header": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
      "_headerSent": true,
      "_keepAliveTimeout": 0,
      "_last": true,
      "_redirectable": "[Circular]",
      "_removedConnection": false,
      "_removedContLen": false,
      "_removedTE": false,
      "_trailer": "",
      "aborted": false,
      "agent": {
        "_events": {},
        "_eventsCount": 2,
        "defaultPort": 80,
        "freeSockets": {},
        "keepAlive": false,
        "keepAliveMsecs": 1000,
        "maxFreeSockets": 256,
        "maxSockets": null,
        "maxTotalSockets": null,
        "options": {
          "path": null
        },
        "protocol": "http:",
        "requests": {},
        "scheduling": "lifo",
        "sockets": {
          "127.0.0.1:4522:": [
            {
              "_closeAfterHandlingError": false,
              "_events": {
                "close": [
                  null,
                  null
                ],
                "connect": [
                  null,
                  null,
                  null
                ]
              },
              "_eventsCount": 8,
              "_hadError": true,
              "_host": null,
              "_httpMessage": "[Circular]",
              "_parent": null,
              "_pendingData": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
              "_pendingEncoding": "latin1",
              "_readableState": {
                "autoDestroy": true,
                "awaitDrainWriters": null,
                "buffer": {
                  "head": null,
                  "length": 0,
                  "tail": null
                },
                "closeEmitted": true,
                "closed": true,
                "constructed": true,
                "dataEmitted": false,
                "decoder": null,
                "defaultEncoding": "utf8",
                "destroyed": true,
                "emitClose": false,
                "emittedReadable": false,
                "encoding": null,
                "endEmitted": false,
                "ended": false,
                "errorEmitted": true,
                "errored": {
                  "address": "127.0.0.1",
                  "code": "ECONNREFUSED",
                  "errno": -61,
                  "port": 4522,
                  "syscall": "connect"
                },
                "flowing": true,
                "highWaterMark": 16384,
                "length": 0,
                "multiAwaitDrain": false,
                "needReadable": true,
                "objectMode": false,
                "pipes": [],
                "readableListening": false,
                "reading": true,
                "readingMore": false,
                "resumeScheduled": false,
                "sync": false
              },
              "_server": null,
              "_sockname": null,
              "_writableState": {
                "afterWriteTickInfo": null,
                "allBuffers": true,
                "allNoop": true,
                "autoDestroy": true,
                "bufferProcessing": false,
                "buffered": [],
                "bufferedIndex": 0,
                "closeEmitted": true,
                "closed": true,
                "constructed": true,
                "corked": 0,
                "decodeStrings": false,
                "defaultEncoding": "utf8",
                "destroyed": true,
                "emitClose": false,
                "ended": false,
                "ending": false,
                "errorEmitted": true,
                "errored": {
                  "address": "127.0.0.1",
                  "code": "ECONNREFUSED",
                  "errno": -61,
                  "port": 4522,
                  "syscall": "connect"
                },
                "finalCalled": false,
                "finished": false,
                "highWaterMark": 16384,
                "length": 157,
                "needDrain": false,
                "objectMode": false,
                "pendingcb": 1,
                "prefinished": false,
                "sync": false,
                "writelen": 157,
                "writing": true
              },
              "allowHalfOpen": false,
              "connecting": false,
              "parser": null,
              "server": null
            }
          ]
        },
        "totalSocketCount": 1
      },
      "chunkedEncoding": false,
      "destroyed": false,
      "finished": true,
      "host": "127.0.0.1",
      "maxHeadersCount": null,
      "maxRequestsOnConnectionReached": false,
      "method": "GET",
      "outputData": [],
      "outputSize": 0,
      "parser": null,
      "path": "/v1/company/53/messages/unread",
      "protocol": "http:",
      "res": null,
      "reusedSocket": false,
      "sendDate": false,
      "shouldKeepAlive": false,
      "socket": {
        "_closeAfterHandlingError": false,
        "_events": {
          "close": [
            null,
            null
          ],
          "connect": [
            null,
            null,
            null
          ]
        },
        "_eventsCount": 8,
        "_hadError": true,
        "_host": null,
        "_httpMessage": "[Circular]",
        "_parent": null,
        "_pendingData": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
        "_pendingEncoding": "latin1",
        "_readableState": {
          "autoDestroy": true,
          "awaitDrainWriters": null,
          "buffer": {
            "head": null,
            "length": 0,
            "tail": null
          },
          "closeEmitted": true,
          "closed": true,
          "constructed": true,
          "dataEmitted": false,
          "decoder": null,
          "defaultEncoding": "utf8",
          "destroyed": true,
          "emitClose": false,
          "emittedReadable": false,
          "encoding": null,
          "endEmitted": false,
          "ended": false,
          "errorEmitted": true,
          "errored": {
            "address": "127.0.0.1",
            "code": "ECONNREFUSED",
            "errno": -61,
            "port": 4522,
            "syscall": "connect"
          },
          "flowing": true,
          "highWaterMark": 16384,
          "length": 0,
          "multiAwaitDrain": false,
          "needReadable": true,
          "objectMode": false,
          "pipes": [],
          "readableListening": false,
          "reading": true,
          "readingMore": false,
          "resumeScheduled": false,
          "sync": false
        },
        "_server": null,
        "_sockname": null,
        "_writableState": {
          "afterWriteTickInfo": null,
          "allBuffers": true,
          "allNoop": true,
          "autoDestroy": true,
          "bufferProcessing": false,
          "buffered": [],
          "bufferedIndex": 0,
          "closeEmitted": true,
          "closed": true,
          "constructed": true,
          "corked": 0,
          "decodeStrings": false,
          "defaultEncoding": "utf8",
          "destroyed": true,
          "emitClose": false,
          "ended": false,
          "ending": false,
          "errorEmitted": true,
          "errored": {
            "address": "127.0.0.1",
            "code": "ECONNREFUSED",
            "errno": -61,
            "port": 4522,
            "syscall": "connect"
          },
          "finalCalled": false,
          "finished": false,
          "highWaterMark": 16384,
          "length": 157,
          "needDrain": false,
          "objectMode": false,
          "pendingcb": 1,
          "prefinished": false,
          "sync": false,
          "writelen": 157,
          "writing": true
        },
        "allowHalfOpen": false,
        "connecting": false,
        "parser": null,
        "server": null
      },
      "strictContentLength": false,
      "timeoutCb": null,
      "upgradeOrConnect": false,
      "useChunkedEncodingByDefault": false,
      "writable": true
    },
    "_currentUrl": "http://127.0.0.1:4522/v1/company/53/messages/unread",
    "_ended": true,
    "_ending": true,
    "_events": {},
    "_eventsCount": 3,
    "_options": {
      "agents": {},
      "headers": {
        "Accept": "application/json, text/plain, */*",
        "User-Agent": "axios/0.27.2"
      },
      "hostname": "127.0.0.1",
      "maxBodyLength": 1000000000,
      "maxRedirects": 21,
      "method": "GET",
      "nativeProtocols": {
        "http:": {
          "METHODS": [
            "ACL",
            "BIND",
            "CHECKOUT",
            "CONNECT",
            "COPY",
            "DELETE",
            "GET",
            "HEAD",
            "LINK",
            "LOCK",
            "M-SEARCH",
            "MERGE",
            "MKACTIVITY",
            "MKCALENDAR",
            "MKCOL",
            "MOVE",
            "NOTIFY",
            "OPTIONS",
            "PATCH",
            "POST",
            "PROPFIND",
            "PROPPATCH",
            "PURGE",
            "PUT",
            "REBIND",
            "REPORT",
            "SEARCH",
            "SOURCE",
            "SUBSCRIBE",
            "TRACE",
            "UNBIND",
            "UNLINK",
            "UNLOCK",
            "UNSUBSCRIBE"
          ],
          "STATUS_CODES": {
            "100": "Continue",
            "101": "Switching Protocols",
            "102": "Processing",
            "103": "Early Hints",
            "200": "OK",
            "201": "Created",
            "202": "Accepted",
            "203": "Non-Authoritative Information",
            "204": "No Content",
            "205": "Reset Content",
            "206": "Partial Content",
            "207": "Multi-Status",
            "208": "Already Reported",
            "226": "IM Used",
            "300": "Multiple Choices",
            "301": "Moved Permanently",
            "302": "Found",
            "303": "See Other",
            "304": "Not Modified",
            "305": "Use Proxy",
            "307": "Temporary Redirect",
            "308": "Permanent Redirect",
            "400": "Bad Request",
            "401": "Unauthorized",
            "402": "Payment Required",
            "403": "Forbidden",
            "404": "Not Found",
            "405": "Method Not Allowed",
            "406": "Not Acceptable",
            "407": "Proxy Authentication Required",
            "408": "Request Timeout",
            "409": "Conflict",
            "410": "Gone",
            "411": "Length Required",
            "412": "Precondition Failed",
            "413": "Payload Too Large",
            "414": "URI Too Long",
            "415": "Unsupported Media Type",
            "416": "Range Not Satisfiable",
            "417": "Expectation Failed",
            "418": "I'm a Teapot",
            "421": "Misdirected Request",
            "422": "Unprocessable Entity",
            "423": "Locked",
            "424": "Failed Dependency",
            "425": "Too Early",
            "426": "Upgrade Required",
            "428": "Precondition Required",
            "429": "Too Many Requests",
            "431": "Request Header Fields Too Large",
            "451": "Unavailable For Legal Reasons",
            "500": "Internal Server Error",
            "501": "Not Implemented",
            "502": "Bad Gateway",
            "503": "Service Unavailable",
            "504": "Gateway Timeout",
            "505": "HTTP Version Not Supported",
            "506": "Variant Also Negotiates",
            "507": "Insufficient Storage",
            "508": "Loop Detected",
            "509": "Bandwidth Limit Exceeded",
            "510": "Not Extended",
            "511": "Network Authentication Required"
          },
          "globalAgent": {
            "_events": {},
            "_eventsCount": 2,
            "defaultPort": 80,
            "freeSockets": {},
            "keepAlive": false,
            "keepAliveMsecs": 1000,
            "maxFreeSockets": 256,
            "maxSockets": null,
            "maxTotalSockets": null,
            "options": {
              "path": null
            },
            "protocol": "http:",
            "requests": {},
            "scheduling": "lifo",
            "sockets": {
              "127.0.0.1:4522:": [
                {
                  "_closeAfterHandlingError": false,
                  "_events": {
                    "close": [
                      null,
                      null
                    ],
                    "connect": [
                      null,
                      null,
                      null
                    ]
                  },
                  "_eventsCount": 8,
                  "_hadError": true,
                  "_host": null,
                  "_httpMessage": {
                    "_closed": false,
                    "_contentLength": 0,
                    "_defaultKeepAlive": true,
                    "_ended": false,
                    "_events": {},
                    "_eventsCount": 7,
                    "_hasBody": true,
                    "_header": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
                    "_headerSent": true,
                    "_keepAliveTimeout": 0,
                    "_last": true,
                    "_redirectable": "[Circular]",
                    "_removedConnection": false,
                    "_removedContLen": false,
                    "_removedTE": false,
                    "_trailer": "",
                    "aborted": false,
                    "agent": "[Circular]",
                    "chunkedEncoding": false,
                    "destroyed": false,
                    "finished": true,
                    "host": "127.0.0.1",
                    "maxHeadersCount": null,
                    "maxRequestsOnConnectionReached": false,
                    "method": "GET",
                    "outputData": [],
                    "outputSize": 0,
                    "parser": null,
                    "path": "/v1/company/53/messages/unread",
                    "protocol": "http:",
                    "res": null,
                    "reusedSocket": false,
                    "sendDate": false,
                    "shouldKeepAlive": false,
                    "socket": "[Circular]",
                    "strictContentLength": false,
                    "timeoutCb": null,
                    "upgradeOrConnect": false,
                    "useChunkedEncodingByDefault": false,
                    "writable": true
                  },
                  "_parent": null,
                  "_pendingData": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
                  "_pendingEncoding": "latin1",
                  "_readableState": {
                    "autoDestroy": true,
                    "awaitDrainWriters": null,
                    "buffer": {
                      "head": null,
                      "length": 0,
                      "tail": null
                    },
                    "closeEmitted": true,
                    "closed": true,
                    "constructed": true,
                    "dataEmitted": false,
                    "decoder": null,
                    "defaultEncoding": "utf8",
                    "destroyed": true,
                    "emitClose": false,
                    "emittedReadable": false,
                    "encoding": null,
                    "endEmitted": false,
                    "ended": false,
                    "errorEmitted": true,
                    "errored": {
                      "address": "127.0.0.1",
                      "code": "ECONNREFUSED",
                      "errno": -61,
                      "port": 4522,
                      "syscall": "connect"
                    },
                    "flowing": true,
                    "highWaterMark": 16384,
                    "length": 0,
                    "multiAwaitDrain": false,
                    "needReadable": true,
                    "objectMode": false,
                    "pipes": [],
                    "readableListening": false,
                    "reading": true,
                    "readingMore": false,
                    "resumeScheduled": false,
                    "sync": false
                  },
                  "_server": null,
                  "_sockname": null,
                  "_writableState": {
                    "afterWriteTickInfo": null,
                    "allBuffers": true,
                    "allNoop": true,
                    "autoDestroy": true,
                    "bufferProcessing": false,
                    "buffered": [],
                    "bufferedIndex": 0,
                    "closeEmitted": true,
                    "closed": true,
                    "constructed": true,
                    "corked": 0,
                    "decodeStrings": false,
                    "defaultEncoding": "utf8",
                    "destroyed": true,
                    "emitClose": false,
                    "ended": false,
                    "ending": false,
                    "errorEmitted": true,
                    "errored": {
                      "address": "127.0.0.1",
                      "code": "ECONNREFUSED",
                      "errno": -61,
                      "port": 4522,
                      "syscall": "connect"
                    },
                    "finalCalled": false,
                    "finished": false,
                    "highWaterMark": 16384,
                    "length": 157,
                    "needDrain": false,
                    "objectMode": false,
                    "pendingcb": 1,
                    "prefinished": false,
                    "sync": false,
                    "writelen": 157,
                    "writing": true
                  },
                  "allowHalfOpen": false,
                  "connecting": false,
                  "parser": null,
                  "server": null
                }
              ]
            },
            "totalSocketCount": 1
          },
          "maxHeaderSize": 16384
        },
        "https:": {
          "globalAgent": {
            "_events": {},
            "_eventsCount": 2,
            "_sessionCache": {
              "list": [],
              "map": {}
            },
            "defaultPort": 443,
            "freeSockets": {},
            "keepAlive": false,
            "keepAliveMsecs": 1000,
            "maxCachedSessions": 100,
            "maxFreeSockets": 256,
            "maxSockets": null,
            "maxTotalSockets": null,
            "options": {
              "path": null
            },
            "protocol": "https:",
            "requests": {},
            "scheduling": "lifo",
            "sockets": {},
            "totalSocketCount": 0
          }
        }
      },
      "path": "/v1/company/53/messages/unread",
      "pathname": "/v1/company/53/messages/unread",
      "port": "4522",
      "protocol": "http:"
    },
    "_redirectCount": 0,
    "_redirects": [],
    "_requestBodyBuffers": [],
    "_requestBodyLength": 0,
    "_writableState": {
      "afterWriteTickInfo": null,
      "allBuffers": true,
      "allNoop": true,
      "autoDestroy": true,
      "bufferProcessing": false,
      "buffered": [],
      "bufferedIndex": 0,
      "closeEmitted": false,
      "closed": false,
      "constructed": true,
      "corked": 0,
      "decodeStrings": true,
      "defaultEncoding": "utf8",
      "destroyed": false,
      "emitClose": true,
      "ended": false,
      "ending": false,
      "errorEmitted": false,
      "errored": null,
      "finalCalled": false,
      "finished": false,
      "highWaterMark": 16384,
      "length": 0,
      "needDrain": false,
      "objectMode": false,
      "pendingcb": 0,
      "prefinished": false,
      "sync": true,
      "writecb": null,
      "writelen": 0,
      "writing": false
    }
  },
  "service": "app",
  "stack": "Error: connect ECONNREFUSED 127.0.0.1:4522\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1278:16)",
  "syscall": "connect",
  "timestamp": "2023-08-27 19:03:35"
}

@Skryabind
Copy link
Author

@trueleo I've just confirmed: If you send the object from my previous message to stream, You'll get this error.

@trueleo
Copy link
Contributor

trueleo commented Aug 28, 2023 •
edited
Loading

@Skryabind Yeah unfortunately this exact json won't work with Parseable as of today. The data is very nested with chain of struct of list of structs. We use parquet as our underlying format which is schema sensitive.

But I am assuming you want to log api errors and stuff which you can do in an alternative way. You just need to remove noise from the error you are throwing and have it how you want to look at it later on.

I will recommend looking at a logging library for node which will format and send the logs to Parseable. If the logging library does not support http / json or requires a specific connector logic to Parseable we can look into it.

We also have a basic example for sending logs in JS but you will need to hook this logic to your node server which i am unsure of how to do. https://www.parseable.io/docs/log-ingestion/applications/javascript

@Skryabind
Copy link
Author

@trueleo I agree that this object is not realy sutable for log. In real projects I use winston logger and send more informative and short logs. Now I'm looking for a new logging system and testing Parseable in different ways. I think that the thing that sending "incorrect" log breaks access to the stream (include other normal logs) is not good. Maybe you can reject such logs in an insert stage if you know limitations to avoid latter problems with stream.

@nitisht
Copy link
Member

nitisht commented Sep 3, 2023 •
edited
Loading

Thanks @Skryabind I agree we should avoid ingesting logs that can potentially break the stream. We’ll add a fix

@nitisht nitisht added this to the Release v0.7.1 milestone Sep 19, 2023
@nitisht nitisht added bug Something isn't working and removed triage labels Sep 22, 2023
@nitisht nitisht removed this from the Release v0.7.1 milestone Dec 8, 2023
@nitisht nitisht assigned theteachr and unassigned trueleo Dec 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Skryabind @nitisht @trueleo @theteachr