help collecting syslog

[chrislawso]

I read instructions for collecting syslog at https://signoz.io/docs/userguide/collecting_syslogs/

I setup and have a running Self-Hosted SigNoz with docker compose following https://signoz.io/docs/install/docker/

I am confused on how to proceed. I have a iot device that outputs a syslog stream and can only specify a destination host ipv4 address but I cannot specify the destination port or udp or tcp protocol. The iot device default will only stream syslog to destination udp 514.
I followed documentation at https://signoz.io/docs/userguide/collecting_syslogs/ and only replaced port 54527 from the instructions to 514.

I perform
docker compose up --force-recreate --build -d
and all the containers successfully startup. Inside the signoz webgui I do not see any of the syslog stream from my iot device appearing.

Am I incorrectly setting up collecting syslog? How can I troubleshoot this and get syslog captured and displayed?

Thank you

[nitishfy]

Hi @chrislawso, can you provide answers to couple of questions:

1. Can you please share your otel-collector file? I assume you know that you've to make changes here.

2. Did you check if all the containers are working well, especially the clickhouse container?

[chrislawso]

Hi,

Thank you very much for replying.

1. Yes, I did read and follow the instructions https://signoz.io/docs/userguide/collecting_syslogs/ under the section titled "Collect Syslogs in Self-Hosted SigNoz"

After I edited the docker-compose.yaml and the otel-collector-config.yaml as instructed in the above link I executed the command:
docker compose up --force-recreate --build -d

I copied the entire otel-collector-config.yaml file from the server to this link https://pastebin.com/NgqDrxJn
Below is the the only section I added below receivers:

  syslog:
    tcp:
      listen_address: "0.0.0.0:514"
    protocol: rfc3164
    location: UTC
    operators:
      - type: move
        from: attributes.message
        to: body

2. Yes, the container using image clickhouse/clickhouse-server:24.1.2-alpine and named signoz-clickhouse displays status up and healthy. However when I review docker logs of this container their are errors displayed. Here is a paste of the entire docker logs of the clickhouse container https://pastebin.com/0Y5mg8EH
   I will also paste the errors below here:

2024.06.13` 04:29:47.063939 [ 47 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, String const&) @ 0x00000000153a1b5f in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a2bbd in /usr/bin/clickhouse
2. Poco::Net::StreamSocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a5296 in /usr/bin/clickhouse
3. Poco::Net::HTTPSession::write(char const*, long) @ 0x00000000153908b3 in /usr/bin/clickhouse
4. Poco::Net::HTTPHeaderIOS::~HTTPHeaderIOS() @ 0x000000001538bbdb in /usr/bin/clickhouse
5. Poco::Net::HTTPHeaderOutputStream::~HTTPHeaderOutputStream() @ 0x000000001538bf1f in /usr/bin/clickhouse
6. DB::HTTPServerResponse::send() @ 0x0000000012942988 in /usr/bin/clickhouse
7. DB::HTTPServerConnection::sendErrorResponse(Poco::Net::HTTPServerSession&, Poco::Net::HTTPResponse::HTTPStatus) @ 0x000000001293ecda in /usr/bin/clickhouse
8. DB::HTTPServerConnection::run() @ 0x000000001293e97b in /usr/bin/clickhouse
9. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
10. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse
11. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
13. ? @ 0x000075ac33f12609
14. ? @ 0x000075ac33e37353
 (version 24.1.2.5 (official build))
2024.06.13 04:29:48.294861 [ 712 ] {8a73e698-39da-4270-96d2-a6b33d71fe70} <Error> executeQuery: Code: 36. DB::Exception: Table doesn't have any table TTL expression, cannot remove. (BAD_ARGUMENTS) (version 24.1.2.5 (official build)) (from 0.0.0.0:0) (in query: /* ddl_entry=query-0000000203 */ ALTER TABLE signoz_metrics.time_series_v2 REMOVE TTL), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
1. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000721a243 in /usr/bin/clickhouse
2. DB::AlterCommands::validate(std::shared_ptr<DB::IStorage> const&, std::shared_ptr<DB::Context const>) const @ 0x0000000011c97fea in /usr/bin/clickhouse
3. DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x0000000011390865 in /usr/bin/clickhouse
4. DB::InterpreterAlterQuery::execute() @ 0x000000001138dd71 in /usr/bin/clickhouse
5. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000011904974 in /usr/bin/clickhouse
6. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001190824a in /usr/bin/clickhouse
7. DB::DDLWorker::tryExecuteQuery(DB::DDLTaskBase&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x0000000010d02188 in /usr/bin/clickhouse
8. DB::DDLWorker::processTask(DB::DDLTaskBase&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x0000000010d00755 in /usr/bin/clickhouse
9. DB::DDLWorker::scheduleTasks(bool) @ 0x0000000010cfdd33 in /usr/bin/clickhouse
10. DB::DDLWorker::runMainThread() @ 0x0000000010cf708e in /usr/bin/clickhouse
11. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010d0ebd4 in /usr/bin/clickhouse
12. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c8ed6fe in /usr/bin/clickhouse
13. ? @ 0x000075ac33f12609
14. ? @ 0x000075ac33e37353

2024.06.13 04:29:48.295231 [ 712 ] {8a73e698-39da-4270-96d2-a6b33d71fe70} <Error> DDLWorker: Query /* ddl_entry=query-0000000203 */ ALTER TABLE signoz_metrics.time_series_v2 REMOVE TTL wasn't finished successfully: Code: 36. DB::Exception: Table doesn't have any table TTL expression, cannot remove. (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
1. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000721a243 in /usr/bin/clickhouse
2. DB::AlterCommands::validate(std::shared_ptr<DB::IStorage> const&, std::shared_ptr<DB::Context const>) const @ 0x0000000011c97fea in /usr/bin/clickhouse
3. DB::InterpreterAlterQuery::executeToTable(DB::ASTAlterQuery const&) @ 0x0000000011390865 in /usr/bin/clickhouse
4. DB::InterpreterAlterQuery::execute() @ 0x000000001138dd71 in /usr/bin/clickhouse
5. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000011904974 in /usr/bin/clickhouse
6. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000001190824a in /usr/bin/clickhouse
7. DB::DDLWorker::tryExecuteQuery(DB::DDLTaskBase&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x0000000010d02188 in /usr/bin/clickhouse
8. DB::DDLWorker::processTask(DB::DDLTaskBase&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x0000000010d00755 in /usr/bin/clickhouse
9. DB::DDLWorker::scheduleTasks(bool) @ 0x0000000010cfdd33 in /usr/bin/clickhouse
10. DB::DDLWorker::runMainThread() @ 0x0000000010cf708e in /usr/bin/clickhouse
11. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<void (DB::DDLWorker::*)(), DB::DDLWorker*>(void (DB::DDLWorker::*&&)(), DB::DDLWorker*&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000010d0ebd4 in /usr/bin/clickhouse
12. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c8ed6fe in /usr/bin/clickhouse
13. ? @ 0x000075ac33f12609
14. ? @ 0x000075ac33e37353
 (version 24.1.2.5 (official build))
2024.06.13 04:29:48.337203 [ 49 ] {c59756f1-cdaf-44d7-ae2e-20c39ffa5ede} <Error> executeQuery: Code: 36. DB::Exception: There was an error on [clickhouse:9000]: Code: 36. DB::Exception: Table doesn't have any table TTL expression, cannot remove. (BAD_ARGUMENTS) (version 24.1.2.5 (official build)). (BAD_ARGUMENTS) (version 24.1.2.5 (official build)) (from 172.18.0.6:56228) (in query: ALTER TABLE signoz_metrics.time_series_v2 ON CLUSTER cluster REMOVE TTL;), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
1. DB::DDLQueryStatusSource::generate() @ 0x00000000118efe71 in /usr/bin/clickhouse
2. DB::ISource::tryGenerate() @ 0x000000001297acf5 in /usr/bin/clickhouse
3. DB::ISource::work() @ 0x000000001297a743 in /usr/bin/clickhouse
4. DB::ExecutionThreadContext::executeTask() @ 0x000000001299371a in /usr/bin/clickhouse
5. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001298a170 in /usr/bin/clickhouse
6. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000012989380 in /usr/bin/clickhouse
7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000129970a3 in /usr/bin/clickhouse
8. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c8ed6fe in /usr/bin/clickhouse
9. ? @ 0x000075ac33f12609
10. ? @ 0x000075ac33e37353

2024.06.13 04:29:48.337364 [ 49 ] {c59756f1-cdaf-44d7-ae2e-20c39ffa5ede} <Error> TCPHandler: Code: 36. DB::Exception: There was an error on [clickhouse:9000]: Code: 36. DB::Exception: Table doesn't have any table TTL expression, cannot remove. (BAD_ARGUMENTS) (version 24.1.2.5 (official build)). (BAD_ARGUMENTS), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in /usr/bin/clickhouse
1. DB::DDLQueryStatusSource::generate() @ 0x00000000118efe71 in /usr/bin/clickhouse
2. DB::ISource::tryGenerate() @ 0x000000001297acf5 in /usr/bin/clickhouse
3. DB::ISource::work() @ 0x000000001297a743 in /usr/bin/clickhouse
4. DB::ExecutionThreadContext::executeTask() @ 0x000000001299371a in /usr/bin/clickhouse
5. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001298a170 in /usr/bin/clickhouse
6. DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000012989380 in /usr/bin/clickhouse
7. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000129970a3 in /usr/bin/clickhouse
8. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c8ed6fe in /usr/bin/clickhouse
9. ? @ 0x000075ac33f12609
10. ? @ 0x000075ac33e37353

2024.06.13 04:30:48.155981 [ 47 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, String const&) @ 0x00000000153a1b5f in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a2bbd in /usr/bin/clickhouse
2. Poco::Net::StreamSocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a5296 in /usr/bin/clickhouse
3. Poco::Net::HTTPSession::write(char const*, long) @ 0x00000000153908b3 in /usr/bin/clickhouse
4. Poco::Net::HTTPHeaderIOS::~HTTPHeaderIOS() @ 0x000000001538bbdb in /usr/bin/clickhouse
5. Poco::Net::HTTPHeaderOutputStream::~HTTPHeaderOutputStream() @ 0x000000001538bf1f in /usr/bin/clickhouse
6. DB::HTTPServerResponse::send() @ 0x0000000012942988 in /usr/bin/clickhouse
7. DB::HTTPServerConnection::sendErrorResponse(Poco::Net::HTTPServerSession&, Poco::Net::HTTPResponse::HTTPStatus) @ 0x000000001293ecda in /usr/bin/clickhouse
8. DB::HTTPServerConnection::run() @ 0x000000001293e97b in /usr/bin/clickhouse
9. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
10. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse
11. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse
13. ? @ 0x000075ac33f12609
14. ? @ 0x000075ac33e37353
 (version 24.1.2.5 (official build))
2024.06.13 04:33:48.581377 [ 47 ] {} <Error> ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 32, I/O error: Broken pipe, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, String const&) @ 0x00000000153a1b5f in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a2bbd in /usr/bin/clickhouse
2. Poco::Net::StreamSocketImpl::sendBytes(void const*, int, int) @ 0x00000000153a5296 in /usr/bin/clickhouse
3. Poco::Net::HTTPSession::write(char const*, long) @ 0x00000000153908b3 in /usr/bin/clickhouse
4. Poco::Net::HTTPHeaderIOS::~HTTPHeaderIOS() @ 0x000000001538bbdb in /usr/bin/clickhouse
5. Poco::Net::HTTPHeaderOutputStream::~HTTPHeaderOutputStream() @ 0x000000001538bf1f in /usr/bin/clickhouse
6. DB::HTTPServerResponse::send() @ 0x0000000012942988 in /usr/bin/clickhouse
7. DB::HTTPServerConnection::sendErrorResponse(Poco::Net::HTTPServerSession&, Poco::Net::HTTPResponse::HTTPStatus) @ 0x000000001293ecda in /usr/bin/clickhouse
8. DB::HTTPServerConnection::run() @ 0x000000001293e97b in /usr/bin/clickhouse
9. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in /usr/bin/clickhouse
10. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in /usr/bin/clickhouse
11. Poco::PooledThread::run() @ 0x000000001549f047 in /usr/bin/clickhouse
12. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in /usr/bin/clickhouse

[nitishfy]

Since your IoT device sends logs via UDP and you cannot change this on the device, you need to adjust your OpenTelemetry Collector configuration to listen for UDP instead of TCP. Update your otel-collector-config.yaml to include a UDP listener for syslog:

yaml
receivers:
  syslog:
    udp:
      listen_address: "0.0.0.0:514"
    protocol: rfc3164
    location: UTC

Also, make sure that your Docker compose configuration exposes UDP port 514 so that logs can be received from your IoT device. Here’s how you can expose UDP ports in your Docker Compose file:

yaml
services:
  otel-collector:
    ports:
      - "514:514/udp"