Skip to content

Tuning SELKS

Jump to bottom
Peter Manev edited this page Aug 18, 2017 · 50 revisions

Tuning, configuration and considerations

Each major component can be independently upgraded of the others(unless there is inter dependency of course for some version).

Tuning Suricata and Elasticsearch is a process and is highly dependent on a number of variables you can find some suggestions further down in this document.

Please carefully consider and test HW needs before putting into production.

SELKS is based on Debian live.

Suricata

Suricata is build with dropping privileges capability.Hyperscan is also available and enabled in suricata Suricata 4.X dev at the moment of this writing - example below) is installed and configured with the following Suricata Configuration (info produced by suricata --build-info command):

root@SELKS:~# suricata --build-info
This is Suricata version 4.0.0-dev (rev b8428378)
Features: NFQ PCAP_SET_BUFF AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT  HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 6.3.0 20170516, C version 199901
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.25, linked against LibHTP v0.5.25

Suricata Configuration:
 AF_PACKET support:                       yes
 PF_RING support:                         no
 NFQueue support:                         yes
 NFLOG support:                           no
 IPFW support:                            no
 Netmap support:                          no
 DAG enabled:                             no
 Napatech enabled:                        no

 Unix socket enabled:                     yes
 Detection enabled:                       yes

 Libmagic support:                        yes
 libnss support:                          yes
 libnspr support:                         yes
 libjansson support:                      yes
 hiredis support:                         no
 hiredis async with libevent:             no
 Prelude support:                         no
 PCRE jit:                                yes
 LUA support:                             yes, through luajit
 libluajit:                               yes
 libgeoip:                                yes
 Non-bundled htp:                         yes
 Old barnyard2 support:                   no
 CUDA enabled:                            no
 Hyperscan support:                       yes
 Libnet support:                          yes

 Rust support (experimental):             no
 Experimental Rust parsers:               no
 Rust strict mode:                        no

 Suricatasc install:                      yes

 Profiling enabled:                       no
 Profiling locks enabled:                 no

Development settings:
 Coccinelle / spatch:                     no
 Unit tests enabled:                      no
 Debug output enabled:                    no
 Debug validation enabled:                no

Generic build parameters:
 Installation prefix:                     /usr
 Configuration directory:                 /etc/suricata/
 Log directory:                           /var/log/suricata/

 --prefix                                 /usr
 --sysconfdir                             /etc
 --localstatedir                          /var

 Host:                                    x86_64-pc-linux-gnu
 Compiler:                                gcc (exec name) / gcc (real)
 GCC Protect enabled:                     yes
 GCC march native enabled:                no
 GCC Profile enabled:                     no
 Position Independent Executable enabled: yes
 CFLAGS                                   -g -O2 -fstack-protector-strong -Wformat -Werror=format-security
 PCAP_CFLAGS                               -I/usr/include
 SECCFLAGS                                -fstack-protector -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security
root@SELKS:~#

Suricata Tuning Considerations

There are a number of things that do come into consideration and do require copious research when tuning your Suricata IDS/IPS - traffic type, size, pps, avg packet size, number/type of rules enabled, runmode/type, HW used/available, NIC and so on. If you are willing and have the time and determination - read SEPTun

Some quick recommendations for better performance on high speed traffic you could consider are adjusting the below values(in /etc/suricata/suricata.yaml):

  • increase the default stream: memcap: and stream: reassembly: memcap: values
  • decrease the default flow-timeouts: (at least for TCP)
  • set checksum-validation: no
  • increase max-pending-packets value
  • change to high profile in -> detect-engine: profile: high
  • increase the request-body-limit: and response-body-limit: values

and further more:

For more information on what and how you are highly advised to refer to the Suricata's documentation page:

Elasticsearch Tuning Considerations

Elasticsearch is capable of digesting of tens of thousands of logs per second and subsequently giving you the power of sorting, searching and filtering of billions of log entries seamlessly with the right set up/tuning. All that with native JSON log importing - a standard output of Suricata IDS/IPS.

When all JSON outputs are enabled in Suricata's /etc/suricata/selks4-addin.yaml on a high speed traffic - that could easily result in hundreds of millions of events per day on a high speed network IDS/IPS-ing and the SELKS default configuration must be adjusted accordingly.

Some quick recommendations for better performance could be to increase the heap size and enable mem locking in /etc/default/elasticsearch (add to the bottom of the file) for example:

ES_HEAP_SIZE=8g
MAX_LOCKED_MEMORY=unlimited

save it and restart systemctl restart elasticsearch

If you have the memory - optimal mem config settings for a one node elasticsearch in SELKS configuration could be:

ES_HEAP_SIZE=30g
MAX_LOCKED_MEMORY=unlimited

NOTE: Please note you will need around 60G RAM for that set up for Elasticsearch - optimal performance.

There is much MUCH more to tuning Elasticsearch than just changing those values. Elasticsearch is a powerful and wonderful Open Source application and much more is needed to be able to understand it and tune it properly. Here are some basic recommendation links:

NOTE: Elasticsearch recommends using Oracle JAVA

In general you can use Elasticsearch to centralize all your logging needs from different systems across your organization and with that in mind - to get actually not only Suricata IDS/IPS data events but also correlate those to other systems logs as well - the pinnacle of it all.

Logstash Tuning Considerations

Below are some links to the official guide for configuration and performance tuning for Logstash :

Clone this wiki locally