Skip to content

Commit 498eb5e

Browse files
author
Guy Boertje
authored
Add better support for file rotation (#192)
* Add better support for file rotation * Move common_restat for watched and active to one iteration to better handle when a file is rotated that has never been active. Change the way @previous_inode_size is cached, it needs to be set only when an inode change is detected. Change discover associate behaviour if the matched watched_file is open. Add spec for when a file is rotated that has never been active. Alter specs because we can rely on ordering with "file_sort_by" * Ensure single file `path` option is ok, reverts earlier change in this PR * Improve and expand our use of FFI on Windows * jnr and ffi interop, yay * put self back in * use fieldId label * handle rotations better. especially in the case where the rotated files are discoverable - in this case we were not the moving the state and the sincedb record correctly. Next - abstract the *nix and Windows stat calls and structure into two classes with the same API. * fixes for rebase from master * Abstract Stat part 1 * Abstract Stat part 2 * Finally have (all) the kinks worked out * Try to fix travis failures. * Try to fix travis failures 2 * Try fix travis 3 * Try fix travis 4 * Try fix travis 5 * Try fix travis 6 * Remove io based stat reliance. travis jruby 1.7.27 should pass now. Move all tail processing state iteration into its own method Clean up rspec sequencing usage. Remove all Mutex use in favour of AtomicBoolean, AtomicArray, AtomicHash Add wait for completely_stopped before removing sincedb in file_tail spec * Some windows fixes * more windows fixes * windows changes 2 * rename rspec run tag in ci/build.sh * move one trace logging line * add first run discovery methods * fix regression on files seen after inital run, travis 2 use docker. * add execute permissions * fix path ordering travis failures * fix jar loading so it works for tests and when installed in LS * reorder the jar require statements Fixes #198
1 parent c3c7847 commit 498eb5e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+2417
-946
lines changed

‎.travis.yml

Lines changed: 12 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,16 @@
11
---
2-
sudo: false
3-
language: ruby
4-
cache: bundler
2+
sudo: required
3+
services: docker
4+
addons:
5+
apt:
6+
packages:
7+
- docker-ce
58
matrix:
69
include:
7-
- rvm: jruby-9.1.13.0
8-
env: LOGSTASH_BRANCH=master
9-
- rvm: jruby-9.1.13.0
10-
env: LOGSTASH_BRANCH=6.x
11-
- rvm: jruby-9.1.13.0
12-
env: LOGSTASH_BRANCH=6.0
13-
- rvm: jruby-1.7.27
14-
env: LOGSTASH_BRANCH=5.6
10+
- env: ELASTIC_STACK_VERSION=5.6.10
11+
- env: ELASTIC_STACK_VERSION=6.3.0
12+
- env: ELASTIC_STACK_VERSION=6.4.0-SNAPSHOT
13+
- env: ELASTIC_STACK_VERSION=7.0.0-alpha1-SNAPSHOT
1514
fast_finish: true
16-
install: true
17-
script: ci/build.sh
18-
jdk: oraclejdk8
15+
install: ci/unit/docker-setup.sh
16+
script: ci/unit/docker-run.sh

‎JAR_VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
1.0.0
1+
1.0.1

‎README.md

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,6 @@
22
Travis Build
33
[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-file.svg)](https://travis-ci.org/logstash-plugins/logstash-input-file)
44

5-
Jenkins Build
6-
[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-file.svg)](https://travis-ci.org/logstash-plugins/logstash-input-file)
7-
85
This is a plugin for [Logstash](https://github.com/elastic/logstash).
96

107
It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

‎ci/build.sh

Lines changed: 0 additions & 21 deletions
This file was deleted.

‎ci/setup.sh

Lines changed: 0 additions & 26 deletions
This file was deleted.

‎ci/unit/Dockerfile

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
ARG ELASTIC_STACK_VERSION
2+
FROM docker.elastic.co/logstash/logstash:$ELASTIC_STACK_VERSION
3+
WORKDIR /usr/share/logstash/logstash-core
4+
RUN cp versions-gem-copy.yml ../logstash-core-plugin-api/versions-gem-copy.yml
5+
COPY --chown=logstash:logstash . /usr/share/plugins/this
6+
WORKDIR /usr/share/plugins/this
7+
ENV PATH=/usr/share/logstash/vendor/jruby/bin:${PATH}
8+
ENV LOGSTASH_SOURCE 1
9+
RUN jruby -S gem install bundler
10+
RUN jruby -S bundle install --jobs=3 --retry=3
11+
RUN jruby -S bundle exec rake vendor

‎ci/unit/docker-compose.yml

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
version: '3'
2+
3+
# run tests: docker-compose -f ci/unit/docker-compose.yml up --build --force-recreate
4+
# only set up: docker-compose -f ci/unit/docker-compose.yml up --build --no-start --force-recreate
5+
# start manually: docker-compose -f ci/unit/docker-compose.yml run logstash
6+
services:
7+
logstash:
8+
build:
9+
context: ../../
10+
dockerfile: ci/unit/Dockerfile
11+
args:
12+
- ELASTIC_STACK_VERSION=$ELASTIC_STACK_VERSION
13+
command: /usr/share/plugins/this/ci/unit/run.sh
14+
environment:
15+
LS_JAVA_OPTS: "-Xmx256m -Xms256m"
16+
OSS: "true"
17+
tty: true

‎ci/unit/docker-run.sh

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
3+
# This is intended to be run the plugin's root directory. `ci/unit/docker-test.sh`
4+
# Ensure you have Docker installed locally and set the ELASTIC_STACK_VERSION environment variable.
5+
set -e
6+
7+
docker-compose -f ci/unit/docker-compose.yml run logstash

‎ci/unit/docker-setup.sh

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/bin/bash
2+
3+
# This is intended to be run the plugin's root directory. `ci/unit/docker-test.sh`
4+
# Ensure you have Docker installed locally and set the ELASTIC_STACK_VERSION environment variable.
5+
set -e
6+
7+
if [ "$ELASTIC_STACK_VERSION" ]; then
8+
echo "Testing against version: $ELASTIC_STACK_VERSION"
9+
10+
if [[ "$ELASTIC_STACK_VERSION" = *"-SNAPSHOT" ]]; then
11+
cd /tmp
12+
wget https://snapshots.elastic.co/docker/logstash-"$ELASTIC_STACK_VERSION".tar.gz
13+
tar xfvz logstash-"$ELASTIC_STACK_VERSION".tar.gz repositories
14+
echo "Loading docker image: "
15+
cat repositories
16+
docker load < logstash-"$ELASTIC_STACK_VERSION".tar.gz
17+
rm logstash-"$ELASTIC_STACK_VERSION".tar.gz
18+
cd -
19+
fi
20+
21+
if [ -f Gemfile.lock ]; then
22+
rm Gemfile.lock
23+
fi
24+
25+
docker-compose -f ci/unit/docker-compose.yml down
26+
docker-compose -f ci/unit/docker-compose.yml up --no-start --build --force-recreate logstash
27+
else
28+
echo "Please set the ELASTIC_STACK_VERSION environment variable"
29+
echo "For example: export ELASTIC_STACK_VERSION=6.2.4"
30+
exit 1
31+
fi

‎ci/unit/run.sh

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
3+
# This is intended to be run inside the docker container as the command of the docker-compose.
4+
set -ex
5+
6+
bundle exec rspec -fd --pattern spec/**/*_spec.rb,spec/**/*_specs.rb

0 commit comments

Comments
 (0)