inotify monitor doesn't reliably detect fast recursive file creation

[gbroques]

I'm attempting to use this library (on Linux) to recursively watch real time creation of .git/objects when users interact with their repository.

Upon commands such as git add and git commit, git creates subirectories in .git/objects/, and immediately after creates a file in that subdirectory such as .git/objects/47/c83adec4eb349f06d8e5fea7373d97f0e8ed84.

I noticed this library detects this file creation most of the time, but not always.

It's a bit hard to reproduce because I believe it's a race condition.

My understanding of inotify is that it doesn't support recursive watches, so you have to establish watches on subdirectories yourself.

If a file is created very quickly in that newly created subdirectory, then you might miss it because you haven't established the new watch yet.

I opened emcrisostomo/fswatch#330, and their approach was to scan newly watched directories and emit a "synthetic event" following a similar approach to inotify-wait (see inotify-tools/inotify-tools#121).

[SpartanJ]

Hi @gbroques!
I understand your problem and everything makes sense. Yes, events will be triggered depending on how fast the new watcher is created and if it's fast enough to catch the event. Watches are created immediately after the created event is received, but this could be not enough on some occasions. I think that the only solution is the one you chatted about in the other issues, which is creating synthetic (fake) events. I'm honestly not convinced about it as a solution, given that as you already mentioned, it can create duplicated events, and probably quite easily honestly. For example on move operation with multiple sub-directories. efsw also tries to emulate directories move operation since so this could be problematic. I think that, as in the inotify-tools issue is mentioned, this synthetic events could be optional and activated with a flag. Basically after creating the watch and sending the event the library can scan the whole directory tree recursively and emit a create event for each file. But in your case you could just do it yourself given that you know exactly what you're looking for (not telling you to do it, but just clarifying that for your case this is an "easy fix").

Git is particularly very demanding to the file system so this does not surprises me. I, in fact, do watch for events in .git for my Git integration in my code editor, ecode, but I took a different approach for my use case, I just wait for changes in the .git dir tree (except for lock files) and then delegate updating my local git status (and UI) by calling the git binary directly (reference), with the simple catch that I debounce the update call for a second or two (due to exactly what you observed, the amount events triggered in a branch switch can be extreme, I'm talking about thousands of events).

So, in short, if you think this is a solution for your use case I have no problem adding it (it might take a few days though), but let me read a little further if there's a better option or I'm missing something.

[gbroques]

@SpartanJ Thank you for the fast and detailed response. I appreciate those insights!

I'm not exactly sure on the solution, and I appreciate you looking into it.

It would be nice to have more control with an event delivery option of "at-least once" or "at-most once" — if "exactly once" isn't possible.

This problem reminds me a bit of message delivery guarantees in distributed messaging systems, but I'm not sure how much overlap there is with this.

But in your case you could just do it yourself given that you know exactly what you're looking for

I could, but it'd be nice if the cross-platform "file system events" library handled this for me. This seems like a Linux-specific quirk of inotify as macOS / FSEvents doesn't have this issue apparently. I'm not sure about Windows.

Monitoring for .git/objects is just the beginning of an educational git tool I'm working on. In some cases, I may not know what I'm looking for, and rather discover something new by examining how git interacts with the filesystem.

[SpartanJ]

Hi @gbroques. Initially I added a new option called LinuxProduceSyntheticEvents. You might use it for your purpose. I personally wasn't able to reproduce your issue by watching the .git folder and doing some operations but since I'm not sure what I'm expecting to see I might just be missing it. Let me know if that helps.

[ed-sat]

Hi!
I am trying to apply your patch about syntetic events and it seems like you forget to set syntheticEvents for the new WatcherInotify

	WatcherInotify* pWatch = new WatcherInotify();
	pWatch->Listener = watcher;
	pWatch->ID = parent ? parent->ID : wd;
	pWatch->InotifyID = wd;
	pWatch->Directory = dir;
	pWatch->Recursive = recursive;
	pWatch->Parent = parent;
        pWatch->syntheticEvents = syntheticEvents; // <--- your forget to add here 

https://github.com/SpartanJ/efsw/blob/master/src/efsw/FileWatcherInotify.cpp#L135-L141

[SpartanJ]

Oh damn! Thanks for reporting it, since I wasn't able to reproduce the issue I never noticed. I'll patch it right away. Thanks again.

[ed-sat]

Hi! Yesterday I fixed some our tests and have made some fixes about syntetic events.
Please take a look.
#194

[SpartanJ]

Thank you very much for taking the time to investigate, fix and share the fix for this issue. With your merged fix I think we can consider this issue fixed, right? As far I understood you have internal tests that verify that the synthetic events are working properly, right?

[ed-sat]

Yes it looks like syntetic events work good. All our tests became green with synctetic events and fixes.
Could you please also rebase also branch modernize with last changes on master branch because we use it in our product.

[SpartanJ]

Yes it looks like syntetic events work good. All our tests became green with synctetic events and fixes.

Cool!

I think I'll just merge modernize into master! I waited long enough.

[SpartanJ]

I think I'll just merge modernize into master!

Done!