Everything 1.4 does not detect changes to docker VMDK

[aviasd]

Hi
I noticed some issue with everything on my machine.
For some reason files changed by docker (modified , size change etc) are not updated in everything's index. (referring to virtualbox vmdk files)
When rebuilding the index the changes appear.
On explorer the changes appear (size,modified date)

I'm indexing all the possible options , including recent changes.
on the ntfs tab for that drive everything is enabled , USN Max is 32768 KB allocation delta is 4096KB.
Closing and reopening everything does not help
Other than this issue, every change is instantly appearing on everything, so my guess is it has to do with the way docker/virtualbox is committing changes to disk...
procmon is detecting the changes to disk ( filter-> category is "write" )
My OS is win7 x64 sp1 latest updates are installed
Haven't had a chance to reproduce on second machine, when I do will post ...
Thx

Update: Just tested on a second machine (Another Win7), the issue is the same.
Just to note docker on windows 7 works differently than on newer os's - it's using virtualbox as the virtualization engine, on newer windows,it uses the native virtualization that comes with the os, so maybe there everything will behave differently, just fyi

[aviasd]

I did some further tests , it seems the issue is coming from virtualbox (latest build) and not specific to docker.
I've retested again with latest everything build with vanilla configuration, same issue.
I think the issue could point to a serious problem if everything misses NTFS changes to disk in some method of write by some apps, there could be other apps that could use that same write method that would fail detection by everything, making it unreliable when using for monitoring changes to disk,locating growing files, etc...

[NotNull]

I did some further tests , it seems the issue is coming from virtualbox (latest build) and not specific to docker.

It *is* coming from VirtualBox. Docker is 'just' a virtual application engine, where the application runs isolated and it's settings are saved somewhere else on the (virtual) Linux machine. It is just another program running on the virtual machine

The VMDK belongs to the virtual Linux mahine. It has nothing to do with Docker.

Everything has a couple of ways to get information about (NTFS) files and folders
- Reading the MFT. That is what is done the first time you index a disk AND when you rebuild the index
- Reading (monitoring( the USN journal. This is the NTFS logging of all file changes

File Explorer uses a different method to get information about files and folders.
Suggested reading: viewtopic.php?t=6231


First guess:
- I *think* Everything responds to a CloseFile (do you see those in procmon?) event. Only when the file is closed, and a corresponding entry is logged in he USN journal, Everything will update it's information of that file.
Your VMDK file stays opened as long as your virtual machine is running.

See if the index gets updated when you shutdown your VM.


Second guess:
Your VMDK is probably configured as a Sparse file ("grow on demand").
I don't know how Everything handles sparse files (easy to test, but not today..). Maybe that is an issue (don't expect so, but hey, it's just a guess ..)


Workaround:
Folder-index the folder where your VMDK is (/VMDK's are) in addition to the already configured NTFS index of it's disk.
Folder indexing uses the same mechanism as File Explorer, so it will (probably; untested) pick up on changes to the VMDK file.
(Side effect: those files will be shown twice in your result list)

[aviasd]

It *is* coming from VirtualBox. Docker is 'just' a virtual application engine, where the application runs isolated and it's settings are saved somewhere else on the (virtual) Linux machine. It is just another program running on the virtual machine

The VMDK belongs to the virtual Linux mahine. It has nothing to do with Docker.

The reason I specified docker is because it's running headless virtualbox with vmdk disk format which is not the native format virtualbox uses ( .vdi ) files . if I run a VM through virtualbox's interfaces, changes are immediately appearing on everything..

First guess:
- I *think* Everything responds to a CloseFile (do you see those in procmon?) event. Only when the file is closed, and a corresponding entry is logged in he USN journal, Everything will update it's information of that file.
Your VMDK file stays opened as long as your virtual machine is running.

See if the index gets updated when you shutdown your VM.

I don't see close file events on procmon when the machine is running.
Yes, when i stop the VM, everything gets updated.
I guess your guess (:)) is close to the truth here, I actually observed the same issue while downloading a simple file from the internet (using chrome) , the file information is not updated in everything, only until the file has finished downloading - it's sees the creation but during the appending of data into the file, it does not pick it up, just when the file has finished downloading and the handle is closed..
explorer by the way does see the changes but i guess the ntfs driver just gets the start and finish points...
-- so it's nothing related to virtualbox/docker - just how everything works..
It's a bit of a shame cause I rely on everything to monitor log file appends and file growth etc on large servers that write to multiple logs constantly as well as basic troubleshooting to see which program is misbehaving and writing too much to disk etc - and now I cannot rely on that - for an IT guy who does a lot of real time monitoring on things it's super useful to see live what gets updated and procmon outputs just too much and it's performance overhead is an overkill...
maybe as a feature request void could add a way to monitor open file handles and just monitor them for updates. hope it's not an overkill ..

Second guess:
Your VMDK is probably configured as a Sparse file ("grow on demand").
I don't know how Everything handles sparse files (easy to test, but not today..). Maybe that is an issue (don't expect so, but hey, it's just a guess ..)

Yes , it is a sparse file but so are my vmware machines and everything handles them without an issue ( as they grow)..

Workaround:
Folder-index the folder where your VMDK is (/VMDK's are) in addition to the already configured NTFS index of it's disk.
Folder indexing uses the same mechanism as File Explorer, so it will (probably; untested) pick up on changes to the VMDK file.
(Side effect: those files will be shown twice in your result list)

You're right, folder index does detect the change, It's seen clearly as the duplicate grows and the normaly monitored file stays the same.
btw - I tried to exclude that folder from monitoring using the exclude dir option but that just excluded both of them (the journaled index and the folder index) , funny - i think it should have separate exclusions per index type - just to handle these cases...
Thx ! and after all it's still one of my favorites !

[NotNull]

Thanks for the feedback! (I wih more people would do that ..)
I'm glad it helped (at least in understanding what's going on)

You're right, folder index does detect the change, It's seen clearly as the duplicate grows and the normaly monitored file stays the same.
btw - I tried to exclude that folder from monitoring using the exclude dir option but that just excluded both of them (the journaled index and the folder index) , funny - i think it should have separate exclusions per index type - just to handle these cases...
Thx !

The "Exclude" entry is separate from the NTFS and Folders entry under Indexes and thus should (and does) work on all of the indexes.

Everything being Everything (highly configurable), what you want is actually possible:

Under Menu:Tools > Options > Indexes > NTFS , configure Include only = !C:\MyFolder
Mind the "!" (meaning NOT)
You'll have to experiment with "" the path or not.

[aviasd]

Everything being Everything (highly configurable), what you want is actually possible:

Under Menu:Tools > Options > Indexes > NTFS , configure Include only = !C:\MyFolder
Mind the "!" (meaning NOT)
You'll have to experiment with "" the path or not.

Cool, how do I miss those things...
It works with the quotes.
i.e !"C:\Users\avi\.docker" removes the entry from the index

[Stamimail]

Everything being Everything (highly configurable), what you want is actually possible:

Under Menu:Tools > Options > Indexes > NTFS , configure Include only = !C:\MyFolder
Mind the "!" (meaning NOT)
You'll have to experiment with "" the path or not.

Cool, how do I miss those things...
It works with the quotes.
i.e !"C:\Users\avi\.docker" removes the entry from the index

IMHO, this ! (for exclude) should be added to the tooltip help.

[NotNull]

IMHO, this ! (for exclude) should be added to the tooltip help.

It isn't in the regular help either.

[aviasd]

Actually also macros works in that box
I created a filter macro ExcludeList with all the folder I want "explorer's monitoring" and I can add all the exclusions to that macro and use the same macro to all the drives i have just typing excludelist: in that box... also more organized that way

[NotNull]

Do you notice a performance hit/higher CPU usage with these settings?
I mean: this filter has to be evaluated real-time for every new USN entry. And there are gazillions of those, especially on a Win10 machine. Even without this (include/exclude) setting, Everything has a hard time keeping up. And you introduced yet another layer (the filter) to resolve.
Just curious...