Option to periodically sync the database to disk

[Dariush]

From what I understand from searching this forum, Everything keeps the index in memory and syncs it to disk only when shutting down. However, I rarely shut down my computer and thus the on-disk version is frequently days out of date. I use this DB to keep track of files which were lost between two daily backups or as a pseudo-backup for files which I can easily recover from online sources, so I'm worried about losing data if a disk dies and crashes my computer, since Everything will not be able to sync the updated file list. Due to this I'd like to request an option to set a schedule (or at least a frequency) at which Everything.db on disk will be updated - e.g. every hour, or every day at 13:00 and 20:00.

[void]

I'll consider adding this to Everything, so it is done automatically.

For now, you can create a scheduled task and run Everything.exe with the -update command line option:
Everything.exe -update

The -update command line option will write the database to disk.

[Dariush]

Thanks for the tip, this (plus '-startup') did almost exactly what I wanted (the only minor downside is the Task Scheduler launching Everything if I have it closed).

[void]

Please try using the command line option -no-first-instance:
Everything.exe -update -no-first-instance

This will only save the database to disk if Everything is already running.

[Dariush]

…Thanks for the tip, this did exactly what I wanted.

[Stamimail]

Is there a possibilty to save the db file to other folder and to name it by a date? so that the user will be able to set a command in tasks schedular, to save a db file every day/week and so make an archive/log of db files. Something like:

2017-04-16 2200.db
2017-04-17 2200.db
2017-04-18 2200.db
2017-04-19 2200.db
2017-04-20 2200.db
...

[therube]

Perhaps, BACKUP9.EXE.

[Stamimail]

Perhaps, BACKUP9.EXE.

This solution is based on using Everything daily. But what about if the user doesn't use Everything daily, and he just wants to run a bat file that will do something like this:

• Run Everything in background for a moment
• Copy/Export the new database to <CurrentDateHour>.db/.csv/.txt/.efu
• Exit Everything

[Link]

I'm interested in this feature. I rarely shutdown my computer so if the computer unexpectedly lose power or similar then I'll end up with a very old database so I'm unable to see what files were for example on my ram drive or if Iost some recent files on my normal harddrives.

Having a script that restart everything isn't an option since I have constantly several search windows open and in use, a restart will make them disappear.

Can you please add this option?

[void]

I want to avoid writing to the disk. However, the next major version of Everything will provide options to save the index to disk at a specific interval.

The next major version of Everything will also save the database to disk when you close an Everything Search window after 24 hours has elapsed from starting Everything or last saving the database to disk.

Please try making your script run:
Everything.exe -update

This should save your Everything database to disk without closing Everything.

[dolos]

I'm interested in this feature. I rarely shutdown my computer so if the computer unexpectedly lose power or similar then I'll end up with a very old database [...]

I'm much in the same boat. Except that Win10 is full of bugs and sometimes their shitty internal COM RPC stops responding, hung on some shell function (thumbnailing, populating the "open with" menu) at which point it's just easier to force terminate. When the DB is really old, it will rescan everything anyway, but when the db is only "somewhat" old, i.e. still in the range of the USN change journal, it might take Everything what feels like ages in "Updating DB" state. It's actually faster a lot of times to force-kill the process, delete the DB file and let Everything re-scan everything.

Not sure what causes this, but reading USN changes should be somewhat fast, and I can see Everything saturates a CPU core (so that bit is single-threaded), so I would suspect that some operation (like updates, removed) on the actual in-memory data structures it uses is a bottleneck, that has a fastpath/isn't hit when constructing the db from scratch.

I want to avoid writing to the disk. However, the next major version of Everything will provide options to save the index to disk at a specific interval.

The next major version of Everything will also save the database to disk when you close an Everything Search window after 24 hours has elapsed from starting Everything or last saving the database to disk.

Please try making your script run:
Everything.exe -update

This should save your Everything database to disk without closing Everything.

I'd rather have a more complex trigger like "if there have been more than X changes, save database, but at most once every 20mins" (to avoid thrashing). There isn't much of a point saving an unchanged (or almost unchanged) db periodically.

It would also help if the db format was deltas/"chunked"/snapshot or whatever you'd like to call it. Saving by only appending the most recent changes to the end or a new file to reduce I/O load, until a certain threshold of "old" chunks is reached; after the threshold the db gets resaved as a single merged chunk to avoid consuming too much disk space.
Reconstructing the in-memory DB from these chunks should be easy enough, just log file removes in the later chunks as well, and updated file information for existing files can just replace in-memory any previous information from an earlier chunk.

Knowing what was changed can be a bit in the in-memory entries or a dedicated change list (such as a write ahead log).

Things like redis can do that (aof-use-rdb-preamble, auto-aof-rewrite-percentage), or LevelDB of course...

[void]

Thanks for your feedback.

Everything will use a single thread for *most* updating.
Renaming folders (and deleting folders too if there are orphaned sub-files) is expensive in Everything and will use all available cores when possible.

I'd rather have a more complex trigger like "if there have been more than X changes, save database, but at most once every 20mins" (to avoid thrashing). There isn't much of a point saving an unchanged (or almost unchanged) db periodically.

I'll look into adding a customizable option to do this.

It would also help if the db format was deltas

Streaming the index journal (db deltas) to disk might be useful for folder indexes.
Not so useful for NTFS volumes, as it would be identical to the USN Journal.
I'll consider implementing persisting all db deltas to disk in a future release.



It's worth noting while Everything is saving to disk you will be unable to search.
Saving usually takes 1 second per one million files. (not too long)
Choosing the time to save will be important.
I'm experimenting with saving to disk when you close the Everything Search window when a scheduled save was missed.

Thanks for the suggestions.