Indexes of Folders disappear after reboot

[Scottes]

I have several network drives mapped, one of them to a 20+ TB NAS. Every time I reboot the indexes for all mapped drives "disappears" until a rescan is performed. Scanning the NAS takes almost an hour, so Everything shows nothing for these files until an hour after reboot.

Is there any way to persist the indexes for mapped drives, and just let it update at 3 AM like it's configured to do?

If it matters, Options... Indexes... NTFS... Automatically Remove Offline Volumes is unchecked.

Update: 1.5 hours later and Everything is "Not Responding" and the commandline tool is just hanging.

[NotNull]

And the indexed local NTFS volumes are not removed from the database?

IIRC, you have Store settings and data in %APPDATA%\Everything enabled (Menu:Tools > Options > General), meaning that Everything's database is stored in C:\users\Scottes\appdata\local\everything\everything.db

Do you have 'something' running at startup/shutdown that cleans userprofiles or user's %localappdata% ?
(roaming profiles enabled?)

Before you start Everything, can you verify that the Everything.db is there? And what is it's size?

[Scottes]

That setting is enabled.
Everything.db is 66,184,462 bytes, dated 1/23/20.

I don't have anything - that I know about - that cleans this dir, and that seems to be the case given the file's timestamp.

I was editing my message as your were entering yours apparently, just to add:
"Update: 1.5 hours later and Everything is "Not Responding" and the commandline tool is just hanging."

Since it hung I killed it and restarted. It started scanning the smaller mapped drives (Folders), but now it's "Not Responding" again, but the commandline tool is responding. It recovered. It seems to hang every time it starts scanning the NAS, but usually recovers.

[NotNull]

Sorry, was posting in 2 threas at the sam time. Things got mixed up.

Will clean it up ... Done!

[NotNull]

66 MB is way too small to contain all your 11+ million files and folders.
That is in the range 1-2 million files ..

Furthermore: that date is 5 days ago!
It sounds like Everything doesn't have enough time to write it's database to disk before your computer reboots.
Is there a Everything.db.tmp (can't remember the exact name) in the same folder?

My suggestion (for now; more to come):

• Exit Everything (right-click the Everything icon in the systemtray and choose Exit
• Delete Everything.db
• Start Everything
• Let Everything rescan all disks and folders (that should be done automatically)
• Leave it alone for 1 hour (until it completes indexing)
• Exit Everything
• Wait 5 minutes (to make sure Everything has enough time to write it's database to disk)
• REboot your computer
• Start Everything
  If things go as I "calculated", Everything will start right away and the files on your network disks are still in the index.

[Scottes]

Without the folders my system has 1.5 million objects, so the size is correct for 66MB.
And yeah, 5 days ago...
There's no every*.tmp file anywhere, and that folder only contains everything.db

And I'm a moron to forget something important - my system crashed. It did not close down cleanly.
So I would now have to assume that Everything only writes the database to disk when it exits. Is that correct? That would explain everything.

As a double-check, I once had this configured to run as Administrator. I checked the file perms and the user scott has full access, so it's apparently not a permissions issue. But the owner is Administrator... Well it was - I just changed it to scott, just in case.

Once Everything finishes scanning I will exit it cleanly, per your suggestion, and check the file size.


Still though, Everything is "Not Responding" and the commandline hangs...


I'm wondering if I should backup everything.ini, uninstall, delete all the directories, re-install, exit everything, restore the ini file, and start everything.
Thoughts on this?

[NotNull]

Still though, Everything is "Not Responding" and the commandline hangs...


I'm wondering if I should backup everything.ini, uninstall, delete all the directories, re-install, exit everything, restore the ini file, and start everything.
Thoughts on this?

Patience, my friend, patience ....

So I would now have to assume that Everything only writes the database to disk when it exits. Is that correct?

See the "more to come" post

[NotNull]

more to come

[1]
When you schedule to scan disks/folders at 03:00 and your machine (or Everything) is not running at that time or that folder is not available, Everything will miss that schedule.
When the conditions are right again, it will start re-scanning to make up for that missed schedule.
That can cause the scanning to happen at unexpected moments.
You can overrule that behaviour by enetring this in the Everything searchbar:

/folder_update_rescan_asap=0

and press ENTER.
That way it will only scan at the scheduled times.

[2]
Everything "lives" in RAM. When it is started, it loads the database from disk; when it is stopped, it will write the database to disk. In the meantime 'everything' happens in RAM.
If you want to save your database to disk in between, issue this command from prompt or scheduled task or whatever:

everything.exe -update

[3]
Make sure Compress database is unchecked (under Menu:Tools > Options > Indexes).
Compressing can take quite some time with such a big database.

[Scottes]

Patience, my friend, patience ....

If I had patience I would have stuck with my old, slower indexing program!

[NotNull]

HAhaha! You earned the "quote-of-the-day-award"!

[Scottes]

After 1.5 hours and a well-known lack of patience, I killed everything.exe.

I started it, and immediately deleted the NAS folder, leaving 3 much much smaller folders on another PC.
I let them rescan, ensured that Everything could find a file on the other PC, and clicked on File... Exit.
And then I right-clicked on the Evrything icon and chose Exit.

DB is 103MB, 2.6M objects per the status line. Yay!

So I added the NAS back, and it's scanning.
And the commandline is hanging...

And 32 minutes later I noticed that it's done. "-update" and now 370MB on disk, 9.6M objects.


So it seems like everything is working as designed, though I'm not a fan of it hanging while it scans that NAS. But this really shouldn't matter because that NAS will get scanned at 3 AM and I'll be asleep so I won't care if it hangs.

Question 1:
This PC is typically on 24x7x365 and only gets rebooted due to Windows Updates or crashes, which would mean that the database would rarely get written to disk. So it sounds like I should schedule a task to run "everything.exe -update" every day at 5 AM (2 hours after the Folders get scanned), just to ensure that the database gets written to disk. Or is there a built-in way to do this?

Question 2, depending on the answer to #1:
Can I request a feature to optionally write the DB to disk after scanning a Folder? It might make more sense to do so periodically for people who only have NTFS drives, but then again that's a much lesser issue than losing 11 million objects requiring a 1.5-hour scan...


Once again, thanks for the help, for bearing with me, and especially the Award!

[NotNull]

I don't think that Everything "hangs"; it is just very busy ...
There is also an option to rearrange the load over multiple cores/threads, as you have 12 cores. Maybe that can help (I am not familiar with these settings, but other people here are)

Question 1:

There is not yet a built-in way to write the database to disk. However ...
It is on the to do list for the next major version of Everything (Everything 1.5) (link)

(and that should answer your Question 2 too )
But if you have other good ideas .. just suggest them here.

Once again, thanks for the help, for bearing with me, and especially the Award!

Glad that I could help. Enjoy Everything!

[Scottes]

I don't think that Everything "hangs"; it is just very busy ...
There is also an option to rearrange the load over multiple cores/threads, as you have 12 cores. Maybe that can help (I am not familiar with these settings, but other people here are)

I've left a commandline search running for 12 minutes, and I often can't type in the GUI which displays "Not Responding" for extended periods of time. Looks like a duck... It does eventually recover though.

As for the core usage, I will check into that. I should have my new Ryzen 3900x running this weekend so I'll have 24 cores to play with. I'm sure that I can give 4 to Everything at 3 AM...

[NotNull]

I think @void can help you further than I can. He is the author of (and mastermind behind) Everything.
He can most likely tell you why it looks like duck ... (I can't)

BTW:

Ryzen 3900x

You take that lack of patience very seriously!

[void]

The "scanning" status should only occur once.
Folders are updated in the background.

Please make sure Everything is saving your database to disk successfully:

• In Everything, from the Tools menu, click Options.
• Click the General tab on the left.
• Check Store settings and data in %APPDATA%\Everything.
• Click OK.

Your Everything database should be stored in %LOCALAPPDATA%\Everything

Are you using a custom database location under Tools -> Options -> Indexes -> Database location? -Please try leaving this empty or moving it to a drive other than your C: drive.

If you exit Everything and restart Everything does the scanning always occur? -or does it only occur after restarting your PC?

There could be database corruption forcing Everything to rebuild each time Everything.exe is run.

Could you please try running Everything in debug mode, this will show the rebuild reason:

• In Everything, type in the following search and press ENTER:
  /debug=1

The next time Everything restarts a debug console will be shown.
Look for the following line:
Rebuild: <rebuild reason here>

What is the rebuild reason? -If unsure, please send your debug output to support@voidtools.com

To disable debug mode:

• In Everything, type in the following search and press ENTER:
  /debug=0
• Restart Everything.

[Scottes]

I think much of this has been solved by learning that Everything only writes the database to disk on a proper shutdown. Since my system was up for days and crashed the database never got written to disk thus it had to rescan when it rebooted.

To mitigate this I added a Windows task to write the database to disk, and it runs a couple hours after Everything is scheduled to scan the folders.

For now, I seem to be OK and don't expect any further problems.
If any happen, I'll come back and run the above.

[void]

Everything really tries to avoid writing to disk.

The next major version will periodically write the index to disk (~every 24 hours).

For now, as you have probably already found, please schedule Everything to flush its database to disk with the following command:
Everything.exe -update

This can be scheduled with a Task Scheduler.

https://www.voidtools.com/support/everything/command_line_options/#-update

[Greg Mc Fazdean]

Hi all,

after having some troubles with lost/forgotten search folders and trying some of the easyer before written hints simply closing everthing via "ctrl + q" instead of "alt F4" solved the issue of everything not writing the chosen folders to everyting.ini. Lucky me – hope this may help some of you!

[chad.beery]

I found that moving the database file off of the network drive and back into the Appdata/Everything location fixed my problem.

I have drives mapped to A, B, R and S and when I would restart the PC it would rescan and skip A, and B but it would scan R and S. I'm not sure why, but I think the A and B drives were not showing up soon enough so it would skip them, as soon as I would tell it to rescan the missing drives it would fix it as well.
I had the database file located on the B drive as well which I believe was forcing it to rescan each time since it couldn't see the B drive load initially.
As soon as I login, when I go to File Explorer and watch the drives populate, R and S are always there right away followed by A and B about 15-20 seconds later.
I then tested it by pointing the database location directly to where I had it on the server without the drive letter and it worked.