Preparing list issue

[sk2107]

Hi,

Since starting using Everything long ago, this is the first time I am facing this problem.

I have 4 external hard drives which I use for backup and I connect them all or some at the same time and I had no problem listing them because I am using them in one laptop only.

Few days back I faced a problem which started while indexing properties which took around a full day, but when it reached 90% the indexes were restarted from zero (my previous post which was answered thankfully), I disconnected all drives and tried to rebuild the list again for the same hard drives (which are still having the same number of files ) but it what happens that it reads the drive and then having the message (Sorting filenames...),I waited for more than a day but no change.

Before, for the same list it used to take 1 or 2 hours to show the list but not now, I tried to read first the local drives and it was OK, but after adding one external drive it behaves this way (reading drive the stick to the status "Sorting filenames..."), now the percentage is 83% and did not change for last 6 hours.

Is there a way to trace or debug the process in order to figure out what went wrong?
I installed the latest version (1.5.0.1356a x64) but the problem happened with the version before (when it was fine and started behaving this was with same version installed, so surely it is not an Everything issue but something around is causing the problem) . Nothing changed in my windows environment.


Best Regards,

[void]

now the percentage is 83% and did not change for last 6 hours.

What is shown when you hover over the progress bar with the mouse cursor?
-It should show the filename of the current file Everything is reading.

Is there a way to trace or debug the process in order to figure out what went wrong?

Please check the last rebuild reason reported under:
Tools -> Debug -> Statistics -> Build -> Last Rebuild Reason
What is shown?

Enabling debug logging may also provide more information.

[sk2107]

What is shown when you hover over the progress bar with the mouse cursor?

It shows only 83% and no file name.

Last rebuild reason: Add NTFS volume.

While writing this reply, progress bar percentage has increased to 93% but no file name is showing.

I think the process is running but very slow.

(I started feeling that it could be a problem with the USB ports as it happens with external drives only or having a problem with reading some files)
I will keep monitoring and see what happens.

Just one more question: will Starting debug logging affect the speed of indexing?


Best regards

[sk2107]

At last it is completed. But I changed the md5 property to limit the size of files to be calculated and it starts all over again

Is it possible to not force rebuild in case of changing a property setting as property indexing comes after rebuild (I think).

Best regards

note: I started debug logging the time it started rebuilding, shall I keep it of stop it, indexing reached 50%

[sk2107]

Sorry to have so many notes but I a m trying to get to the bottom of this.

I noticed that it takes a very long time on 83%, before and after the speed is acceptable, so is there any way to know why?

Best Regards

the debug is still running.

[sk2107]

Hello,
After many tests with a different machine with results the same symptoms, I found out that in case there are no properties defined the speed is acceptable but if there are properties defined it takes a very long time with no files in the list. If it is the case I hope the process of updating properties to be in the background (I know it is running in the background but it seems it is affecting drives scanning somehow).



Thank you.

[sk2107]

Dears,

Last test I did is installing an older version randomly (v. 1.5.0.1335a.x64) and applied the test I did before, the speed of the old version was much much much faster (around 45 minutes for one external drive) than the latest one (more than a day) to get the list.

Can somebody please reply to my comments with lots of thanks.

Best regards

[sk2107]

Hello,

I started installing all versions starting latest version and down.
I found that the problem started from v. 1.5.0.1347a.x64

Can you verify that please.

with all respect and gratitude.

Best Regards

[sk2107]

Dears,

the conclusion is that when no properties defined all versions are fast but if any properties defined specially MD5 the speed dies for the recent versions.

Of course I am talking about file indexing stage when the list becomes empty.

Please somebody advise as I got no reply except one since the beginning.

I am waiting for any comments.

Best regards

[therube]

Posting your, Help | Troubleshooting Information, couldn't hurt.
(You might want to nix the, ntfs_volumes=[{"path":..., line.)

Testing, with a new -instance might provide some insight?
So if current is slow, copy 1356a of Everything.exe into a new directory, perhaps called, /1356/ & start it with the -instance switch, & maybe calling it "-instance 1356".
Then add md5 Property.
No other changes.

Does that appear to again be taking days rather then hours?


You can test 1346 similarly.

Similarly, you need not index ALL files' md5.
For testing, you can limit it to a particular directory - to get a feel for time comparison between Everything versions.
A large enough directory so that timings are meaningful, but small enough that you don't need to speed all day waiting...

I think the process is running but very slow.

Task Manager or similar should show CPU usage & I/O, which might give you a clue.

it could be a problem with the USB ports as it happens with external drives only

"USB" is not necessarily the limiting factor. 3.0 > 2.0 > 1..., but even with that, some USB devices are simply dog slow.
(Sorry dogs about that derogatory comment.)

I changed the md5 property to limit the size of files to be calculated and it starts all over again

I'm thinking that's to be expected?
(But... since "At last it is completed", why. In any case...)

[sk2107]

First thank you for your reply.

I already tried all the possibilities you suggested (a new instance, new folder, other pc, x64 and x86, total uninstall and reinstall, new machine in which Everything was never installed, Virtual machines with windows 10 and windows 7, RAM between 2GB to 16GB, with antivirus and without), MD5 I limited the maximum size to 500mb.

Almost the conclusion is the same.

In small number of files it is acceptable but fraction of seconds can lead to a very big delay.

I excluded the MD5 from the properties index for now and the speed improved and acceptable but it is important to me because I am dealing with multi external hard drives for backup and using the MD5 for comparison.

If indexing and sorting the properties can be done after the scanning of drives is completed, it would be perfect (at least I can perform other searches until the whole process of properties is completed ).


Thank you for sharing and best regards

[void]

Sorry for the delay in my reply.

Last rebuild reason: Add NTFS volume.

While writing this reply, progress bar percentage has increased to 93% but no file name is showing.

Sounds like Everything is reindexing a new NTFS volume.
Please try removing the USB drive and see if it helps with performance.

Just one more question: will Starting debug logging affect the speed of indexing?

Generally, there shouldn't be any notice performance loss when running with debug logging enabled.
Verbose debug logging might slow things done a little.
Verbose output is off by default.

At last it is completed. But I changed the md5 property to limit the size of files to be calculated and it starts all over again

Changing property settings will force Everything to regather your properties from disk.

Last test I did is installing an older version randomly (v. 1.5.0.1335a.x64) and applied the test I did before, the speed of the old version was much much much faster (around 45 minutes for one external drive) than the latest one (more than a day) to get the list.

I started installing all versions starting latest version and down.
I found that the problem started from v. 1.5.0.1347a.x64

I suspect the issue is caused by multithreaded access to a drive.

In the latest version, could you please send your Tools -> Debug -> Statistics to support@voidtools.com
This text file contains information about your current volumes and multithreading settings.

[sk2107]

I will start the now and send you the result.

After all the tests I found the case is this:

I use the properties: height, width, length and MD5.

Last test was today using the latest version with height, width, length and MD5 properties (with the minimum MD5 parameters) and I started the process 8 hours ago and still nothing on the screen - the message is Sorting filenames..., so I just removed the MD5 property and restated Everything, it took only 20 minutes.

Thank you and Best regards

[sk2107]

Additionally I feel that the properties calculation starts before displaying the list. That's why it is clear (the list); when I add the MD5 because MD5 takes time to calculate specially for big files.


Best regards

[void]

Thank you for the debug logs.

The issue is caused by setting a maximum size for your md5 property index.

Everything is gathering the size for each file to check if it should be included in your property index.
My intention is to perform this size check after building the index.
Unfortunately, it's happening during the indexing phase.

I am working on a fix..

For now, please try removing the maximum size for your md5 property index.

[sk2107]

Thanks a lot and Best Regards

[void]

Everything 1.5.0.1357a fixes an issue with indexing properties when setting a maximum size or excluding files with the "not content indexed" or "recall on data access" attributes.

The initial index should now be fast.
Property indexing will now run in the background.

[sk2107]

Thanks a Lot, so appreciated.


Best regard