[FIXED] Content Indexing - x86/x64

[tuska]

Hello,

In 1.5.0.1245a (x86) "Index file content" is stuck at 11% regularly.
There is no problem with the x64 version.

Code: Select all

Typ     Name/Ext.		Size		Date			Attr.	"Indexing properties" *)
-----------------------------------------------------------------------------------------------------
x86     Everything-1.5a.db	  157 064 917	19.03.2021 14:39 	-a--	  11%
x64	Everything-1.5a.db	6 951 640 328	19.03.2021 12:25	-a--	 100% **)
	**) c:|d: files:ext:doc;docx;pdf;txt;xls;xlsx -> ca. 17,000 files

*) "Indexing properties" <-> "Index file content" ??? ...
   Everything Options -> Properties: Field "Properties" is empty!

I have done the following:

1. x64 [ ] Everything Service -> Everything Service unticked/uninstalled (no longer visible in Task Manager)
2. x64 Exit
3. x86 started
4. x86 [X] Everything Service installed -> (visible in Task Manager)
5. x86 no longer responding -> Task terminated with the Task Manager
6. x86 restarted - Indexing properties 11% is shown
7. [x] Index file content -> Indexing: X.pdf ... Index stuck at 11%!
   --> Include only folders: C:\Tools;D:\
8. x86 EXIT! -> Tooltip "Indexing properties.... 11%" remains displayed above Systray!
9. x86 "Everything (32 Bit)" is still displayed in the Task Manager although the programme has been terminated! (?)
10. Name/Ext.             Size             Date
    Everything-1.5a.db 157 031 954 19.03.2021 12:35
    File size no longer changes, the write process should therefore be finished.
11. x86 Task terminated with the Task Manager --> Tooltip in the Systray is no longer present
12. x86 Restart "Indexing properties... 11%" hangs
    -> This time, at [x] Index file content -> Indexing: Y.pdf -> another .PDF file is displayed

Settings:

Code: Select all

no_incur_seek_penalty_multithreaded=0

C:\Tools\Everything_1.5\Everything-1.5a.ini
C:\Tools\Everything_1.5\Everything_1.5x86\Everything-1.5a.ini
Reference: viewtopic.php?p=35537#p35537

Additional question:
Since the file "Everything-1.5a.db" for the x64 version is 6.47 GB in size,
Windows regularly informs me when shutting down that this file has not yet been written completely.

Am I allowed to continue with the shutdown of the PC in such a case or do I have to save
the file "Everything-1.5a.db" every time in time before?

Thanks!

___________________________________________________
Windows 10 Pro (x64) Version 20H2 (OS build 19042.868)
'Everything' 1.5.0.1245a (x86, x64)
Intel(R) Core(TM) i7-4770 CPU @ 3.4OGHz, 16 GB RAM

[horst.epp]

...
Since the file "Everything-1.5a.db" for the x64 version is 6.47 GB in size,
Windows regularly informs me when shutting down that this file has not yet been written completely.

Am I allowed to continue with the shutdown of the PC in such a case or do I have to save
the file "Everything-1.5a.db" every time in time before?

Thanks!

___________________________________________________
Windows 10 Pro (x64) Version 20H2 (OS build 19042.868)
'Everything' 1.5.0.1245a (x86, x64)
Intel(R) Core(TM) i7-4770 CPU @ 3.4OGHz, 16 GB RAM

Even having plenty of free memory I wouldn't let the index grow to this size.
I have limited the content indexing to importand file formats and dirs
and it runs perfect this way.
I will never want to save a db of that size on shutdown.
Unfortunately I see many complains coming up from tries to index to much.
For me Everything content indexing is not the replacement of Windows indexing.

[tuska]

2horst.epp
Thank you for your input!

I am still in the "finding out phase".

The settings were mainly made once for testing purposes to find out,
how and whether the content search in Everything works (also in comparison to the Windows search).

The comparison already yielded some interesting results, but I haven't checked them yet.

PS: The working memory, for example, shows a load of 65%.

[void]

Thank you for the bug report tuska,

3. x86 started
4. x86 [X] Everything Service installed -> (visible in Task Manager)
5. x86 no longer responding -> Task terminated with the Task Manager

This can occur if the Everything Service is installed, but not running.
After 1 minute Everything should recover.

Everything should no longer block if the Everything Service is not running in version 1.5.0.1246.
Everything should now show "Waiting for Everything Service..." in the statusbar.

x86 Restart "Indexing properties... 11%" hangs

It's odd Everything is not getting stuck 100% of the time..

Please try verbose debugging:

• In Everything, from the Tools menu, under the Debug submenu, click Verbose.
• In Everything, from the Tools menu, under the Debug submenu, click Console.
• When Everything appears stuck, what is shown in the console, there's two important lines you might see:
  • get chunk
  • get text
• If there's no console activity, it means the pdf iFilter is not responding.
• This issue could also be brought on by a low memory situation, the x86 will only allow you to allocate 2GB of memory, it looks like you might be reaching this limit. Everything should silently fail when reading property/content in low memory conditions. The iFilter may get stuck in low memory conditions. -How much memory is Everything using when it hangs?

[tuska]

Hi,
Thanks for your support.

The situation with me is as follows:

• x86 - 1.5.0.1246a started
• ⨀ Install the "Everything" service
• No response after clicking on "OK"; "Indexing properties" is displayed in the status bar; Bar at approx. 11%
• No response is shown in Task Manager
• Status unchanged - file size, date (please see above...)

  Code: Select all

  Typ     Name/Ext.		Size		Date			Attr.	"Indexing properties"
  -----------------------------------------------------------------------------------------------------
  x86     Everything-1.5a.db	  157 064 917	19.03.2021 14:39 	-a--	  11%

• Task terminated after approx. 8(!) minutes via the Task Manager
• Task Manager: Everything (1.5a) Service - Status: Running
• x86 Everything.exe restarted - "Loading recent changes..." -> "Indexing properties... 11%" is shown
• Tools -> Debug -> Verbose -> Display in the status bar unchanged
• Tools -> Debug -> Console -> "Everything 1.5.0.1246a (x86) (1.5a) Debug Console" window is shown and content running
  Display in the status bar unchanged
• Console: CTRL-A, CTRL-C, CTRL-V - result: contains 315 lines with "get chunk" and 315 lines with "get Text"
• Task Manager: The working memory shows a load of 24%, CPU: about 22%.
• Console window: terminated after approx. 5(!) minutes as I could no longer get to the menu items in Everything.
  This has also ended Everything x86.

[void]

Thanks for the info tuska,

Could you please send some debug output:

• Terminate Everything
• Launch Everything with the -debug-log and -verbose command line options:
  Everything.exe -debug-log -verbose
• When Everything hangs, could you please send me your %TEMP%\Everything Debug Log.txt

Since the file "Everything-1.5a.db" for the x64 version is 6.47 GB in size,
Windows regularly informs me when shutting down that this file has not yet been written completely.

Am I allowed to continue with the shutdown of the PC in such a case or do I have to save
the file "Everything-1.5a.db" every time in time before?

Is this the message you see?:

Saving Everything database to disk.

You can leave Windows on this screen and Everything should eventually complete the save to disk and Windows will shutdown.

[tuska]

Hi,

Here is a description of my activities:

• x86 - 1.5.0.1246a -> Task Manager: Everything (1.5a) Service - Status: Running
• C:\Tools\Everything_1.5\Everything_1.5x86\Everything.exe -debug-log -verbose <enter>
• C:\Tools\Everything_1.5\Everything_1.5x86\Everything.exe <enter> -> Everything (32 Bit) shown in Task Manager as a process;
  Everything (32 bit) does not open, i.e. no task available
• %TEMP%\Everything Debug Log.txt -> Everything Debug Log-1.5a.txt

  Code: Select all

  Name/Ext.                       Size    Date               Attr.
  Everything Debug Log-1.5a.txt	1 098	20.03.2021 14:12   -a--
  ----------------------------------------------------------------------------------
  2021-03-20 14:11:30.605: Everything
  2021-03-20 14:11:30.605: Version 1.5.0.1246a (x86)
  2021-03-20 14:11:30.605: Windows NT 10.0
  2021-03-20 14:11:30.605: Processors 8
  2021-03-20 14:11:30.605: IsAdmin 0
  2021-03-20 14:11:30.605: AppData 0
  2021-03-20 14:11:30.621: Service 2
  2021-03-20 14:11:30.621: cmdline "C:\Tools\Everything_1.5\Everything_1.5x86\Everything.exe" -debug-log -verbose
  2021-03-20 14:11:30.621: load localization
  2021-03-20 14:11:30.621: do install 0 0 0
  2021-03-20 14:11:30.621: create mutex
  2021-03-20 14:12:00.623: send message failed 1460
  2021-03-20 14:12:00.730: mutex exists
  2021-03-20 14:12:00.730: shutdown
  2021-03-20 14:12:00.730: shutdown ui help
  2021-03-20 14:12:00.730: shutdown ui global dialogs
  2021-03-20 14:12:00.730: shutdown ui options
  2021-03-20 14:12:00.730: shutdown ui
  2021-03-20 14:12:00.730: shutdown undo
  2021-03-20 14:12:00.730: shutdown plugin
  2021-03-20 14:12:00.730: shutdown ipc
  2021-03-20 14:12:00.730: shutdown tray
  2021-03-20 14:12:00.730: shutdown threads
  2021-03-20 14:12:00.730: shutdown mutex
  2021-03-20 14:12:00.730: kill

• Background process 'Everything (32 Bit)' shown in Task Manager terminated
• C:\Tools\Everything_1.5\Everything_1.5x86\Everything.exe -debug-log -verbose <enter> -> tried again
• Everything 32 Bit starts and shows "Waiting for Everything Service..." in status bar
  then after some time "Updating..."
• Everything Debug Log-1.5a.txt is growing and growing...
• There is no progress in indexing file contents - remains at 11%.
• After about 15 minutes, I closed Everything 32 Bit via F10 -> File "Everything-1.5a.db" remained unchanged!
• After another 5 minutes, I terminated the process "Everything 32 Bit" in the Task Manager,
  also because the file "Everything Debug Log-1.5a.txt" was continuously getting bigger and bigger.

%TEMP%\Everything Debug Log.txt
I have uploaded the following files to OneDrive and sent you a link by email.

Code: Select all

Name/Ext.					  Size in Bytes	   Date	    	        Attr.
2021-03-20_142923_Everything Debug Log-1.5a.txt	    162 752 681	   20.03.2021 14:29	-a--
2021-03-20_144710_Everything Debug Log-1.5a.txt	  1 535 115 264	   20.03.2021 14:47	-a--

Since the file "Everything-1.5a.db" for the x64 version is 6.47 GB in size,
Windows regularly informs me when shutting down that this file has not yet been written completely.

Am I allowed to continue with the shutdown of the PC in such a case or do I have to save
the file "Everything-1.5a.db" every time in time before?

Is this the message you see?:
Saving Everything database to disk.

You can leave Windows on this screen and Everything should eventually complete the save to disk and Windows will shutdown.

Thank you for the information!

Yes, I can confirm that the file "Everything-1.5a.db" is saved before shutting down, even if one does not react on this message:
"...
Saving Everything database to disk
Shut down anyway | Cancel"

About 10 - 15 seconds later, the PC shuts down after the message.

Thank you!

[void]

Thanks for the Logs tuska,

The logs show Everything trying to read the content for a PDF file.

The pdf-associated iFilter is constantly returning the same content chunk over and over again.
It's also returning malformed chunk data.

Everything will continuously read the content in the background forever.

This will explain why there is always an Everything.exe process left over after exiting Everything, as Everything is still stuck trying to read this content.

Everything version 1.5.0.1247 will now abort reading content from iFilters that return the same content chunk.

I've also fixed an issue with the Everything.exe not connecting immediately to the Everything Service when another Everything.exe is hosting the Everything Service.

About 10 - 15 seconds later, the PC shuts down after the message.

Thanks for confirming.
Everything will use a shutdown block reason: "Saving Everything database to disk".

Normally you will not see this message.
A 6GB database is quite large. You may want to further reduce the number files you are content indexing.

[tuska]

Everything version 1.5.0.1247 will now abort reading content from iFilters that return the same content chunk.

Hi,

Thank you for checking the problem!

Following situation:

• Task Manager: Everything (1.5a) Service -> RUNNING
• 1.5.0.1247a (x64): [✓] Everything Service (NOT unticked)
• x64 closed and then waited until the .db file was written
• 1.5.0.1247a (x86) started
• "Loading recent changes..." for a few seconds, then search queries can be started without delay
• "Indexing properties... 11%" is shown in the footer - Index stuck at 11%!
• 1.5.0.1247a (x86) started (2 Tasks), then one task terminated with F10 -> both tasks were closed
  Everything-1.5a.db has increased
• 1.5.0.1247a (x86) started - new attempt ...
  Everything (32 Bit) shown in Task Manager as a process;
  Everything (32 bit) does not open, i.e. no task available. ...

As you have already found out, it will be due to the pdf-associated iFilter,
because I cannot display a preview for a PDF file (but I can show the preview for *.doc;*.docx;*.txt;*.xls;*.xlsx)
in Version 1.5.0.1247a (x86).

In Version 1.5.0.1247a (x64) I can make a preview for a PDF file.

In version 1.5.0.1247a (x86) I have now taken out *.pdf and the file content indexing was done in 3 minutes!
Since then, the x86 version can also be started and closed several times without any problems!

Code: Select all

Name/Ext		Size in Bytes	Date			Attr
Everything-1.5a.db	404 274 488	21.03.2021 13:56	-a--

Since version 1.5.0.1247a (x86) is only a test version for me and I only use it for the purpose of testing
whether an unsuccessful search query, etc. in the x64 version works in the x86 version, I would leave it at that.

My knowledge is not sufficient to quickly install iFilter for PDF (x86) anyway
and I don't want to affect my existing working configuration (x64).

Index file content:
Sooner or later I will of course make a change under "Index file content"!
(But first I wanted to test the topic under my "normal conditions").

Many thanks for your support!

[tuska]

In version 1.5.0.1247a (x64) I made the following change: [✓] Index file content

Code: Select all

FROM:: C:\Tools;D:\
TO ..: C:\Tools;D:\Daten\Programme\TotalCommander\TCD_USB-Stick\TC32-64USB\

Indexing: Test2.pdf hangs --> (Include only files: *.doc;*.docx;*.pdf;*.txt;*.xls;*.xlsx)
Indexing properties... --> "Index file content" is stuck at 13% in x64

Now I'm not sure if it's the iFilter problem.

[✓] Index file content ... stucks when there is a problem with a PDF file [in Adobe Acrobat XI Pro Version 11.0.23 - 32-Bit], e.g.

• if you get the following error message when trying to open a pdf file:
  "Acrobat could not open "Test2.pdf" because either the file type is not supported or the file is corrupted.
  (e.g. an incorrectly decoded e-mail attachment) ..."
  1. Example 1: PDF file with a size of 0 bytes ("Test2.pdf")
  2. Example 2: PDF file with a size of 1 201 678 bytes;
     1. Preview in Everything x64: No
     2. Preview in Windows Explorer: No
     3. Preview in Total Commander: Sometimes but not always
• if it is a password protected PDF-file.
  Other comments:
  With 'content:' it is understandably not possible to search for a file content and preview it at the moment.
  Perhaps it would be possible in Everything, however, to enable a preview - after prompting for a password.

Does a "Force Rebuild" actually also update "Index file content"? (It looks like that to me, so yes.)
It would also explain my crashes during "Force Rebuild"?
I sent a "Mini crash dump" by email to Support on 21.3.

PS:
Index file content
Include only files: ... without *.pdf ... reduced the file size of the file "Everything-1.5a.db" from 8 GB to 230 MB!

[void]

Thanks for the feedback tuska,

Everything 1.5.0.1248a should fix the stuck iFilter issue.

Perhaps it would be possible in Everything, however, to enable a preview - after prompting for a password.

This would be up to the preview handler.
Implementing my own PDF property handler and preview handler is out of the question at this stage.
I've had a quick look at the PDF spec and PDF supports 8+ compression methods, not something I feel comfortable implementing..

Does a "Force Rebuild" actually also update "Index file content"? (It looks like that to me, so yes.)

Yes, (and no)

Yes if the volume is online.
No if the volume is offline (Everything will keep the existing index and properties).

It would also explain my crashes during "Force Rebuild"?
I sent a "Mini crash dump" by email to Support on 21.3.

Thanks for the mini crash dumps,
the crash dumps show the PDF iFilter crashing in GetChunk()
Please see if Everything 1.5.0.1248a helps.
It's odd this is only occurring now..

PS:
Index file content
Include only files: ... without *.pdf ... reduced the file size of the file "Everything-1.5a.db" from 8 GB to 230 MB!

I wonder if there's a single bad PDF in there consuming 7 GB on its own.
I'll look into a way to find the largest PDF content in Everything.

The following search might be a start:
len:content:>100000000
=show files containing content with over 100,000,000 characters.

[horst.epp]

Thanks for the feedback tuska,

Everything 1.5.0.1248a should fix the stuck iFilter issue.

Perhaps it would be possible in Everything, however, to enable a preview - after prompting for a password.

This would be up to the preview handler.
Implementing my own PDF property handler and preview handler is out of the question at this stage.
I've had a quick look at the PDF spec and PDF supports 8+ compression methods, not something I feel comfortable implementing..

Does a "Force Rebuild" actually also update "Index file content"? (It looks like that to me, so yes.)

Yes, (and no)

Yes if the volume is online.
No if the volume is offline (Everything will keep the existing index and properties).

It would also explain my crashes during "Force Rebuild"?
I sent a "Mini crash dump" by email to Support on 21.3.

Thanks for the mini crash dumps,
the crash dumps show the PDF iFilter crashing in GetChunk()
Please see if Everything 1.5.0.1248a helps.
It's odd this is only occurring now..

PS:
Index file content
Include only files: ... without *.pdf ... reduced the file size of the file "Everything-1.5a.db" from 8 GB to 230 MB!

I wonder if there's a single bad PDF in there consuming 7 GB on its own.
I'll look into a way to find the largest PDF content in Everything.

The following search might be a start:
len:content:>100000000

I suggest to try another IFilter for .pdf
SumatraPDF runs fine
https://www.sumatrapdfreader.org/prerelease.html
also the IFilter which comes with Tracker PDF-XChange is ok.
I also have good results with the TED PDF iFilter
https://www.pdflib.com/de/download/tet-pdf-ifilter/
There is a NirSoft tool which allows you to change the IFilter for any extension from the installed versions
https://www.nirsoft.net/utils/search_filter_view.html

[tuska]

2void
Thank you very much for the new version and explanations!

With version 1.5.0.1248a, all the problems described above have been solved(!):
Content indexing, Force Rebuild, file size of Everything-1.5a.db.

Here are my test results:

• 1.5.0.1248a (x86) started
• [✓] Index file content - changes
  Include only folders: Include only files:

  Code: Select all

  FROM:	C:\Tools;D:\		*.doc;*.docx;*.txt;*.xls;*.xlsx
  TO:	C:\Tools		*.doc;*.docx;*.pdf;*.txt;*.xls;*.xlsx  (*.pdf added)

  After "Apply" there was no hang!
   
• Indexes - Force Rebuild -> OK!
  "Indexing properties" could be carried out with this version without this function hanging up!
  This is in spite of the 3 PDF files mentioned above!
   
• 1.5.0.1248a (x86) -> File - Exit (ALT+F4)
   -----------------------------------------------------------------------------------------------------------------
• 1.5.0.1248a (x64) started
• [✓] Index file content - changes
  Include only folders: Include only files:

  Code: Select all

  FROM:	C:\Tools;D:\		*.doc;*.docx;*.txt;*.xls;*.xlsx
  TO:	C:\Tools;D:\		*.doc;*.docx;*.pdf;*.txt;*.xls;*.xlsx  (*.pdf added)

  After "Apply" there was no hang!
   
• Indexes - Force Rebuild -> OK!
• "Indexing properties" could be carried out with this version without this function hanging up!
  This is in spite of the 3 PDF files mentioned above!
   
• 1.5.0.1248a (x64) File - Exit (ALT+F4)
  It is interesting that the size of the file: Everything-1.5a.db has changed only slightly compared to yesterday,
  i.e. the file size has been increased from 230 MB to 292 MB. However, this file had a size of 8 GB before!
  I have only included *.pdf(!) in Version 1.5.0.1248a.
  
  After that I started version 1.5.0.1248a (x64) several times and then closed it again to make sure,
  that the file size would stay at 292 MB.
   
• Memory usage: Currently 28%

• Code: Select all

  <"C:\Tools"|"D:"> len:content:>100000000

  Result: 0 objects
  The largest PDF file I have is 113 MB.
   
• A file content search in approx. 6000 pdf files (ext:pdf content:searchword) then produced a search result in Real time!
  I can live well with that!

Thanks again!

Regards,
Karl
-----------------------------------------------------------------------------------------------------------------
Addendum:
After I have set the drive D:\ as well as *.pdf(!) in the x86 version under [✓] Index file content
an "Out of memory" error(-notice) occurred with a subsequent programme crash:

Code: Select all

.\src\mem.c(1062): mem_alloc(): Fatal error: out of memory 9633792

However, after I removed *.pdf again, the following setting is possible on my PC:

Code: Select all

[✓] Index file content
C:\Tools;D:\
*.doc;*.docx;*.txt;*.xls;*.xlsx

Limit tested!

[void]

Thanks for the feedback Karl,

I am glad to hear Everything 1.5.0.1248 improves pdf indexing for you.

The out of memory crash is normal for Everything x86 as there is an allocation limit of 2GB.
The x64 version will not have this limit.


Everything should be silently failing when it cannot allocate memory to read content... I'll look into why this is not the case..
The allocation could be elsewhere (like allocating room for results), in which case Everything will die with the fatal out of memory error.

[tuska]

... Everything should be silently failing when it cannot allocate memory to read content... I'll look into why this is not the case..

Mmhh, if possible, I would be pleased if in such cases there would still exist the note "out of memory".

Because if 'Everything' crashes without a notification, then I would certainly start several (unsuccessful) attempts to find the cause of the error.
With the hint message, I as a user would immediately have a hint as to how I could *perhaps* solve the problem myself...

[void]

Silently fail means the content for a really large file might be treated as empty (if the file cannot be loaded into memory).
Everything will continue to function correctly.

With x86 there is a problem if you try to load a file larger than 2GB, it will always throw the out of memory error, so the best option here is to silently fail.
Note: Everything loads the entire file into memory when reading file content.

Perhaps there could be a sliding scale. For example, anything over 8 MB just silently fail on out of memory and for anything under 8 MB throw a fatal error..

It might not be necessary, because eventually Everything will try to allocate 8 MB for a critical operation and if there's not enough memory it will throw the fatal error.
This is the error you are seeing above, Everything has no memory available to perform a critical operation.

[tuska]

Silently fail means the content for the really large file will be treated as empty.

With x86 there is a problem if you try to load a file larger than 2GB, it will always throw the out of memory error,
so the best option here is to silently fail.
Note: Everything loads the entire file into memory when reading file content.

Perhaps there could be a sliding scale. For example, anything over 8 MB just silently fail on out of memory
and for anything under 8 MB throw a fatal error..

It might not be necessary, because eventually Everything will try to allocate 8 MB.

Thanks for the information!