Indexing stalled

[ChrisK]

LOVE the content indexing/searching!

I've indexed 33K PDF & DOCX files, and it works amazingly.

However, I've run into about 6 files that stall/hang the indexer (ie. it shows that it's working on file X for hours/days at time - even after reboots).

I enabled debug logging (file below), but it does not seem to show anything useful. Here's a link to one of the files that gets stuck (it's a PDF e-book on APT and Malware) https://drive.google.com/file/d/1WaRI00 ... sp=sharing

Any guidance would be appreciated.

Thanks!


Debug Log.txt
2021-10-08 08:39:29.722: Everything
2021-10-08 08:39:29.722: Version 1.5.0.1278a (x64)
2021-10-08 08:39:29.722: Windows NT 10.0
2021-10-08 08:39:29.722: Processors 16
2021-10-08 08:39:29.722: IsAdmin 0
2021-10-08 08:39:29.722: AppData 1
2021-10-08 08:39:29.722: Service 0
2021-10-08 08:39:29.722: cmdline "C:\Program Files\Everything\Everything64.exe"
2021-10-08 08:39:31.843: MENUBAR ENTER OPEN
2021-10-08 08:39:31.843: MENUBAR POPUP 7
2021-10-08 08:39:32.551: MENUBAR LEAVE OPEN
2021-10-08 08:39:32.551: MENUBAR ENTER OPEN
2021-10-08 08:39:32.551: MENUBAR POPUP 8
2021-10-08 08:39:33.069: MENUBAR LEAVE OPEN
2021-10-08 08:39:33.069: MENUBAR ENTER OPEN
2021-10-08 08:39:33.070: MENUBAR POPUP 7
2021-10-08 08:39:35.640: #32768
2021-10-08 08:39:35.647: MENUBAR LEAVE OPEN
2021-10-08 08:39:35.647: COMMAND 40075
2021-10-08 08:39:35.784: Everything
2021-10-08 08:39:35.784: Version 1.5.0.1278a (x64)
2021-10-08 08:39:35.785: Windows NT 10.0
2021-10-08 08:39:35.785: Processors 16
2021-10-08 08:39:35.785: IsAdmin 0
2021-10-08 08:39:35.785: AppData 1
2021-10-08 08:39:35.786: Service 0
2021-10-08 08:39:35.786: cmdline "C:\Program Files\Everything\Everything64.exe"
2021-10-08 08:39:35.788: WM_ACTIVATE 0000000000000000 0000000000000000, lastfocus 0000000000120566, current focus 0000000000120566
2021-10-08 08:39:38.013: WM_ACTIVATE 0000000000000002 0000000000000000, lastfocus 0000000000120566, current focus 0000000000000000
2021-10-08 08:39:38.014: FOCUS 0
2021-10-08 08:39:38.014: FOCUS restore
2021-10-08 08:39:38.021: SYSCOMMAND 000000000000f012 0000000000ea050b
2021-10-08 08:39:43.211: COMMAND 41803
2021-10-08 08:39:43.212: Open pipe failed 2
2021-10-08 08:39:43.212: start all monitors (ntfs: 0 refs: 0)
2021-10-08 08:39:43.213: monitor folder setup
2021-10-08 08:39:43.214: RDC START 0000000000000000 1
2021-10-08 08:39:43.216: open volume \\.\E:
2021-10-08 08:39:43.216: opened 000000000000073c 0.000312
2021-10-08 08:39:43.217: RegisterDeviceNotification E: failed! 1066
2021-10-08 08:39:43.217: _db_monitor_process_update_event_available_event_proc
2021-10-08 08:39:43.217: check for folder updates 0000000000000000
2021-10-08 08:39:43.217: _db_indexed_property_request_available_event_proc 0
2021-10-08 08:39:43.217: indexed property request session created 1

[void]

Thank you for your feedback ChrisK,

Thank you for the problem PDF.
Thank you for the debug logs.

I am able to reproduce the issue my end.

The PDF iFilter is dead locked when trying to GetOutputDeviceColorContext.
The PDF iFilter does not always dead lock.

Enabling verbose output will show more debug information.
I'm seeing the PDF iFilter getting stuck requesting a new chunk after chunk 9.

I am working on a fix..
I have had some success with setting the thread COM concurrency modal as multithreaded.
However, this will increase the total number of threads Everything uses.


For now, please consider excluding the problem files.
Also consider using Windows indexing with si:.

[void]

Everything 1.5.0.1280a fixes an issue with reading content getting stuck.

Everything will now use multithreaded COM concurrency modal when gathering properties/reading content.

The default Windows 10 PDF iFilter is not APARTMENTTHREADED safe.

[ChrisK]

Wow! You are truly appreciated! Thank you very much!!!!!!!!!!!!!!!!!!

[void]

Everything 1.5.0.1282a changes all threads back to COINIT_APARTMENTTHREADED as it was causing issues with other COM calls.

Everything 1.5.0.1282a adds a content_pdf_ifilter_coinit_multithreaded ini setting to create a new thread that initializes the COM with COINIT_MULTITHREADED when calling PDF iFilters.

This setting is enabled by default.
This change should reduce the overall number of threads used (vs 1280a).

If you install a third party PDF iFilter you may wish to disable content_pdf_ifilter_coinit_multithreaded for slightly improved performance and less resource usage when searching PDF content:

• In Everything, type in the following search and press ENTER:
  /content_pdf_ifilter_coinit_multithreaded=0
• If successful, content_pdf_ifilter_coinit_multithreaded=0 is shown in the status bar for a few seconds.

Everything 1.5.0.1282a also adds a content_ifilter_coinit_multithreaded ini setting to create a new thread that initializes the COM with COINIT_MULTITHREADED when calling any iFilters.

(I've only run into issues with the PDF iFilter so far)