High DB File and Failure at the end of Indexing

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 8:53 am

Hi,
The db file increases in size exceptionally and at the end of indexing (with contents) it fails with out of memory file. My PC is 16 GB Ram and 500 GB SSD.
https://prnt.sc/Fc22NylBHbZD

Post by **void** » Thu Jan 19, 2023 9:02 am

Everything stores content in RAM.

I recommend not using content indexing, storing your content on a good NVMe drive and just searching the files from disk with content:
Everything will max out your NVMe SSD read speeds (3000+ MB/s)

Please try reducing the number of files you are content indexing:
I recommend indexing only user documents.

In Everything 1.5, from the Tools menu, click Options.
Click the Content tab on the left.
Change the Include only folders to:
%userprofile%\Documents
Change include only files to:
*.doc;*.docx;*.pdf;*.txt;*.xls;*.xlsx
Click OK.

To check how much memory Everything is using to store file content:

In Everything, from the Tools menu, under the Debug submenu, click Statistics.
The total content size is included in Database -> File data size.

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:10 am

Hi,
I am not sure if these solves my problem. I have a 500 GB PC that' half of it PDFs. I had already the PDF, DOC, XLSX filter on. I have also changed it to index only the user profile. But, already all my files are there.

With my current PC conditions: Is there any other settings that I can tweak aside from reducing the number of files?

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:22 am

Please see that the software doesn't obey the folders selected.

https://prnt.sc/uwkFyyYtKuF4

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:23 am

Also, it doesn't obey the "Not Index" order

https://prnt.sc/sGdiV4lb1tQ-

Post by **void** » Thu Jan 19, 2023 9:38 am

Everything will show the current file being property-indexed at the top right.
Hovering over this indexing text will show the full path and filename.

This tip will also show the current file being property indexed (not just content indexed).
Are you indexing any properties under Tools -> Options -> Properties?

Technically, content is just a "property" to Everything and all properties are listed here in this tip.

I recommend not using content indexing, storing your content on your SSD and searching these PDF files with the content: search function.

Indexing all these PDF files, then storing this index on disk is not going to offer any real performance increase.

Please consider indexing these PDF files with Windows Search and searching your Windows index with si:

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:41 am

I might be misunderstanding how Everything works: So, can I use content:"XXX" search methodology without content indexing? I thought that content indexing was already for this? What it adds?
I already use content:"xxx" or content:<aaa bbb>

Post by **void** » Thu Jan 19, 2023 9:45 am

Yes, you can use content: without content indexing.

With no content indexing:
Using content: will just go straight to disk to search your content.

With content indexing:
Everything will store content in memory for instant content searching.
You'll need a bucket load of RAM to enable content indexing for a lot of files.
Everything is only really designed to index a maximum of 1GB of text.
Only indexed content is searched when using content:

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:45 am

This is the property page: https://prnt.sc/Nh2sInf0oQkW

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:46 am

I understand, but why content indexing is not working for me?
With a 500 GB SSD and 16 GB Ram, it doesn't work, should it only work in the gamer PCs?

Post by **void** » Thu Jan 19, 2023 9:47 am

Everything will be trying to read the # property for every single file.

The current file being indexed for # will also be shown under the content page (top right).

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:48 am

I don't remember putting # there, but now removed. And trying to content index again,at least PC seems calmer during indexing. I will inform how it goes without properties.

Post by **void** » Thu Jan 19, 2023 9:55 am

I understand, but why content indexing is not working for me?
With a 500 GB SSD and 16 GB Ram, it doesn't work, should it only work in the gamer PCs?

From my limited understanding of your setup, it sounds like you are trying to index 250GB of PDFs with only 16 GB RAM.
You would need about 250GB of RAM to index that much PDF content with Everything.
Granted, your PDFs will not be text only and contain images, the memory usage will still be insanely high.

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:58 am

It's the same thing again.

I have tried to content index a folder with 72 GB of content (PDFs).

During indexing: https://prnt.sc/pSyER5zo9PoE

After some point, the out of memory failure. I couldn't get a screenshot in each time, because it fails itself and closes, (and somehow Teams too.)

Do you have any idea? Because I was happily using the content search function for now. I am ready to submit any data you need.

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 9:59 am

You would need about 250GB of RAM to index that much PDF content with Everything.

Really? Is this how it works? So, for a typical PC with 16-32 GB ram, content index will only work for 16-32 GB of files?
Also: It worked before!!

Post by **void** » Thu Jan 19, 2023 10:04 am

Yes, index content in Everything is stored in RAM.

Also: It worked before!!

Content indexing worked before? or content searching worked before?

Maybe you have a PDF file that is returning bogus content causing Everything to read a massive amount of text content that isn't there.
Does Everything appear to get stuck on a particular PDF?

16GB is a very large amount of text.. I would assume that these PDFs are not just all text and contain images.

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 10:05 am

Of course, they are technical books and reports. With many many images.
Content indexing worked before. Before, I wasn't specifying a specific folder, it was indexing everything that are present in the search folders. It worked smoothly.
I cannot understand why it fails now. Maybe I can go back to an older version. I can't understand what changed.

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 10:07 am

Another clue: I have 500 GB PC but it says 1.35 TB here, I don't know why. No external SSD is connected.
https://prnt.sc/mx-rUy9jeHxT

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 10:09 am

Another note: Without content indexing, content:"xxx" search results in downloading online files of Onedrive.

nesedenyana · Post by **nesedenyana** » Thu Jan 19, 2023 10:54 am

I might have underestimated the statement that Everything indexes the files in RAM. Because, what I can see is the ram used by Everything increases as the indexing continues, and when the file size reaches to something close to my ram capacity, everything starts to fail.

So, this is really interesting. Do we say that Everything will store all the files in RAM always? Don't we have any other option? This means that we can content index only a small number of files. It's not very relevant in that case, right?

Searching without content indexing: I didn't understand what you propose, because if I search without content indexing, Everything just starts to download all the online files not stored in my PC (Onedrive). So, I cannot use that, and I cannot imagine how effective it can be.

Post by **void** » Fri Jan 20, 2023 12:38 am

I suspect there's an issue with the PDF iFilter and Everything is treating some PDF files as text/plain.

The next alpha update will never treat PDF files as text/plain when content indexing.

Could you please check the largest indexed content size for some files:
Copy and paste the following into your Everything search box:
addcol:col1 col1:=len($content:) sort:col1-descending

This will list all files by indexed-content size from largest to smallest.
Please compare the size shown in the Column 1 column with the size of the file.
The column 1 size should be much smaller than the size of the file.

Please make sure Everything is installed correctly and running as a standard user:

In Everything, from the Tools menu, click Options.
Click the General tab on the left.
Check Store settings and data in %APPDATA%\Everything.
Uncheck Run as administrator.
Check Everything Service. (Please make sure this is tick-checked and not square-checked)
Click OK.
Exit Everything (right click the Everything tray icon and click Exit).
Restart Everything.

So, this is really interesting. Do we say that Everything will store all the files in RAM always? Don't we have any other option? This means that we can content index only a small number of files. It's not very relevant in that case, right?

Everything will store all the file content text in RAM.
An alternative to content indexing is to just search the content from disk when needed with content:
Indexing over 1 GB of text is not really recommended with Everything.
1 GB of text is a lot of text.

if I search without content indexing, Everything just starts to download all the online files not stored in my PC (Onedrive). So, I cannot use that, and I cannot imagine how effective it can be.

Please try excluding your OneDrive folder by including the following in your search:
!C:\Users\*\OneDrive\**

If you find this useful, please consider creating a filter from Search -> Add to filters.

Filters can be activated from the Search menu, Filter bar (View -> Filters), right clicking the status bar, filter macro or filter keyboard shortcut.

nesedenyana · Post by **nesedenyana** » Fri Jan 20, 2023 9:15 am

Please compare the size shown in the Column 1 column with the size of the file.

You can see that the sizes are exactly same. (The left is normal sizes, right is the code you sent)
https://i.imgur.com/msaOwkK.png

Please make sure Everything is installed correctly and running as a standard user:

I maded the checks. Everything as you said.

Indexing over 1 GB of text is not really recommended with Everything.

This is different. I have 100-200 GB PDFs, but of course I don't know how much text does that mean. But I am 100% sure it was working before, and now it doesn't. It might be a broken PDF as you said, but the figure above shows that Everything just uses the same file size.

Please try excluding your OneDrive folder by including the following in your search:
!C:\Users\*\OneDrive\**

That doesn't help me. I need to include Onedrive.

Post by **void** » Sat Jan 21, 2023 4:29 am

Thanks for the screenshot.

The screenshot appears to be missing the "Column 1" column.
The new "Column 1" column should be added to the right. You may need to scroll it into view.

Please compare the size column with the "Column 1" column, with the following search:

addcolumn:column1 column1:=len($content:) sort:column1-descending

Are the values huge in column 1? (equal to or larger than the file size in bytes?)
-If so, this would indicate an issue with the PDF iFilter.

nesedenyana · Post by **nesedenyana** » Sat Jan 21, 2023 10:31 am

See the screenshot. Does this give us hope about future update?
https://i.imgur.com/cQrntOd.png

Post by **void** » Sat Jan 21, 2023 11:25 pm

Thank you for the screenshot.

This confirms Everything is treating PDFs as text/plain.

I'm still investigating why this is suddenly occurring..
(The system is basically saying there's no PDF iFilter)

The next alpha update will never treat PDFs as text/plain.

I'll have a fix soon.

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 10:10 am

Thanks a lot! Waiting impatiently. In these short months content searching became crucial part of my daily work as a design engineer.

Post by **void** » Sun Jan 22, 2023 10:17 am

What PDF viewer are you using nesedenyana? Have you changed it recently?

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 12:00 pm

Oh, does that affect? Because, I changed a lot.
In the recent months, I have changed between Adobe Pro, PDF X-Change and Sumatra PDF. I changed it back from Sumatra to Adobe couple days ago.

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 1:01 pm

I noticed Windows search also doesn't search inside the PDFs now. It just returns the results from Words and Excels. Will report back.

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 1:04 pm

Here is the problematic setting, I believe. But, I am 100% sure that it wasn't the case before. It can be something I have done at last couple of weeks that changed this settings.
Now, I will wait for couple of hours for Windows to index and see if it affects the Everything.
https://prnt.sc/VR_5GD1F30Q-

horst.epp · Post by **horst.epp** » Sun Jan 22, 2023 1:52 pm

nesedenyana wrote: ↑Sun Jan 22, 2023 1:04 pm Here is the problematic setting, I believe. But, I am 100% sure that it wasn't the case before. It can be something I have done at last couple of weeks that changed this settings.
Now, I will wait for couple of hours for Windows to index and see if it affects the Everything.
https://prnt.sc/VR_5GD1F30Q-

That doesn't effect the Everything index
as long as you don't use si: to query the Windows index.
Everything uses the installed PDF iFilter to get the PDF content for its own index.
What PDF search filter do you have installed ?
You can check it with the NirSoft tool SearchFilterView
https://www.nirsoft.net/utils/search_filter_view.html
It look like this:

: Screenshot - 22.01.2023 , 14_55_05.png (55.73 KiB) Viewed 54892 times

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 3:15 pm

Thanks for that.
Very interesting, I have around 5 PDF viewer including the paid Adobe Pro, and no iFilter.
https://prnt.sc/XFAVs068nVfP

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 3:34 pm

Another interesting observation I think: When I checked the registry key in HKEY_CLASSES_ROOT\.pdf\PersistentHandler as recommended by Adobe, the key that I found is explained in the link as plain text.
{5E941D80-BF96-11CD-B579-08002B30BFEB}

https://strontic.github.io/xcyclopedia/ ... 0bfeb.html

horst.epp · Post by **horst.epp** » Sun Jan 22, 2023 4:25 pm

nesedenyana wrote: ↑Sun Jan 22, 2023 3:15 pm Thanks for that.
Very interesting, I have around 5 PDF viewer including the paid Adobe Pro, and no iFilter.
https://prnt.sc/XFAVs068nVfP

Using the Nirsoft tool you can set the required PDF iFilter
by removing the extension from the current one
and add the extension to another one.

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 4:32 pm

Thanks..
I have added pdf to the PDF-XChange's iFilter. I don't know why, but even though I reinstalled the Adobe Pro, I cannot see their iFilter here.
https://prnt.sc/hpUFtTn0weyH
And also I don't know why once installed and now gone ilovepdf is still shown in the pdfs as TypeName.

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 4:38 pm

This is good news, right?

horst.epp · Post by **horst.epp** » Sun Jan 22, 2023 4:56 pm

nesedenyana wrote: ↑Sun Jan 22, 2023 4:32 pm Thanks..
I have added pdf to the PDF-XChange's iFilter. I don't know why, but even though I reinstalled the Adobe Pro, I cannot see their iFilter here.
https://prnt.sc/hpUFtTn0weyH
And also I don't know why once installed and now gone ilovepdf is still shown in the pdfs as TypeName.

The TypeName has no relevance at all.
Its a relict from the last PDF software installation
and should be changed when you assing another software to open PDFs.

nesedenyana · Post by **nesedenyana** » Sun Jan 22, 2023 9:59 pm

Hi again.
All problems are solved after I made sure iFilter is added using NirSoft. It must be a mistake due to some PDF viewer installations that my iFilter was set to plain text. After sorting out the iFilter issue, everything worked great in Everything. I have over 13000 documents indexed and works flawlessly.

So, for the next release, should there be a automatic check for the iFilter?

Post by **void** » Thu Feb 02, 2023 5:13 am

Everything 1.5.0.1336a fixes an issue with Everything treating PDF files as text/plain when no iFilter is installed.

I will consider falling back to known iFilters. (even when they are not installed correctly)

For now, Everything will use the system installed iFilter.

nesedenyana · Post by **nesedenyana** » Mon Feb 06, 2023 2:07 pm

Indexing significantly slowed down after this update. I am waiting to test the new update, but it's been a half-day since I started while the same number of files were done in half an hour before the update.

nesedenyana · Post by **nesedenyana** » Mon Feb 06, 2023 2:11 pm

As you can see, Everything is indexing files that shouldn't index (content)

Post by **void** » Tue Feb 07, 2023 6:20 am

As you can see, Everything is indexing files that shouldn't index (content)

Where do you see that?

The column 1 value will be 0 for files that Everything doesn't content index.

Everything will show content and properties being indexed in the top right status text.
(Indexed properties are found under Tools -> Options -> Properties)
(Technically, content is just a property to Everything)
It looks like Everything should be indexing PDF files from your include filters.

Perhaps the change in PDF iFilter is making Everything take longer to index.
Does Everything appear stuck on a particular PDF file? or is the PDF indexing just slow?

nesedenyana · Post by **nesedenyana** » Tue Feb 07, 2023 6:22 am

Oh yeah, content is 0, understood.

No, the process moves on, but much much slowly. Since I close my PC at the end of day, I havent been able to complete one single indexing yet. Every day, we start from scratch.

Post by **void** » Tue Feb 07, 2023 6:27 am

What is reported under Tools -> Debug -> Statistics -> Build?

and

What is reported under Tools -> Debug -> Statistics -> Save?

nesedenyana · Post by **nesedenyana** » Tue Feb 07, 2023 1:08 pm

nesedenyana · Post by **nesedenyana** » Tue Feb 07, 2023 10:23 pm

Taken from another time where content indexing is at 61%.

Post by **void** » Wed Feb 08, 2023 3:29 am

Thank you for the statistics.

The stats show Everything is successfully saving the database to disk and loading the database from disk.
There could be an issue with a volume being flagged as out of date.

The next time Everything performs a reindex of your content, could you please:

Copy and paste the following into your Everything search box:
/restart-now-debug-log
Press ENTER in your Everything search box.
Everything will restart immediately without saving your database to disk and restart in debug logging mode.
Once Everything starts reindexing your content, could you please:
From the Tools menu, under the Debug submenu, click Stop Debug Logging.
---This will open your %TEMP%\Everything Debug Log.txt in Notepad---
Could you please send this file to: support@voidtools.com

Privacy

nesedenyana · Post by **nesedenyana** » Fri Feb 10, 2023 1:40 pm

Post by **void** » Mon Feb 13, 2023 12:40 am

Thank you for the logs.

The debug logs show the USN Journal on your C: drive is filled between closing Everything and restarting Everything.
This will cause Everything to see the C: drive as a new drive and thus rescan all your properties/content.

What are the USN Journal settings for your C: ?
-To view these settings:
In Everything, from the Tools menu, click Options.
Click the NTFS tab on the left.
Select your C: drive.
What is shown in this screen?
-Please try increasing your USN Journal maximum size to: 131072 KB

Please make sure you don't have any stuck Windows updates.
-I've seen stuck updates make millions of file system changes on reboot.

-Please check your Index Journal in Everything.
This will show all changes to your file systems.
Is there something spamming your file system?

nesedenyana · Post by **nesedenyana** » Mon Feb 13, 2023 8:16 am

I have Sharepoint connected to my system which is also indexed by Everything. It gets updated quite a lot by my colleagues etc.
You can see the screenshot below, I increased the size. I don't know where I canssee the journal.