Refusing to index all NTFS drives automatically.

[raccoon]

Below is an image of Everything 1.4 and 1.5 being run from a test folder, with no settings, no db, no rogue processes or services running. Fresh.
I typed the search phrase "parent:" to display which drives have been indexed. For context, the drive for volume N: is the same brand, model and capacity disk as volumes D:, E:, and O:, each plugged in by USB cable in the same USB host bus.



It would appear that both 1.4 and 1.5 begin scanning all the volumes immediately on start. After indexing completes, both of them ask for either Administrative or Service permission. I gave Service permission to both, and then they both rescanned again with the same results. The first scan was long and time consuming, the second scan was quick.

So, I guess 2 bugs to report here.

1) For some reason 1.5 fails to index all of my drives, and I'm not sure why.

2) Everything (1.4 and 1.5) should be asking for Administrator or Service permissions before conducting a backwards-compatibility scan of each drive. There should also be a 4th option added to this list:

• Run as administrator
• Start the "Everything" service (recommended)
• Use slow indexing method on NTFS volumes
• Do not index NTFS volumes

[raccoon]

Update: I just remembered something. There *is* one difference about all these NTFS volumes that 1.5 is refusing to index (by defalt). This is a very hacky hackish thing, and I'd be very surprised to learn why and how Everything is detecting this.

The disks for NTFS volumes for A, D, E, J, K, and O were especially partitioned with the reserve partition destroyed prior to creating the data partition and formatting. Here is a DISKPART log to explain what I mean.

Why is Everything 1.5 able to know this, and react differently?

Code: Select all

DISKPART> sel vol n
Volume 4 is the selected volume.
DISKPART> list part
  Partition ###  Type              Size     Offset
  -------------  ----------------  -------  -------
  Partition 1    Reserved           128 MB    17 KB
* Partition 2    Primary           4657 GB   129 MB

DISKPART> sel vol o
Volume 3 is the selected volume.
DISKPART> list part
  Partition ###  Type              Size     Offset
  -------------  ----------------  -------  -------
* Partition 1    Primary           4657 GB  1024 KB

For clarity, Everything 1.5 will index these volumes fine, I just have to turn on "[x] Include in database" myself. Everything has them not-included by default.

[void]

Thank you for your feedback raccoon,

Everything 1.4 will see removable drives as fixed drives.

Everything 1.5 will see removable drives as removable.



Everything 1.5 is most likely seeing your missing drives as removable.
Everything 1.5 will see a drive as removable if it:

• has removable media.
• has media hotplug support.
• has device hotplug support.

Everything does not include removable drives by default.
You will need to manually include these from Tools -> Options -> NTFS.

I will look into making Everything index all NTFS drives on first run.
This would make the behavior similar to Everything 1.4.
New drives after the initial index will be ignored.


Could you please send some debug output when viewing Tools -> Options -> NTFS:

• In Everything, please make sure the option window is closed.
• From the Tools menu, under the Debug menu, click Start Debug Logging.
• From the Tools menu, click Options.
• Click the NTFS tab.
• Close the Options window.
• In Everything, from the Tools menu, under the Debug menu, click Stop Debug Logging.
• This will open your Everything Debug Log.txt, what is shown in your debug log?

The debug output will show how Everything detects which drives to include.

It would appear that both 1.4 and 1.5 begin scanning all the volumes immediately on start. After indexing completes, both of them ask for either Administrative or Service permission. I gave Service permission to both, and then they both rescanned again with the same results. The first scan was long and time consuming, the second scan was quick.

Only 'system' drives will require administrative privileges to index.
Once administrative privileges are gained, Everything will quickly reindex the missing system drives (most likely just the C: drive)

Use slow indexing method on NTFS volumes

There is no slow indexing method.
There is the really slow folder indexing method, which most users would not want.

[raccoon]

Hi void,

I performed the debug log as you asked, but I can't seem to identify where Everything is identifying one drive or the next as fixed or removable.

To be sure, all of these volumes (except for B:) are Fixed or "Local Disk" according to Windows, as their hardware RMB (Removable Media Bit) reports 0. They are high capacity spinning platter hard disk drives via USB. Not thumb drives. The Drive Icon in Everything shows them to be Fixed.

I am not sure if you saw my followup comment, describing how I deliberately hacked and removed the reserved partition on all of these unindexed disks, which seems to correlate this behavior. Remember, drive N is being indexed by Everything, but drives D, E and O are not, even though they are the same make and model and year disk drive.


The screen shot I took says "Everything requires administrative privileges to index NTFS volumes." But it does not say, "needs administrative privileges to index the system disk. Indeed, the drives are indexed before this prompt appears. Selecting either Run as administrator, or Start the service, will cause Everything to index them again, a second time. ((Upon additional testing, this also appears to cause 1.5 to re-index all the new File Properties, with becomes especially time consuming as well.))

Does Everything need administrator rights to index NTFS volumes, or just the system volume (drive C:)?
Can we initiate this prompt before performing any indexing? It can take 5 or 10 minutes after launching Everything before the prompt appears. The user may have already left the room to wait for indexing to complete, only to have it index again after accepting the prompt to Start Service.

[void]

platter hard disk drives via USB. Not thumb drives. The Drive Icon in Everything shows them to be Fixed.

Look for the following lines in the debug log:

found volume <drive-type> <guid> <path> <root>

where <drive-type> will be one of the following:

• 2 = Removable
• 3 = Fixed
• 4 = Remote
• 5 = CDROM
• 6 = Ramdisk

What drives are reported as removable (2) / fixed (3)?

Removable in "Everything" means any volume or device that can be 'safely' removed.
Windows may still report these volumes or devices as fixed.

Removing the reserved partition shouldn't make any difference with Everything detecting the volume as removable.

The screen shot I took says "Everything requires administrative privileges to index NTFS volumes." But it does not say, "needs administrative privileges to index the system disk.

I've changed this to say:
"Everything" requires administrative privileges to index system NTFS volumes.

Selecting either Run as administrator, or Start the service, will cause Everything to index them again, a second time

The existing indexes are kept.
Only the new system volumes will be indexed.

((Upon additional testing, this also appears to cause 1.5 to re-index all the new File Properties, with becomes especially time consuming as well.))

The existing indexed properties are kept.
The list of existing properties to index is kept.
Only properties on the new system volumes will be added to list to be indexed.

Does Everything need administrator rights to index NTFS volumes, or just the system volume (drive C:)?

Only system volumes.
Admin privileges are not required to index non-system volumes.

I can't tell you what makes a volume a 'system' volume.
Usually a system volume is the volume where the windows folder is located and any volume that has a page file or hibernation file.
Most external volumes will be non-system.

Can we initiate this prompt before performing any indexing?

I will consider an option to do this.
Currently, Everything doesn't know a volume is a system volume until it attempts to open and read the volume.
There are also complications for offline volumes, which Everything will treat as system volumes.

It can take 5 or 10 minutes after launching Everything before the prompt appears. The user may have already left the room to wait for indexing to complete, only to have it index again after accepting the prompt to Start Service.

Adding the system volumes to an existing index should be quick.
How many files is Everything indexing?

[raccoon]

Ok, looking at the debug log and it is as you say.

And I discovered why. It's the Quick Removal / Better Performance policy.

I am curious which Windows API you are using to make this detection? And why it's basing on the Removal Policy.

You should be aware: > "Beginning in Windows 10 version 1809, the default policy is Quick removal. In earlier versions of Windows, the default policy was Better performance." -- https://docs.microsoft.com/en-us/window ... rage-media (25 Nov 2020)

I'm sitting behind Windows 7 right now, so the policy defaults to "Better performance" for RMB=0 drives, and "Quick removal" for RMB=1 drives. I had manually set disks D, E, J, K and O to "Quick Removal" while disk N was still set to "Better performance". But now Windows 10 will make all USB drives use "Quick removal" (by default) regardless of the hardware Removable Media Bit value.

I changed my drive N to "Quick Removal" and now Everything's behavior has changed for this drive, after a reboot. I did not remove the reserved partition.

The GetDriveTypeW() function returns 3 DRIVE_FIXED (for me) for all of these volumes, regardless of their removal policy. I wonder what Everything is doing differently, or perhaps sees differently as Administrator, Service, or User?

[void]

Everything uses IOCTL_STORAGE_GET_HOTPLUG_INFO to determine if a drive is 'removable'. (can be safely removed)

Everything ignores the DRIVE_FIXED value from GetDriveTypeW as this will be set for almost all drives.

Everything 1.5.0.1278a will now index removable NTFS and ReFS drives by default.

I will trial this and see how it goes...
There has been issues in the past where users do not want Everything to access dying drives.
However, these cases should be rare.

There might also be issues for users that have many drives that are plugged in/unplugged during the day.
They may end up with hundreds of D: drives in the index as Everything 1.5 no longer auto removes offline volumes.
I would suggest to these users to disable the auto include removable volumes from Tools -> Options -> NTFS.


Note:
You can right click the indexing progress bar and click Cancel Index (Ctrl + Break) to abort the current index.
When aborted, you can manually setup which volumes Everything indexes under Tools -> Options -> NTFS.

[therube]

Everything 1.5.0.1278a will now index removable drives by default.

I take it, that is NTFS (& ReFSs) removable drives?

I'm thinking that should not be a default behavior - or at least not for me.
I have my "drives" - which are indexed.
And because I happen to plug in some external, removable drive, it gets its data added to "my drives"?

In cases like that, where I specifically may want to index a removable drive - but don't want it to pollute "my drives", I'll set up a separate instance to do that.

I'm working on someone else's computer & I pull their drive & stick it into an external box & plug it into my computer, I may want to index their drive, but I don't want its' data commingled with "my drives".

Automatic, I'm polluted.
Another instance, I'm free to add it as I want.

(And of course, the dying drive syndrome .)

[raccoon]

therube: You're still using Windows 7 right? Your

Everything does not discern whether the drive is USB or not, directly, but whether the drive policy is "Quick Removal" or "Better Performance."

All Windows operating systems before November 2020 will index RMB=0 large capacity external hard drives as "Better Performance" and therefor they would be automatically indexed by Everything, anyway, regardless of these proposed changes. Only Windows versions from the past 11 months will notice a difference.

I had to specifically and manually change my ext HDD disk policy from "Better Performance" to "Quick Removal", one-by-one on each drive, so they behave the same way that Windows 10/11 does.

https://docs.microsoft.com/en-us/window ... rage-media (25 Nov 2020)

It's my opinion that everyone is using external harddrives today for data storage. 1 TB, 5 TB, 12 TB and 18 TB drives are very popular. These should be automatically indexed, because that's where all the files are. Most people leave these plugged into their computer 24/7. They often require an external power cable. I am not talking about tiny capacity solid state thumb drives (RMB=1).

Microsoft calls Removable Media Bit RMB=0 drives "Fixed" and RMB=1 drives "Removable".
Microsoft does not believe that Fixed Drives should require a screw driver to install into the computer, but just a simple USB cable.

If you only want Everything to index a new hard disk drive that you installed with a screwdriver, then uncheck both of these boxes.



Then the next time you reach for your screw driver, you can manually add that drive to the list.

[void]

I will keep experimenting with auto including removable volumes.

I predict Everything will run into issues with too many removable volumes being left in the index.


For now, please use the -no-auto-include command line option when starting a fresh instance.
-or-
Press Ctrl + Break or right click the Index progress bar and click Cancel to abort the first index.

[raccoon]

I'm still a little confused by your decision to:

Everything uses IOCTL_STORAGE_GET_HOTPLUG_INFO to determine if a drive is 'removable'. (can be safely removed)

Everything ignores the DRIVE_FIXED value from GetDriveTypeW as this will be set for almost all drives.

I think this is the wrong approach and you ought to consider reverting to GetDriveTypeW(). The words "Fixed" and "Removable" are already defined by Microsoft and have user familiarity. Fixed == Removable Media Bit 0, and Removable == Removable Media Bit 1, not how a drive is plugged into the machine (SATA cable vs. USB host) nor the Better Performance / Quick Removal policy.

People tend to keep high capacity USB drives plugged in forever and assign permanent drive letters on them, the same way they do with SATA drives. And people definitely want Everything to index these files. That's where they store their 18 terabytes of media.

[horst.epp]

People tend to keep high capacity USB drives plugged in forever and assign permanent drive letters on them, the same way they do with SATA drives. And people definitely want Everything to index these files. That's where they store their 18 terabytes of media.

Please don't assume that other poeple think as you.
I never want to automatically index all available NTFS drives.
If a user wants this he can set the indexing option and all is fine.

[raccoon]

Please don't assume that other poeple think as you.
I never want to automatically index all available NTFS drives.
If a user wants this he can set the indexing option and all is fine.

Then you should disabled "automatic indexing" in Everything's Options.

A program named Everything is supposed to automatically index Everything, except for the rare exceptions like your situation.

[void]

Thank you for the feedback.

I will make some adjustments with how Everything determines a drive as fixed/removable.

Ideally I would like Everything to index all NTFS hard disk drives by default.

[void]

Everything 1.5.0.1280a will no longer treat fixed volumes as removable when they can be safely removed.

In other words, Everything will treat volumes as fixed/removable from the GetDriveType API call.

Removable volumes are no longer included by default.
This is the same behavior as Everything 1.4