Allow custom remapping of file paths

[.oisyn]

Hi,

First, I'd like to point out that Everything is an amazing tool and is frequently used in our daily workflow. I work at a AAA game development company and as such we're dealing with tens of millions of files in a deeply layed out folder structure, and soon after I introduced it here it became part of the standard setup for every new PC in the office. So, hats off to you guys!

But I do have a feature request. We're often working with NTFS directory junctions to redirect some directories to other drives. The main use case is to speed up code and asset build times by putting intermediate files on an SSD. For example, we have a d:\game\intermediate that points to f:\intermediate. Obviously, when searching for a file that resides under that folder, e.g. "bla.txt", Everything will report to me that it found it at f:\intermediate\bla.txt. This is obviously correct, but I'd really like the path to be d:\game\intermediate\bla.txt where the same file can also be accessed, since that is the path that all our tools are expecting.

What I'd like to see is a simple folder remapping feature, so that I can remap every file that starts with f:\intermediate\ to point to d:\data\intermediate\. It doesn't have to be anything complicated like automatically detecting NTFS reparse points (which could potentially blow up as you could have infinite loops) - just a textual replacement in the path based on manual entries would be perfect. I guess such a feature could also be used with static file lists of which the files have since been moved to another location, which saves the user the trouble to manually edit the file list using a text editor


Regards,

.oisyn

[void]

Thanks for your feedback.

To make Everything follow the junction point D:\game\intermediate to F:\game\intermediate:

Remapping paths is a bit convoluted. However, it can be done with Everything 830b or later:

• Make sure Everything is not running (right click the Everything system tray icon and click Exit)
• Open your %APPDATA%\Everything\Everything.ini
• Change the following lines:
  
  auto_remove_offline_ntfs_volumes=1
  ntfs_volume_guids="\\\\?\\Volume{A DRIVE GUID}","\\\\?\\Volume{C DRIVE GUID}","\\\\?\\Volume{D DRIVE GUID}","\\\\?\\Volume{F DRIVE GUID}"
  ntfs_volume_paths="A:","C:","D:","F:"
  ntfs_volume_roots="","","",""
  ntfs_volume_includes=0,1,1,1
  ntfs_volume_load_recent_changes=0,0,0,0
  ntfs_volume_include_onlys="","","",""
  ntfs_volume_monitors=1,1,1,1
  
  to:
  
  auto_remove_offline_ntfs_volumes=0
  ntfs_volume_guids="\\\\?\\Volume{A DRIVE GUID}","\\\\?\\Volume{C DRIVE GUID}","\\\\?\\Volume{D DRIVE GUID}","\\\\?\\Volume{F DRIVE GUID}","\\\\?\\Volume{SAME AS F DRIVE GUID}"
  ntfs_volume_paths="A:","C:","D:","F:","D:\\game\\intermediate"
  ntfs_volume_roots="","","","","game\\intermediate"
  ntfs_volume_includes=0,1,1,1,1
  ntfs_volume_load_recent_changes=0,0,0,0,0
  ntfs_volume_include_onlys="","","","",""
  ntfs_volume_monitors=1,1,1,1,1

auto_remove_offline_ntfs_volumes must be disabled, otherwise the custom volume will be removed, because Everything thinks it is "offline".

You may wish to add an exclude filter to exclude f:\game\intermediate.

Eventually this will be added to the NTFS options UI.
This use to work in previous versions, however it has been broken in the last few 1.4.1 builds. Everything would think the drive has been "moved" and automatically removes it.

[yrwin]

Just one simple fix to the previous instructions. Use double backslashes:
ntfs_volume_paths="A:","C:","D:","F:","D:\\game\\intermediate"
ntfs_volume_roots="","","","","intermediate\\dummy"

[void]

Updated, thanks

[.oisyn]

Thank you so much for your reply. I finally got some time to play with these settings, and I can confirm that it works. Thanks!

Any idea when you will be putting a new public beta online? I'm all fine with rolling out the public beta to the team, but I'd rather not do that with a daily build with debug options

[.oisyn]

Might have spoken a bit too soon. I've added F:\intermediate to the exclude list, but as soon as I pressed "Ok", the application seems to be hanging. It isn't taking any CPU time according to the Task Manager. I've tried restarting Everything, restarting the service and even manually removing the database file, but nothing seems to work. When starting the app, it says "Updating database..." in the status bar and responds to nothing.

.edit: Ok I have cleared out the exclude folder list in the .ini file and now it works again. I can live with it finding the files on both locations so it doesn't really matter in the end, but I guess the exclude feature is broken somehow.

I'm on 847b (x64) btw.

[void]

Any idea when you will be putting a new public beta online?

Very soon.

Might have spoken a bit too soon. I've added F:\intermediate to the exclude list, but as soon as I pressed "Ok", the application seems to be hanging.

This is normal (unfortunately) if Everything was already busy rebuilding the index, then making another change will cause Everything to "lock up" until the index is rebuilt. I hope to add a queue system in a future release to avoid the issue.

[.oisyn]

Any idea when you will be putting a new public beta online?

Very soon.

Good to hear

This is normal (unfortunately) if Everything was already busy rebuilding the index, then making another change will cause Everything to "lock up" until the index is rebuilt. I hope to add a queue system in a future release to avoid the issue.

The point is though that it wasn't just waiting. I've left the window open for at least an hour, but nothing happened. If I force a rebuild of the database via the options menu, it doesn't even take 10 seconds to complete. It just seemed deadlocked. The same thing happens when restarting Everything.

[void]

If the hang is reproducible could you please send some debug output:

• In Everything, type in the following search and press ENTER:
  /debug
• Make a change to the exclude list.
• Click OK.
• What is displayed in the debug console when Everything appears to hangs?

There was a recent fix in 847 that prevents a hang when changing the options, but that hang would use CPU.

[.oisyn]

Done, I have sent the debug output.

[void]

Thanks for the debug output. It was very helpful.

There was a locking issue when accessing service monitors, fixed in Everything 849b.

[.oisyn]

Thank you for the swift fix. I can confirm that it works now

[collinchaffin]

Is this known to still be working in the most current beta 1.4.1.921? I ask because when I perform the re-mapping, instead of getting the remap of the mounted junction folder, the "root" defined of the junction folder is instead being remapped to the root drive letter.

Example:
C:\Data\Scripts remaps to a deeper folder on D: in the actual junction. So, I add the D: drive NTFS volume guid, the double-slashed junction path properly as shown, the "Scripts" as the root as shown, and what I get in ES is instead:

If I have a D:\Test1\Test2 folder structure (actual folders) for example.....now when I search ES for "path:C:\Data\Scripts" it shows results of C:\Data\Scripts\Test1\Test2

Is this a bug or should this remapping still be working? In essence, it appears the substitution (remapping) is going backwards and showing all the indexed D:\XXXXX real folders under the new remap root of C:\Data\Scripts\XXXXX.

[void]

Appears to be working for me.

What is the value of your ntfs_volume_roots in your Everything.ini?

[collinchaffin]

Okay so first let me attempt to more easily clarify what is happening and below I will give more detail of my config.

For your above example ini let's say you have the following when you open windows explorer:

F:\Rootfolder
F:\file01.txt
F:\intermediate\THIS_IS_A_TEST_FOLDER
F:\intermediate\THIS_IS_A_TEST_FILE.txt

D:\game\intermediate (intermediate is a folder junction with target of F:\intermediate)
D:\game\intermediate\THIS_IS_A_TEST_FILE.txt
D:\game\intermediate\THIS_IS_A_TEST_FOLDER

the result is upon ES search for "path:D:\game\intermediate\"

D:\game\intermediate\Rootfolder
D:\game\intermediate\file01.txt
D:\game\intermediate\intermediate\THIS_IS_A_TEST_FOLDER
D:\game\intermediate\intermediate\THIS_IS_A_TEST_FILE.txt

So instead of remapping the root D:\game\intermediate to F: instead everywhere ES lists something that used to begin with F: it now begins with (and even appears bolded it seems) D:\game\intermediate + \ + (entire drive F:)

Anyway, here is example without guids of my data:

ntfs_volume_guids="\\\\?\\Volume{bc0d01aa-aaaa-1111-aaaa-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}","\\\\?\\Volume{bc0d01aa-cccc-1111-cccc-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}"
ntfs_volume_paths="C:","D:","E:","C:\\Data","C:\\Data\\Scripts\\Sync\\Scripts","D:\\Data\\Scripts\\Sync\\Scripts"
ntfs_volume_roots="","","","Data","Scripts","Scripts"
ntfs_volume_includes=1,1,1,1,1,1
ntfs_volume_load_recent_changes=0,0,0,0,0,0
ntfs_volume_include_onlys="","","","","",""
ntfs_volume_monitors=1,1,1,1,1,1


C:\Data = Data is a NTFS Junction to D:\Data
C:\Data\Scripts\Sync\Scripts = Scripts is NTFS Junction to D:\Data\My Docs\Buried\Long....Path\Scripts
D:\Data\Scripts\Sync\Scripts = Scripts is NTFS Junction to D:\Data\My Docs\Buried\Long....Path\Scripts

In ES, if I search for path:C:\Data\Scripts\Sync\Scripts\XXXX the XXXX is everything from D:\XXXX (the root onward) and not what is really under the actual NTFS junction target.

All junctions/file ops work properly outside of ES. I have tried to reduce my above config to ONLY one and seem to get the reverse mapping no matter what. Thanks for the help!

[void]

Everything will map D:\game\intermediate to F: instead of F:\intermediate if F:\intermediate does not exist.

This is minor bug, as Everything should add no files and folders when the root is not found.

Could you please double check your root paths exists:
ntfs_volume_roots="","","","Data","Scripts","Scripts"

Does the scripts folder exist on the root of your D: ?
eg: D:\Scripts
or should it be D:\Data\Scripts ?

Please try the following ntfs_volume_root ini value:
ntfs_volume_roots="","","","Data","Data\\Scripts","Data\\Scripts"

[collinchaffin]

Oh wait a min! In your above code example, note that the example path given was D:\game\intermediate but then it appears the method is to take the LAST part of the path "intermediate" and place that on the root line as shown? LOL that must be my issue perhaps you used bad example paths?

(USER ERROR). For the sake of others In the above example, you are remapping D:\games\intermediate to F:\INTERMEDIATE.

I mis-read this and placed the last part of my SOURCE remaps (the "scripts" is from the "C:\data\scripts\sync\SCRIPTS", LOL!). Now with your last comment the light-bulb went off.

So, that example perhaps should be re-written to use a LONGER (not shorter) target remap path which is (usually) what you want to remap short to long to make a less-convenent path more convienient. Either way, make the example remap to a different name like intermediate2 just to make it clearer.

Anyway, I believe my mistake was that it should look more like this (note the * line has changed):

Code: Select all

ntfs_volume_guids="\\\\?\\Volume{bc0d01aa-aaaa-1111-aaaa-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}","\\\\?\\Volume{bc0d01aa-cccc-1111-cccc-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}","\\\\?\\Volume{bc0d01aa-bbbb-1111-bbbb-123456789123}"
ntfs_volume_paths="C:","D:","E:","C:\\Data","C:\\Data\\Scripts\\Sync\\Scripts","D:\\Data\\Scripts\\Sync\\Scripts"
*ntfs_volume_roots="","","","Data","Data\\My Docs\\Buried\\Long....Path\\Scripts","Data\\My Docs\\Buried\\Long....Path\\Scripts"
ntfs_volume_includes=1,1,1,1,1,1
ntfs_volume_load_recent_changes=0,0,0,0,0,0
ntfs_volume_include_onlys="","","","","",""
ntfs_volume_monitors=1,1,1,1,1,1

Correct?

EDIT: I can confirm this was in fact the issue but would still like to verify with you that this looks correct now. Also, you mentioned above about adding an exclude filter to exclude f:\intermediate or in my case for example C:\Data......can you elaborate? I have never had a need to set up a filter to EXclude, so want to verify in this remap case as to the best way to do it. Thanks!

[NotNull]

As an alternative, you might consider *NOT* using a driveletter for these volumes, but instead mount them in their respective folders.
You can do that in diskmgmt.msc. Rightclick the volume and choose Change driveletters and paths.
Here you can select "Mount in the following empty NTFS folder"

[void]

ntfs_volume_roots="","","","Data","Data\\My Docs\\Buried\\Long....Path\\Scripts","Data\\My Docs\\Buried\\Long....Path\\Scripts"

This looks correct.

Just a side note, manually editing roots was an after thought, Everything uses this internally for subst drives eg F: -> D:\game\intermediate

you mentioned above about adding an exclude filter to exclude f:\intermediate

Note: I can't tell what drive your Data\My Docs\Buried\Long....Path\Scripts is actually on.
To exclude this original path, so it can only be found in Everything from C:\Data\My Docs\Buried\Long....Path\Scripts or D:\Data\My Docs\Buried\Long....Path\Scripts:

• In Everything, from the Tools menu, click Options.
• Click the Exclude tab on the left.
• Click Add Folder....
• Select Data\My Docs\Buried\Long....Path\Scripts on the original drive, ie: the target of your junctions and click OK.
• Click OK.

[collinchaffin]

Ahh, perfect! Yea sorry all three of those Junctions are all drive D: due to legacy pathing that also involved cloud storage that was not compatible with seeing a Junction. Certainly not the best example but I figured as kludgy as it was if it worked with a nested junction inside the other, along with yet another duplicate exact path on a different NTFS drive, that then anyone could get this set up for almost any setup. (I know I can now!)

Thanks again for the help and for implementing this! Can't wait for this to hopefully ever make it into the GUI!

[dlong500]

@void Is this broken in v1.4.1.988? I can't seem to get anything to work. Every time I try to modify the settings the app just ignores them, adds all the volumes back to the settings, and nothing is indexed for the actual drive/path I'm trying to create (I'm using DrivePool). And yes, I've closed the app and stopped the service before editing the ini file.

[dlong500]

UPDATE: it looks like you also need to turn off automatically including new fixed volumes (auto_include_fixed_volumes=0). I'm still testing it out but I finally seem to be getting somewhere.

[NotNull]

I have no experience with DrivePool, but other people here have.
If you use the Search function of this forum (top-right), maybe their knowledge can help your case too.

EDIT:
I see you already discovered that