Symbolic links

[yrwin]

I have few folders, which are symbolic links to different folder on the same drive. Is it possible to somehow find files in this symbolic-linked folder please?
Thanks for advice.

[therube]

You could add the symbolic link (Junction) as a Folder index.

Tools | Options | Indexes -> Folders: Add...

[yrwin]

If you mean Tools/Options/Indexes/Folders, Add and enter the folder, which is symbolic link, than unfortunately this cant be done. The symbolic link folder is unpacked to its non-link form.

[collinchaffin]

This is just not the case all the way up to last night's nightly beta build. In fact, if I am ANYWHERE within a symlink's path depth, everything fails to show ANY files - the list is always 100% BLANK.

Example:

Real path: C:\Files\OLDStructure\Cause\It\Is\Cool\Scripts\Sync

Symlink: D:\NewDrive\Scripts\Sync (Sync is a hardlink JUNCTION to real path's Sync folder above. Whether in the Sync folder or all the way down to these depths:

D:\NewDrive\Scripts - (Shows local files in this real path since the junction is the Sync folder within this folder - shows NOTHING in Sync and further).
D:\NewDrive\Scripts\Sync\Perl\Modules\Scripts\Cool\Stuff\More\Files - (Open everything here, show ANY/ALL files - ZERO NO FILES AT ALL RECURSIVELY!! Could be a MILLION files in a structure in subfolders but will not be seen by Everything!).

So, you see, if you make the mistake of a shallow symlink/junction and forget you're root it that of a junction, forget using Everything at ALL right now because it simply finds 100% of ALL files and folders to be invisible simply not there at all. Even without any filters just opening to any subfolder in the path always results in an empty file list.

Because of the depth, I literally wasted HOURS trying to figure out why I must have damaged or bad indexes, etc. and couldn't make everything work at all.

....But....once in the above example I go to this path:

C:\Files\OLDStructure\Cause\It\Is\Cool\Scripts\Sync\Perl\ (Everything functions PERFECTLY and finds 100% all files from here down all subfolders including files in the "More" folder above as example)

So, until this can <hopefully> be fixed (since this behavior is frankly idiotic) to if nothing else show an error but not just act like it's functioning fine on those paths and erronously just show no file/folder data ata ll, rule of thumb is.....reverse resolve all your symlinks and junctions and be sure you go back to rerun your searches ONLY on the physical non-symlinked file structures, or for the time being open another search tool that can at the very least just friggin FOLLOW a symlink/junction, like just about everything on earth these days can.

[yrwin]

Processing symlinks can be problematic. I understand it. But simply unpack given path to base (without symlinks) form should be easy. It will give you worse results e.g.:
after looking for files in c:\Symlink\ (= c:\Some\Strange\Path\PhysicalLocation\) will give you path starting with c:\Some\Strange\Path\PhysicalLocation\
but as i said. This can be easy to implement.
Maybe after finding results, unpacked path can be repacked back (c:\Some\Strange\Path\PhysicalLocation\FoundFile.dat -> c:\Symlink\FoundFile.dat).

[therube]

Assuming I did it correctly (the other day), I set up a symlink (junction), then added that location as a Folder Index.

With that, a search found both:

c:\Some\Strange\Path\PhysicalLocation\FoundFile.dat
and
c:\Symlink\FoundFile.dat

[collinchaffin]

I'm still not tracking. If a file only exists ONCE, at least in my test cases where you have <NOT> criss-crossed junctions (anything can put you in a bad symlink loop just look at backing up "documents and settings" these days )

In my case, the tool should either tell me as the operator "I'm not goinng to provide you ANY data, even though it may be there, because I'm being activated within a junction (it knows this already, that's why it returns nothing), or, just do what every other tool these days does and blindly follow symlinks and if it loops and as the operator I'm a moron and criss-crossed junctions, then whoops the tool, just like every other one, sits and locks up and crashes. Personally, i'm okay with either but option 2 for me option 2 is better since it's what every other tool does from commercial backup tools like Crashplan, to Robocopy.

Anyway, back to my case, if Everything simply followed my junctions, it would not show any files more than once and whether it "displayed" the path within the junctioned structure, or somehow unrolled it to display the real path to me personally is not all that important. Others may disagree but they too right now have zero so I would think we all agree that getting the everything tool to at least do SOMETHING in the many locations that may contain symlinks is probably much better than leaving it the way it is which is to frankly kind of LIE to me as the operator and tell me no data is in a long line of structures, which in a case where I may strike the delete key (have you thought about this?), it just became a VERY DANGEROUS tool frankly to run on a storage device. if you stop and think about that I can strike delete on a folder in Everything thinking it's empty because I trust it, and it says there are no files.......

[collinchaffin]

I guess I can show you multiple screenshots on multiples systems here but if I run this tool within a folder path structure that has stemmed earlier from a JUNCTION, it ALWAYS displays ZERO files. I'll work on shooting you a screen grab video to demonstrate a bit later when I have a few minutes.

[collinchaffin]

Try your test with a JUNCTION to a folder on an ALTERNATE DISK. I think you will find that anywhere in the depth of that junction, even if 10 levels deep after that (which is where this gets very dangerous), the tool will show zero files and folders - no results. So I guess I take back I would be striking delete IN the tool, because the results are just broken and blank, however, a user could still be "convinced" from this quick check of a folder 10 levels deep after not having had the tool open and totally forgetting they are 10 levels under an old junction, see the ZERO results, and open a CMD prompt or switch to explorer or another tool and whack the 10 more levels of folders that everything just told them with no results was empty, when in fact 100gig and 10million files actually sat 3 levels into what they just deleted.

...Kind of an end of the world kind of example, but you get my point, LOL!

[void]

Thanks for the bringing this to my attention.

In Everything 838b or later you can now select a reparse point or junction when browsing for a folder.

For previous versions you can edit your Everything.ini to directly specify a junction.

Also see Allow custom remapping of file paths.
This method will use NTFS monitoring, which is much more reliable than folder indexing.

[therube]

(That's odd... I think the deal is...

On XP:
I created the symbolic link, adding that path to Everything (1.4.1.789b as it was) as a Folder Index.
Everything shows the Folder: as the specified link name (rather then the folder that the link points to).

And with that, the same files are found in both the "source" & the "link" directories".

On Win7:
It seems to me that with Win7, when the link path is added to Everything, the path gets "remapped" to the "source", so that a search only found the file from the source & not its "link".)



(I guess I've done this correctly?)

[void]

I created the symbolic link, adding that path to Everything (1.4.1.789b as it was) as a Folder Index.
Everything shows the Folder: as the specified link name (rather then the folder that the link points to).

And with that, the same files are found in both the "source" & the "link" directories".

This is what should happen.

It seems to me that with Win7, when the link path is added to Everything, the path gets "remapped" to the "source", so that a search only found the file from the source & not its "link".)

Everything 838b should make the behavior the same as XP in Windows 7 or later.
That is, you can add a reparsed folder / junction and everything will not remap it.

[yrwin]

I have another idea about this problem:
Everything can report, that there is a folder with this attribute. e.g. You have directory structure:
c:\DirectoryToSearch\RealFolder\FileToFind.txt
c:\DirectoryToSearch\SymlinkedFolder\FileToFind.txt

And when you search FileToFind.txt, result can be something like:

Code: Select all

c:\DirectoryToSearch\RealFolder\FileToFind.txt
#symlink c:\DirectoryToSearch\SymlinkedFolder

[void]

Thanks for the suggestion, a custom comment displayed next to the path could work well.

Color coding results is on my TODO list, with color coding you could make junctions green.

[collinchaffin]

Sorry for the late reply. I just tested build 846 from one folder deep into a junction structure and no matter what options, it still shows zero files present under it anywhere. Do I need to edit a setting on the new builds to make it work?

Example of my search:

Actual file path (no junctions):

C:\Data\Documents\Cloud\Scripts\Sync\*.* (lots of subfolders and files)

Go do drive D: in a junction back to above:

D:\Data\Scripts\Sync (Sync is a junction back to C: path above).....go here or any folders deeper, in this case let's go to:
D:\Data\Scripts\Sync\psScripts (10 subfolders and thousands files here shown in explorer), but using build 846 I use this path as the "search" path in Everything and tell it to show me * all files including subfolders. According to everything, it's still zero, empty, safe to delete perhaps (NOT!).

That's how I'm testing and a summary of the issue: Navigate anywhere into a structure at or below any NTFS junction to a folder on another drive, and EverythingUI will incorrectly return zero files or folders.

I really do appreciate you looking at this so quickly and hopefully it's close and just something minor still causing this.

[void]

Have you added the junction as a folder index or NTFS index?

[collinchaffin]

I've added these junctions using both the MS included mklink cmdline using the /J switch, and/or the popular shell extension "Hard link shell extension" at http://schinagl.priv.at/nt/hardlinkshel ... nsion.html that uses the exact same MS API calls to make the links that all require NTFS to function.

[void]

Everything doesn't follow junction points by default, you will need to manually add junctions as a folder or NTFS index.

[collinchaffin]

Okay then I'm very confused and feel like we've wasted a bunch of time explaining this over and over just to say it's not supported. This isn't "following" a junction perhaps I've explained this not so well?

Don't you think a file/folder search tool that allows you to search in a "starting" folder "and below" should at least throw an error or warn if it's being used in NOT A JUNCTION/SYMLINK but a true, real folder - but that "starting" path folder just happens to sit 10 levels deep under what happens to be a junction to another drive?

In other words, as it stands, this tool is rendered 100% broken and cannot find files literally in a folder that, in explorer, you can have open that contains files. Take this example to help better understand how bad this really is:

D:\Junction is a NTFS junction to C:\Junction - fine - now we know that if I sit in this junction folder that this tool won't function. However, now navigate to:

D:\Junction\Projects\Source\Github\Code - the Code folder is open in explorer and shows 1,000 files. Now right-click or hit the hotkey to open Everything tool to begin searching in this non-junction, normal NTFS folder "Code" that we can actually see with our eyes has files. Tell Everything to show all * files in current folder "Code" and below. The folder named "1" sits a level below "Code" and also has 1,000 files and is also a regular folder, not a symlink or junction. Explorer search shows 2,000 total files and 2 folders, the Everything tool incorrectly shows blank - zero results - the folder is empty despite neither the folder it is being launched from, nor the 1 folder below it being symlink or junctions.

So I guess if this is by design and this tool will not be able to ever function even in a real NTFS normal folder that is not the actual junction/symlink folder/link itself, but instead below it, then again I'll put it out there that to avoid potentially catastrophic actions from users that ONLY are looking at this tool for results before using a cmdline to whack folders, etc. that this tool at the very least is coded to RECOGNIZE as a new feature that it is being launched in the "danger Will Robinson zone" that is ANYWHERE at OR BELOW a junction point, and instead of showing zero search results, at least show a single error that it cannot operate in that structure. That to me would be the worst, least effort way out if you wanted to be the only search tool on the market that simply won't acknowledge or work anywhere within a junction/symlink structure - I think you have to at least be required to acknowledge it and tell the operator there is a compatibility issue.

Otherwise, the tool simply looks silly when you have 20 levels below a root junction to another drive and have to continually remind yourself NOT to use this tool ANYWHERE in that structure to prevent taking cmdline delete actions based on this tools' inability to traverse anywhere under these kind of links. Just my 2cents worth but I know I'll have no choice but to abandon using this tool and go back to Filelocator Pro because I have too many systems with too many drives with too many junctions to only be able to use Everything 10% of the time if it cannot be updated to begin following symlinks/junctions like I think every other search tool I have used for quite a while. These symlink/junctions have been around in UNIX for four decades, and in NTFS for almost 3 decades if you do not want to support them that is obviously your prerogative, but being such a powerful tool otherwise that decision would really surprise and shock me since they are obviously never going away and are still supported in ReFS. Hopefully you can run some more tests like my test above and see for yourself how the way it is today is really not such a good idea and give it some additional thought.

[collinchaffin]

So I've continued to update my nightly x64 build almost daily and haven't seen any other activity on this but have to say, I am truly blown away that the functionality to search FLAC tags of all things (and I'm actually an avid FLAC music user myself) is anywhere in the same UNIVERSE as what we are discussing here.

For a file/folder search tool that has always blown past the competition and frankly been able to continue to do that for one large reason and that is the ability to tie into specifically NTFS file system journaling - something missing from almost every other search tool and really what sets EVERYTHING apart (and always has) - if you put that into perspective it was NTFS in the MS world that finally added support for NTFS reparse point support.

....so for the leader of the NTFS low-level instant search to put searching music tags above truly catching up to every other search tool on the market that doesn't puke if run on an NTFS formatted drive containing various NTFS reparse points to other fixed NTFS drives and simply perform it's normal file-level searching and actually show files/folders - it leads me to ask - is there just an inherent problem due to exactly what has made the Everything tool so fast (searching NTFS journals) that makes trying to fix this either overly difficult or worse, is it possibly impossible since this tool uses the journal so that's why the contents are invisible because a reparse to another drive means a whole different DRIVE journal then needs to be searched, rather than the other tools that are simply doing directory crawling that makes following reparses easy?

I guess since searching file CONTENT for FLAC metadata tags is a crawler, perhaps the file/folder crawler fallback that has to exist already for searching the drives that don't have their journals indexed could be what is targeted for some kind of "fallback" searching when a simple detection is triggered that execution is happening within a reparsed structure (which is what every other non-journaling search tool does today)? This tool is SO much better than the others I'm just trying to hold out hope because doing away with system and data drive after drive just because they are utilizing a native NTFS supported function like a junction obviously is not going to be an option and I really do want to be able to keep using this search tool into the future!

[rgbigel]

I totally agree with @colinchaffin -- whose post is from 2017, but no real answer seems to have been given.

Using a file for each junction within a folder I would like to search is totally unpractical. Consider this, with search addressing backups with names\dates like \20110203:

\folder to search
\" \junction to search[\ 20110203] --> H:\Backups\20110203
\" other stuff*.*

If I now need to add another junction for a new backup folder (like \20200810), it would be very dangerous to forget adding that as an extra folder to index!!!!

As to the possible recursion in Junctions, I think it would be very simple to catch -- see XYPlorer.exe -- it just stops when the same path occurs again, and shows this, too. People who know about junctions typically are experienced enough to avoid such constructs anyway (except for those working at Microsoft).

Please do put this in your to-do list at the very top!

Rolf

[void]

Indexing folder junctions is not so difficult, it is maintaining the index where Everything will run into performance issues.

Every single file/folder change will need to check all of its parents to see if one is a folder junction.

I'll consider an option to index-only folder junctions and not maintain them.
I'll check the performance loss from maintaining these junctions and consider adding the option to monitor junctions.

Thanks for the suggestions.

[rgbigel]

Thanks for your enormous effort developing and maintaining this prog.

I understand the performance issues when updating the index. But looking at a program like "TreeSize Free" shows that there might be another option: you could just do a "live" traversal on request only. Following the selected folder (or, in my case, search filter) through the junctions works quite fast with the above program.

So, essentially, the (global) index would not be needed in this case, or maybe only a temporary one. If the results of the traversing must be stored there to do the display (whichever way this is implemented I would not know), it might be more of an effort?

This could be a great christmas present...

[void]

Thank you for your feedback.

I have on my TODO list: Add support for symbolic links when using NTFS indexing.
I have looked into it briefly. Some notes:

• Symbolic link targets are paths -I thought they might have been file ids, paths will make lookup slower.
• Symbolic link targets can point to anything, not just a folder on the same volume. -This makes indexing complicated, limiting this to symbolic link targets on the same volume might work.
• I will have to index all folders that have reparse tags and maintain this index. -Complexity and performance issues.
• All changes will need to check if they are in one of these symbolic link targets -CPU expensive, even when there is only one reparse point on the volume.

For now, please use folder indexing to follow symbolic links:

• In Everything, from the Tools menu, click Options.
• Click the NTFS tab on the left.
• Select your drive with symbolic links.
• Uncheck Include in database.
• Click the NTFS tab on the left.
• Click Add....
• Select your drive with symbolic links and click Open.
• Click OK.
  -Everything will scan your new folder index which will take a few minutes, you will be unable to search while this scan occurs-

Folder indexing is slower than NTFS indexing.
Everything will only update changes to the current path (not all symbolic links will update). To keep your Folder Indexes up to date, Everything will
schedule a rescan daily which can be customized from Tools -> Options -> Folders.

[JoeBlackBoy]

(sorry my bad english: i'm using a web translator)

Premise:
Everything seems to work with symbolic hardlinks, but not with real hardlinks.

For now I have found this solution:
- in Tools> Preferences> Indexes> NTFS I have removed the check to "include in the database" for each unit that contains the hardlinks
- in Tools> Preferences> Indexes> Folders I have added each unit that contains hardlinks

In this way in Everything I can see in real time all the hardlinks created in real time.

Probably this method makes Everything slower?
Void can certainly explain better than me how this solution works