Integrating with Everything to add folder sizes to Explorer

[m417z]

Hi, I thought you might find this small project of mine interesting around here.

The native file explorer doesn't show folder sizes, so I decided to see what it takes to add this ability. At first, I created a Windhawk mod which hooks the relevant function and calculates the folder size when it's requested. This approach was obviously slow for folders with a large amount of files (although not as slow as I expected), and several users suggested to do what Everything does, since it's so fast

I didn't feel like implementing a filesystem indexer, but I figured that I can use the Everything SDK, which I happily discovered to exist.

The result:



Check out the mod page for installation and for the source code: Better file sizes in Explorer details.

One thing that could still be improved is that currently, each folder size query triggers an IPC call, which in turn sends the WM_COPYDATA message and waits for a reply. It's not as slow as actually calculating the folder size, but it's slightly noticeable, especially when sorting by size. I'm not sure what can be an alternative which is faster but is still as convenient. It's also not a standard use case - given the Explorer source code, I'd just query all the necessary sizes in one go - so perhaps it's not worth optimizing for. Just sharing in case you have any ideas I didn't think of.

[void]

Thank you for creating Better file sizes in Explorer details.

I am currently working on the IPC for Everything 1.5.
I will add a simple API call to get folder sizes:
EVERYTHING3_USERAPI EVERYTHING3_ULONGLONG EVERYTHING3_API Everything3_GetFolderSizeW(LPCWSTR lpFileName);

Everything 1.5 IPC will use pipes instead of WM_COPYDATA which is much faster (~5x times faster to send all results)
There shouldn't be any delay if you keep the pipe connection alive.

[m417z]

That's great, looking forward to trying the new SDK.

[Kilmatead]

For what it's worth, having recently written a similar plugin for the Xplorer2 file manager, I simplified things slightly by using only the structure definitions from Everything and handling all the IPC myself though a persistent HWND_MESSAGE window but using WaitForSingleObject/SetEvent instead of bothering with GetMessage and all that (since the callback is automatically triggered upon receipt of the WM_COPYDATA reply anyway).

The point is just to say that I got the average query/response time down to below 3 milliseconds per query - which is not exactly dawdling, if you know what I mean, and is almost imperceivable (even when spamming 5000 query/responses when testing).

I too look forward to the 1.5 SDK (pipes are indeed even better), but I might add a small request (@void):

Since the 1.4 SDK doesn't really report an error if you try to query a folder object which isn't currently [size] indexed (such as a reparse-point) it's almost impossible to determine if the zero-size uint64_t return is because the folder size is actually 0 bytes (empty) or if it's simply not indexed (so I can then derive an alternative method). I know the roundabout method of registering reparse points as volumes in Everything itself, but for users who use hundreds of such links - power-users are a grumpy bunch - this method isn't practical.

So the request is a means of knowing if a specific folder object is itself [size] indexed or not, so there's no question about when 0 really means 0.

Thanks. (And not to hijack the thread.)

Also (@m417z): I couldn't tell by looking through your code, but on top of defining "path:wholefilename:" (I used "folder:wfn:" myself), be sure to define pQuery -> search_flags = EVERYTHING_IPC_MATCHDIACRITICS, so that "Pokémon" is always a distinct folder from "Pokemon". Just a thought, as this is not the default behaviour.

[NotNull]

Better file sizes in Explorer details.

a similar plugin for the Xplorer2 file manager

Nice, nice, very nice!
Your craftmanship has been added to the list of software that has Everything integrated, to make it easier to find for other interested people.

EVERYTHING_IPC_MATCHDIACRITICS

Consider also adding case sensitivity to the search queries.
A bit uncommon, but a single folder can contain FILE.txt, file.txt, FOLDER and folder, thanks to support for Windows Subsystem for Linux...

each folder size query triggers an IPC call

Not a developer myself, so the following might be nonsense ...

Instead of querying each folder ( let's say each folder under C:\Windows ) query the parent to list all childfolders, including size.
Using the command-line tool ES.exe, it can be done in one call (see below). Maybe something similar is available using the SDK ....

Code: Select all

T:\>c:\Tools\es.exe   folder:parent:"c:\windows"   -size
    14,614,989 C:\Windows\appcompat
    10,385,866 C:\Windows\apppatch
             0 C:\Windows\AppReadiness
   409,296,772 C:\Windows\assembly
       785,153 C:\Windows\bcastdvr
       574,523 C:\Windows\BitLockerDiscoveryVolumeContents
    71,239,574 C:\Windows\Boot
       104,800 C:\Windows\Branding
       200,498 C:\Windows\BrowserCore
             0 C:\Windows\CbsTemp
    52,503,568 C:\Windows\Containers
            68 C:\Windows\CSC
    11,501,377 C:\Windows\Cursors
         3,430 C:\Windows\debug
     4,482,956 C:\Windows\diagnostics
[...]

[m417z]

For what it's worth, having recently written a similar plugin for the Xplorer2 file manager

Looks great, well done! The code looks way cleaner than mine, my excuse is that I was optimizing for time to release
What's the license of the code? I'd love to borrow some of it if possible, of course not without mentioning you and linking to your plugin post.

The point is just to say that I got the average query/response time down to below 3 milliseconds per query - which is not exactly dawdling, if you know what I mean, and is almost imperceivable (even when spamming 5000 query/responses when testing).

I just did a quick test with my implementation and I got around 15 milliseconds per query. I'll look into adapting this method, and will post back the numbers with the new method on the same hardware.

Also (@m417z): I couldn't tell by looking through your code, but on top of defining "path:wholefilename:" (I used "folder:wfn:" myself), be sure to define pQuery -> search_flags = EVERYTHING_IPC_MATCHDIACRITICS, so that "Pokémon" is always a distinct folder from "Pokemon". Just a thought, as this is not the default behaviour.

Thanks for the tip, I'll add it.

[Kilmatead]

...case sensitivity... ...thanks to support for Windows Subsystem for Linux...

Thanks for that - it's funny no matter how thoroughly you look at something, there's always some aspect in the shadows ready to bite you right back.

Just for the fun of it (and as a proof-of-concept to convince the x2 dev that I wasn't completely crazy), an early version of the plugin actually utilised Everything's FRN: filter which I reckoned was about as targeted as you could get (who needs filenames anyway?), but the drawbacks of overhead-processing and NTFS-only scuppered that as a viable approach. It was a fun idea though - and it worked - but I always imagined with anthropomorphic glee Everything receiving the query and doing a double-take before complying.

(I'm easily amused that way.)

[m417z]

Consider also adding case sensitivity to the search queries.
A bit uncommon, but a single folder can contain FILE.txt, file.txt, FOLDER and folder, thanks to support for Windows Subsystem for Linux...

Good idea, I'll add that too, thanks.

Not a developer myself, so the following might be nonsense ...

Instead of querying each folder ( let's say each folder under C:\Windows ) query the parent to list all childfolders, including size.

That should be possible, but it complicates the implementation, and I decided to keep it simple unless performance becomes a problem and no simpler solution emerges. Querying all parent items also has a downside - when you open a folder in Explorer, and unless it's sorted by size, only the top visible items are queried. So if you e.g. open C:\Windows\WinSxS which has around 20k folders, only around 20 are initially visible. Which one of these is faster - querying 20k items at once or querying 20 items one by one - is a good question, but surely neither is optimal

That's what I meant by "given the Explorer source code, I'd just query all the necessary sizes in one go" - if it was an open source project and not a black box that needs to be reverse engineered, it'd be easier to figure out the exact list of folders that needs to be queried.

[Kilmatead]

What's the license of the code? I'd love to borrow some of it if possible...

No license at all (coding is all about learning, and that's all that matters), so take anything you like, my ego and I don't require any credit. Theoretically once compiled code then comes under the compiling license - or so I'd like to think, thus abjuring me of all responsibility in your potentially evil deeds.

Incidentally, the code was refactored for the recently released c23 revision (-std=c23), so I've been playing around with the GCC 14.2 LLVM 19.1.3 MinGW-w64 toolchain - which includes both clang and gcc compilers, and all the fun that entails. But make no mistake, it's pure C, not C++ (the nullptr keyword might give you a false sense of security). My project is really just a Total Commander plugin crafted for a "third-third-party by way of a fourth".

Just out of interest, what is WindHawk? The site is not awfully clear on that - it seems like a platform that brings shell-extension functionality back into explorer (which Windows so unkindly removed over the years in the name of user "safety"). I'd never heard of it.

and I got around 15 milliseconds per query

To get any kind of reliable times I had to switch to QueryPerformanceCounter, as GetTickCount seems somewhat unreliable under 10ms. I timed from when the query was allocated, filled, and sent to right after the callback return event. I don't have to worry about any display formatting routines, the host takes care of that. (I also went down the whole rabbit-hole of "hey, cool, microseconds! How many of these can I save?" sort of thing. It required rehab and everything, so beware of that demon.)

[Kilmatead]

...and unless it's sorted by size, only the top visible items are queried. So... only around 20 are initially visible.

That's what I meant by "given the Explorer source code, I'd just query all the necessary sizes in one go" - if it was an open source project and not a black box that needs to be reverse engineered, it'd be easier to figure out the exact list of folders that needs to be queried.

You mean the shell does that too? Or is that a WindHawk limitation? I discovered the same (frustrating) thing, but I assumed it was a limitation of the plugin architecture. In my case, the file manager has been extended to specifically call my plugin even for the "unseen" objects which makes my job easier, but only for this project. My other plugins suffer the "Can't see it? Can't process it!" dictum.

[m417z]

No license at all (coding is all about learning, and that's all that matters), so take anything you like, my ego and I don't require any credit. Theoretically once compiled code then comes under the compiling license - or so I'd like to think, thus abjuring me of all responsibility in your potentially evil deeds.

Great, borrowed, the new version is much better in all aspects - code size, stability and performance. Some extra thought was required for making it work in a multi-threaded environment. I hoped to have something elegant and lock-free, but ended up having 2 not-so-elegant mutexes. Oh well.

Regarding performance, quick and rough numbers are around 35 ms per query for the previous version and 15 ms for the new one (I used timeGetTime this time), but there's a large variance. A more end-to-end test is sorting a large folder, which in this test results in an impressive x6 speedup.

BTW one common feedback I received when releasing the first version was: what about Everything 1.5a? I saw that you added instructions in your post to disable the alpha_instance option. I ended up looking for the 1.5a window as a fallback, which is easier for users.

Just out of interest, what is WindHawk? The site is not awfully clear on that - it seems like a platform that brings shell-extension functionality back into explorer (which Windows so unkindly removed over the years in the name of user "safety"). I'd never heard of it.

Windhawk is a platform for creating, sharing and installing mods for Windows. A mod is a snippet of C++ code with special metadata which specifies where the snippet (which is compiled into a dll) is injected. Some APIs are available, such as for function hooking and debug symbol loading. You can think of it as plugins for any program on Windows

A simple example: when you rename a file in Windows and change its extension, Windows thoughtfully asks you for confirmation. There's no option to turn this confirmation off, but with Windhawk, you can just create a mod which injects into explorer.exe, hooks the ShellMessageBoxW function, and returns "yes" without showing the confirmation box. It's around 30 lines of code.

A longer introduction can be found here:
Windhawk, the customization marketplace for Windows programs

The documentation for getting started can be found here.

BTW Windhawk uses the same LLVM MinGW toolchain

...and unless it's sorted by size, only the top visible items are queried. So... only around 20 are initially visible.
...

You mean the shell does that too? Or is that a WindHawk limitation?

The GetSize function is called lazily once items are brought into view, and in my case it's a good thing, as the IPC is only fired for visible items. Sorting is more wild, GetSize is called multiple times per file as part of the sorting. I had to add caching for that, otherwise sorting becomes really slow.

[Kilmatead]

Which one of these is faster - querying 20k items at once or querying 20 items one by one - is a good question, but surely neither is optimal

This question kind of struck an amusing chord in my mind, so I did a simple test:

First, as a vague benchmark (it's awkward to create a separate timer thread just for this), I just roughly stopwatched a browsing into C:\Windows\WinSxS which on my system contains 16,400 folders, and letting my plugin query them consecutively took around 41 seconds, which essentially amounts to 41,000 milliseconds / 16,400 == 2.5ms, which is about what I expected given the margin for error.

It did, however, feel like clubbing a baby seal over the head; I guess the plugin passes the robust-ness test.

So setting up a specialised query of parent:"C:\Windows\WinSxS" (collecting not just size but also path/name information since you'd have to sort it all out yourself later), Everything took (on my system) 10ms to return a 4MB chunk of WM_COPYDATA, with the relevant goodies inside. It would take a hilarious amount of void's pointer-math leapfrogging to sort this into bitesize pieces for you to munch on, but you get the idea.

The answer to your somewhat rhetorical question is 41,000 milliseconds vs. 10 milliseconds (plus a lot of sorting, which probably would take more time to think about than to actually execute).

Not sure if it's worth the trouble or not, but there you go.

Great, borrowed, the new version is much better in all aspects - code size, stability and performance.

Hah, I've been chop-sueyed, spliced, daggered, and immortalised in code all at the same time - that was actually fun to read! Thanks for that. Glad it helped.

Incidentally, did you know that an expression of beauty like:

Code: Select all

g_gsReply.liSize = *(int64_t *)(((BYTE *)(list + 1)) + sizeof(EVERYTHING_IPC_ITEM2));

...really just ends up as a call to memcpy in the final ASM? Obviously you know that, of course, I'm just ranting - I found it rather depressing, but predictable. I like my way better. (Though I'm miffed you prefer BYTE to the more elegant byte_t notation - to each their own!)

And thanks for the info on WindHawk - worth a look, for sure. Odd that I never heard of it before, but first time for everything.

Incidentally I've been following that mstorsjo / llvm-mingw project for awhile now, and it gets updates every couple of weeks (if you're into bleeding-edge LLVM stuff). Most people don't update their compilers that often, but it's nice the world keeps churning out toys for us muggles to mess around with.

[m417z]

The answer to your somewhat rhetorical question is 41,000 milliseconds vs. 10 milliseconds (plus a lot of sorting, which probably would take more time to think about than to actually execute).

Not sure if it's worth the trouble or not, but there you go.

Great analysis, Everything is truly impressive! Now it's more tempting to do it, but due to the "void's pointer-math leapfrogging" and the fact that it's not a perceptible problem in my case unless sorting is used, I'll leave it for another day.

A problem that I'm more eager to solve is that when sorting by size, Explorer keeps files and folders separated. It does that not only for size, but for other fields such as modified date too. For size, it'd be much better to not have this separation.

Incidentally, did you know that an expression of beauty like:

Code: Select all

g_gsReply.liSize = *(int64_t *)(((BYTE *)(list + 1)) + sizeof(EVERYTHING_IPC_ITEM2));

...really just ends up as a call to memcpy in the final ASM? Obviously you know that, of course, I'm just ranting - I found it rather depressing, but predictable. I like my way better.

Here's what I see:

Code: Select all

    ; if (list->numitems) {
    ;     g_gsReply.liSize =
    ;         *(int64_t*)(((BYTE*)(list + 1)) +
    ;                     sizeof(EVERYTHING_IPC_ITEM2));
    ;     g_gsReply.bResult = true;
    ; } else {
    ;     g_gsReply.liSize = 0;
    ;     g_gsReply.bResult = false;
    ; }

    cmp dword ptr ds:[rax+0x04], 0x00
    jz short @L00000002
    mov rax, qword ptr ds:[rax+0x1C]
    mov cl, 0x01
    jmp short @L00000003
    ; ...

@L00000002:
    xor eax, eax
    xor ecx, ecx

@L00000003:
    mov qword ptr ds:[$$0F0D8], rax
    mov byte ptr ds:[$$0F0D4], cl

Also, what's "my way"? Isn't it the same? Perhaps I'm missing something.

I like my way better. (Though I'm miffed you prefer BYTE to the more elegant byte_t notation - to each their own!)

Well, it saves me a typedef. And it's more consistent with the rest of the code that's already saturated with WinAPI types.

If anything, since I'm already a sinner who uses C++, I'd go with std::byte for a more confirming code. And in C, I'd probably go with the standard uint8_t.

Edit: hmm, turns out that std::byte and unsigned char are special, while std::uint8_t isn't. TIL.

If we're into style review, I'd drop the L in LPSTR, and add a C where it's const. const is nice, and it's in C way before nullptr, and now in C23 they even added type-generic string functions (e.g. strstr, const in -> const out).

And thanks for the info on WindHawk - worth a look, for sure. Odd that I never heard of it before, but first time for everything.

Yeah, well, Windhawk isn't as popular as Everything. You're welcome! You have the vibes of somebody who's going to enjoy it

[Kilmatead]

Also, what's "my way"? Isn't it the same? Perhaps I'm missing something.

No, you're not missing anything, like I said, I'm just raving (and possibly not looking as pedantically as I should at what I write) - bloody optimising compilers take all the fun out of everything. Wait until someone shoves the contemporaneous pseudo-AI craze into it - then we're really doomed. I first learned C about 45 years ago when I was 12, and I've never left it, but sometimes it feels like it's trying to leave me. (I was a philosophy major in University, so these things hang heavily over my head, for no good reason.)

If we're into style review, I'd drop the L in LPSTR, and add a C where it's const. const is nice, and it's in C way before nullptr, and now in C23 they even added type-generic string functions (e.g. strstr, const in -> const out).

And it's more consistent with the rest of the code that's already saturated with WinAPI types.

Yeah, but you left the int64_t instead of going all LONGLONG on us. All this fuss over 8 little bytes.

I just can't let it rest: Some things are worth the price of a typedef! But yeah, if I was a more considerate coder, I would be far more diligent about using const where appropriate, but... they even "added" peremptory support for throwing exceptions into c... whatever happened to flying by the seat of your pants and living dangerously? I feel like yelling at the kids to get off my lawn.

Anyway, this was all a success... now we get to look forward to some rewrites whenever the 1.5 SDK drops...

[NotNull]

A problem that I'm more eager to solve is that when sorting by size, Explorer keeps files and folders separated. It does that not only for size, but for other fields such as modified date too. For size, it'd be much better to not have this separation.

Everything 1.5 has the possibility to mix files and folders when sorted by one of the indexed properties, like Name, Size, Path.
(see Menu => Tools => Options => Indexes to find out what is currently indexed).
Mixing files and folders is not available in version 1.4 due to different database structure.

The current SDK is based on Everything 1.4 and will be updated soon to support 1.5 and likely support mixed sorting too.
As a workaround for now, add mix-sort: to the search query. Example:
(note the postion of hiberfil.sys and pagefile.sys amidst the folders)

Code: Select all

T:\>c:\tools\es.exe   parent:c:\   mixsort:   sort:size  -size
27,679,114,967 C:\Windows
13,210,692,522 C:\Media
 9,386,144,284 C:\_DL
 7,038,909,110 C:\$Recycle.Bin
 6,745,325,568 C:\hiberfil.sys
 5,353,883,321 C:\Program Files
 4,510,073,744 C:\Program Files (x86)
 2,550,136,832 C:\pagefile.sys
 1,891,948,209 C:\Users
 1,603,295,091 C:\ProgramData
   691,650,898 C:\Tools
   123,974,165 C:\2install
    45,402,982 C:\Apps
    16,777,216 C:\swapfile.sys
        20,568 C:\System Volume Information
        12,288 C:\DumpStack.log.tmp
         4,660 C:\temp
         1,109 C:\Recovery
             0 C:\PerfLogs
             0 C:\Intel

it's not a perceptible problem in my case

Indeed; there are not a lot of folders with more than 100 subfolders. And most of those are not very "inhabitable" and will likely be visited sporadically, if ever ...

BTW: Type or paste the following search-query in Everything 1.5 to list the amount of subfolders per folder:

Code: Select all

folder:   add-column:childfoldercount   sort:childfoldercount-descending

[m417z]

Mixing files and folders is not available in version 1.4 due to different database structure.
...
As a workaround for now, add mix-sort: to the search query.

The limitation here is Explorer, not Everything. It receives the sizes and decides how to sort the list.

it's not a perceptible problem in my case

Indeed; there are not a lot of folders with more than 100 subfolders.

What I meant is that if the list isn't sorted by size, only the top visible folders are queried for size, so the speed penalty is limited by the monitor size, not by the amount of folders in the list. At least not until the user begins to scroll, at which point it's not too challenging to keep up with that.

[NotNull]

What I meant ...

Yes, I got that. The comment was about when sorting these folders by size.
The folders that have > 100 childfolders are likely not visited frequently, which mitigates the issue largely.

The limitation here is Explorer, not Everything. It receives the sizes and decides how to sort the list.

OK, missed that ..

[NotNull]

Anyway, this was all a success... now we get to look forward to some rewrites whenever the 1.5 SDK drops...

If you get bored looking forward (doubt that), a wild suggestion:

WizTree (disk space analyzer) and Directory Opus (file manager) both can show relative size / "Percent of Parent". Implementation is slightly different between the two. See screenshot below.

Would be cool if explorer and xplorer2 could do something similar, now they both are able to "speed-read" foldersizes

Percent of Parent.jpg (118.32 KiB) Viewed 4608 times

[Kilmatead]

...only the top visible folders are queried for size, so the speed penalty is limited by the monitor size, not by the amount of folders in the list. At least not until the user begins to scroll, at which point it's not too challenging to keep up with that.

I said this before, but I really am amazed that this seems to be the way it works - I thought I was the only one fighting it. I find it hard to believe MS somehow imagined Explorer as the original JIT paradigm...

there are not a lot of folders with more than 100 subfolders. And most of those are not very "inhabitable" and will likely be visited sporadically, if ever ...

I imagine there's more than a few people with oversized (and over-organised) music and photo collections which would beg to differ. (Just joking...)

[Kilmatead]

Would be cool if explorer and xplorer2 could do something similar, now they both are able to "speed-read" foldersizes

Doing that numerically is easy enough with yet another plugin, but as seen above, the whole "not processed until scrolled-through" creates a problem for sorting (which is what most people, I imagine, would like that kind of forensic information for (x2 has the usual pie-chart stuff for that). Sometimes fighting with the dev's of these programmes to get (what they tend to refer to as "fetishistic interests") fixed, is a losing proposition, God knows.

Not all developers are as open as void. No pun intended.

Not to mention, because of you I now have to install this linux subsystem thing just to see what kind of extra wrenches it may be tossing into the works that I haven't thought of yet... (joking again) forward-slashes may be another Y2K problem in the making for young and innocent parser algorithms...

[NotNull]

No need to install WSL:

Code: Select all

T:\>md demo

T:\>fsutil file setcasesensitiveinfo t:\demo
Case sensitive attribute on directory t:\demo is enabled.

T:\>cd demo

T:\demo>echo.> file.txt

T:\demo>echo.> FILE.txt

T:\demo>dir
 Volume in drive T is RAMDISK
 Volume Serial Number is C050-3B28

 Directory of T:\demo

09-11-2024  00:10    <DIR>          .
09-11-2024  00:10                 2 FILE.txt
09-11-2024  00:10                 2 file.txt
               2 File(s)              4 bytes
               1 Dir(s)     737.603.584 bytes free

T:\demo>

[Kilmatead]

No need to install WSL...

It's not that I didn't believe you (of course you're right), it's more what sparked my curiosity about other kinds of unintended things that I might find in there that could cause the end of the world. That's what I do, I mess with stuff until it breaks, so I can fix it to mess with other stuff... it's a lifelong occupation.

(Though I am disappointed it doesn't seem to employ forward slashes...?)

[m417z]

I did it!

BTW: Type or paste the following search-query in Everything 1.5 to list the amount of subfolders per folder:

Code: Select all

folder:   add-column:childfoldercount   sort:childfoldercount-descending

I skimmed over it too quickly last time, that's really slick!

[m417z]

@Kilmatead looks like you missed the ChangeWindowMessageFilterEx call that exists in the official SDK. I missed it too until a user reported that elevated programs get a timeout (at least there's a timeout now!). Maybe it makes less sense running x2 elevated, but there might be valid use cases, and all it takes to support it is a single call.

[Kilmatead]

looks like you missed the ChangeWindowMessageFilterEx call that exists in the official SDK.

I hadn't missed it, I was just under the impression that didn't apply to messages below WM_USER (that's not a great authoritative link, but I can't find the one I was going by right now). (0x004A < 0x0400)


ChangeWindowMessageFilterEx

Certain messages whose value is smaller than WM_USER are required to be passed through the filter, regardless of the filter setting. There will be no effect when you attempt to use this function to allow or block such messages.

It's the vague "certain" part of that... not a good thing to do with documentation.

Just in case, I did all sorts of tests against elevated/non-elevated processes and I never received an access failure on full UAC so I didn't worry about it (nor, have I had any user-problems with the plugin). Maybe it has more to do with your actual hooking of the explorer process (which is more invasive)?

However, if it relieves even a possible problem, you're right, a single call is a single call.

What, though, does "elevated programs get a timeout" mean? Never heard of that one. I used SendMessageTimeoutW more in an abundance of caution (against process hangs) over actual causation, but that shouldn't apply to UIPI.

There's also the small matter that MessageFilter might need LoadLibraryW(L"user32.dll") for XP support (which is also in the SDK), but I was hoping that was too legacy... maybe it isn't... except XP doesn't suffer from UIPI. What?

[m417z]

It's the vague "certain" part of that...

Probably a better word would be "selected" messages. Here's some research on the topic.

Just in case, I did all sorts of tests against elevated/non-elevated processes and I never received an access failure on full UAC so I didn't worry about it (nor, have I had any user-problems with the plugin). Maybe it has more to do with your actual hooking of the explorer process (which is more invasive)?

I don't think so. I'd expect that if you'll run Everything as non-elevated, and x2 as elevated, you'll encounter this problem. Does it just work for you?

What, though, does "elevated programs get a timeout" mean? Never heard of that one. I used SendMessageTimeoutW more in an abundance of caution (against process hangs) over actual causation, but that shouldn't apply to UIPI.

SendMessageTimeoutW works, because elevated->non elevated is not a problem, but when Everything tries to reply, that doesn't work. The query then times out in WaitForSingleObject.

There's also the small matter that MessageFilter might need LoadLibraryW(L"user32.dll") for XP support (which is also in the SDK), but I was hoping that was too legacy... maybe it isn't... except XP doesn't suffer from UIPI. What?

IMO it's time to let XP go. Latest MSVC doesn't even have an option to produce executables which run on it. In my case, Windhawk only supports Win7 and up anyway, and it might change to Win10/Win11-only in the not-so-distant future.

[Kilmatead]

Here's some research on the topic.

Ah yes, I remember reading that a few years back... well, most of it anyway. Between marshalling protocols and exactly what the manifest dictates, my eyes started to wander...

I'd expect that if you'll run Everything as non-elevated, and x2 as elevated, you'll encounter this problem. Does it just work for you?

Exactly what I thought was supposed to happen, but I could never get it to fail, so I gave up and decided to pursue it only if people complained (no one did - but possibly no one uses my stuff?). It's a pain to create user accounts and change elevations and restart the machine 4 or 5 times in an afternoon just to test a hypothesis I could never prove or disprove. It's also not impossible that I've so messed up my security setup over the years that I managed to change something permanently which I shouldn't have, so my testing might be meaningless.

I'm also not a "real" developer (in the sense of anyone having a monetary stake in me), so maybe people aren't worried enough to complain directly to me. They do about bugs though... those complaints I hear. And requests. Always requests.

The query then times out in WaitForSingleObject.

Did you encounter this? That's what it's there for, after all... but still, I keep wondering about exactly how much of a DLL's self-created window is actually under the aegis of the process's messaging system - especially for a window which is created explicitly for and dedicated to messaging. I also doubt the danger of errant WM_COPYDATA messages... if a process isn't set up for them, I assume (always a bad idea!) that DefWindowProcW would just delete them and deallocate their resources and be done with it. The WinAPI is a strange place, though, so one never knows.

IMO it's time to let XP go. Latest MSVC doesn't even have an option to produce executables which run on it.

Yeah I use the compiler targeted for MSVCRT simply because x2 still supports those older machines, so it's not quite time to fully embrace the UCRT (if it was, no one would bother creating MSVCRT builds!).

I did it!

I assume the solution is in the latest Windhawk link, so I should read it again? Is this the "scroll to sort" problem?

[m417z]

I'd expect that if you'll run Everything as non-elevated, and x2 as elevated, you'll encounter this problem. Does it just work for you?

...
It's also not impossible that I've so messed up my security setup over the years that I managed to change something permanently which I shouldn't have, so my testing might be meaningless.

Yeah, that can explain it, otherwise I'd expect it to be easily reproducible.

The query then times out in WaitForSingleObject.

Did you encounter this?

Yes, that's what I'm seeing, it ends up failing with ES_QUERY_REPLY_TIMEOUT.

I did it!

I assume the solution is in the latest Windhawk link, so I should read it again? Is this the "scroll to sort" problem?

When sorting by size, files end up in one separate chunk, and folders in another (see screenshot below). That's the default Explorer behavior, which also applies when sorting by other columns. I added an option changes sorting by size to disable this separation.

You can see the changelog on the mod page. Each version also links to the code on GitHub, where you can also see the code history and diffs.

It's no longer a problem related to querying sizes or to Everything, it's just making Explorer behave in a way that makes more sense after it got the sizes.

[Kilmatead]

Yes, that's what I'm seeing, it ends up failing with ES_QUERY_REPLY_TIMEOUT.

Fair enough - I can hardly argue with my own error message. God knows that's enough said on the subject.

When sorting by size, files end up in one separate chunk, and folders in another...

Ok, bear in mind I'm asking this before perusing the code (it's the end of a day of manual labour and I need a shower and sustenance before else), but if I recall correctly, your code receives the objects initially as PIDLs, no? Surely you're not querying non-folders for size via Everything when you can just do it natively at a fraction of the cost.

Everything is absolutely wonderful for folder-sizes, but (assuming you still have that curious 15 millisecond handicap), files would just slow you down. That's how my plugin works - files are never even submitted for consideration.

If, on the other hand, I'm wildly mistaken and am misremembering the code I reviewed the first time, feel free to turn up your expensive collar and ignore this poor pretender's protestations of petition.

[m417z]

When sorting by size, files end up in one separate chunk, and folders in another...

... if I recall correctly, your code receives the objects initially as PIDLs, no? Surely you're not querying non-folders for size via Everything when you can just do it natively at a fraction of the cost.

Essentially, I call the original GetSize function, and proceed with querying Everything only in case of an empty result. In practice, in most (all?) cases it means that files are handled by Explorer, and folders are handled by the mod. If at some point in the future Windows learns to query folder sizes, it should bypass Everything altogether.

[m417z]

A user reached out to me with an interesting problem. Can you spot the root cause?
https://streamable.com/m631xz

From a quick analysis, here's what's going on:

• Everything opens the folder by using SHOpenFolderAndSelectItems
• Explorer opens the folder and lists the contained files and folders
• For each folder, the mod kicks in and queries Everything for the folder size
• Meanwhile, Everything is blocked in SHOpenFolderAndSelectItems, which for some reason seems to return only after the list is fully loaded
• The mod times out, resulting in a long delay and a lack of folder sizes
• Afterwards, refresh works fine as Everything is no longer blocked

On Everything's side, a workaround can be to either run explorer.exe explicitly or to run SHOpenFolderAndSelectItems in another thread. Neither is pretty, but also not horrible. An upside is that regardless of the mod, opening a folder won't block Everything, but I don't think anyone ever complained about that.

On the mod side, I can do the same for Everything.exe by hooking SHOpenFolderAndSelectItems which is the short-term solution I have in mind right now. The mod injects into all processes, including Everything, anyway.

Any better ideas?

BTW interestingly, the problem always happens with the old ribbon UI of Explorer (what's on the video), but only rarely happens with the new UI (on the other screenshots in this thread). Perhaps they changed some of the code to become more asynchronous.

[Kilmatead]

The mod times out, resulting in a long delay and a lack of folder sizes

What part of the mod times out - outgoing, or incoming? Or this this related to your mutexes? This aspect of hooking is your bailiwick, not mine - unless void can chime in more knowledgeably. I can reproduce it though, ribbon or no ribbon, so something's odd.

On a (probably) completely unrelated note, your mod seems to inexplicably fail on the "C:\Windows\assembly" folder - when viewed simply browsing to C:\Windows itself (all other folders surrounding it populate correctly), which is curious because ES doesn't have a problem retrieving its size (855MB for me), if I query it directly (my mod). To be fair, that folder (or at least its contents) are "different", but there's no reason to browse into it to query its size and a refresh does not help, in this case.

It's the only other place I've seen your mod fail to print anything (blank, as opposed to 0 bytes).

[m417z]

The mod times out, resulting in a long delay and a lack of folder sizes

What part of the mod times out - outgoing, or incoming? Or this this related to your mutexes? This aspect of hooking is your bailiwick, not mine - unless void can chime in more knowledgeably. I can reproduce it though, ribbon or no ribbon, so something's odd.

Incoming - ES_QUERY_REPLY_TIMEOUT. SendMessageTimeout succeeds, but WaitForSingleObject times out since WM_COPYDATA doesn't arrive until the whole list is loaded and Everything is no longer blocked.

I created a simple test mod which hooks SHOpenFolderAndSelectItems and runs it in a new thread. That fixes the problem. I'll look into integrating it with the main mod. To try it, open Windhawk, click the bottom right button to create a new mod, replace the code with the test mod code, click "Compile Mod", then "Exit Editing Mode".

On a (probably) completely unrelated note, your mod seems to inexplicably fail on the "C:\Windows\assembly" folder - when viewed simply browsing to C:\Windows itself (all other folders surrounding it populate correctly), which is curious because ES doesn't have a problem retrieving its size (855MB for me), if I query it directly (my mod). To be fair, that folder (or at least its contents) are "different", but there's no reason to browse into it to query its size and a refresh does not help, in this case.

It's the only other place I've seen your mod fail to print anything (blank, as opposed to 0 bytes).

I can't reproduce it, not on my real computer and not in a VM. Can you post debug logs?

• Go to the mod's Advanced tab.
• Set Debug logging to Mod logs.
• Click on Show log output.
• Open C:\Windows, make sure the assembly folder is visible.
• Set Debug logging back to None, grab the logs.

I'd expect a log line saying "Failed to get size: ...".

[Kilmatead]

Not most detailed of logs (?) - there's just a single "Failed: 80004002", but not much else out of place (this is just the relevant fraction):

Code: Select all

11:35:11.603 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.605 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [694:CFSFolder__GetSize_Hook]: Done: 0
11:35:11.605 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.606 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.606 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.607 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.607 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [665:CFSFolder__GetSize_Hook]: Failed: 80004002
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.610 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.611 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.613 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [694:CFSFolder__GetSize_Hook]: Done: 785153
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.614 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [823:PSFormatForDisplayAlloc_Hook]: >
11:35:11.615 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [639:CFSFolder__GetSize_Hook]: >
11:35:11.618 4132 explorer.exe  [WH] [explorer-details-better-file-sizes] [694:CFSFolder__GetSize_Hook]: Done: 69937630

The "empty" bit survives a machine restart and all, it's consistently failing just in that one folder.

[m417z]

Not most detailed of logs (?) - there's just a single "Failed: 80004002", but not much else out of place (this is just the relevant fraction):
...

Seems like this fails with E_NOINTERFACE:

Code: Select all

winrt::com_ptr<IShellFolder2> childFolder;
hr = shellFolder2->BindToObject(itemidChild, nullptr,
                                IID_PPV_ARGS(childFolder.put()));

No idea why. If you want to investigate it, maybe you can create a regular program which does something similar and see if you get a similar error.

[void]

Since the 1.4 SDK doesn't really report an error if you try to query a folder object which isn't currently [size] indexed (such as a reparse-point) it's almost impossible to determine if the zero-size uint64_t return is because the folder size is actually 0 bytes (empty) or if it's simply not indexed (so I can then derive an alternative method). I know the roundabout method of registering reparse points as volumes in Everything itself, but for users who use hundreds of such links - power-users are a grumpy bunch - this method isn't practical.

Everything calculates the folder size from all the child files and grandchild files in your index.
Everything doesn't follow folder junctions with NTFS parsing.
Everything will report folder junctions with a size of 0.

The SDK will return UINT64_MAX (0xffffffffffffffffui64) if folder size indexing is disabled.

One idea is to check the folder for the FILE_ATTRIBUTE_REPARSE_POINT (0x00000400) flag and handle it differently.

Also (@m417z): I couldn't tell by looking through your code, but on top of defining "path:wholefilename:" (I used "folder:wfn:" myself), be sure to define pQuery -> search_flags = EVERYTHING_IPC_MATCHDIACRITICS, so that "Pokémon" is always a distinct folder from "Pokemon". Just a thought, as this is not the default behaviour.

Consider also adding case sensitivity to the search queries.

The new Everything3_GetFolderSizeW API will lookup folders by case first.
A case insensitive search is performed if a folder is not found.
Diacritics are always matched.
You could emulate something similar with Everything 1.4.

Meanwhile, Everything is blocked in SHOpenFolderAndSelectItems, which for some reason seems to return only after the list is fully loaded

I have put on my TODO list to move SHOpenFolderAndSelectItems into a thread.
SHOpenFolderAndSelectItems doesn't return until the list is loaded.
Can you load the folder sizes after the list is loaded?

[m417z]

Meanwhile, Everything is blocked in SHOpenFolderAndSelectItems, which for some reason seems to return only after the list is fully loaded

I have put on my TODO list to move SHOpenFolderAndSelectItems into a thread.
SHOpenFolderAndSelectItems doesn't return until the list is loaded.
Can you load the folder sizes after the list is loaded?

Not easily. It's easier for me to just hook SHOpenFolderAndSelectItems and run it in a new thread. I'll make an extra check that I'm running in the thread that owns the Everything main window, so that once Everything starts doing that too, the mod won't spawn yet another thread.

[Kilmatead]

For anyone interested, playing around with the initial release of the SDK Version 3 (added with ES build 1384) a quick preliminary testing with the pipes using the most simplistic call-per-folder:

Code: Select all

_everything3_client_t *pClient = Everything3_ConnectW(L"1.5a");
if (pClient) {
	FolderSize = Everything3_GetFolderSizeFromFilenameW(pClient, FileName);
	Everything3_DestroyClient(pClient);
}

...results in a response time (in microseconds) ranging from 80µs (min) to 250µs (max). If you leave the pipe open (only connect once and don't call DestroyClient iteratively) those times reduce to 40µs and 110µs respectively.

When you're talking millionths-of-a-second, the deltas of leaving the pipe open (while still halved) are negligible in terms of human perception to using the safer (?) method of connecting/destroying "on demand", as it were. Just something to think about. Without further testing, I don't know how fragile an emotionally upset pipe connection is or how easy it is to pacify it.

Anyway, just thought I'd show the experimental prelim times.

For anyone compiling the entire SDK3 themselves (the only choice at the moment), be forewarned that the code will need a little rejigging if you're using GCC/Clang instead of MSVS. Nothing major (some incompatible function-pointer errors and non-portable numerical suffixes), so it's not quite as "works right out of the box" as the 1.4 SDK was.

As void mentioned earlier it automatically handles diacritics, case, et al, so there's very little left to tinker with... except the inevitable reduction to non-monolithic use.

[m417z]

...results in a response time (in microseconds) ranging from 80µs (min) to 250µs (max). If you leave the pipe open (only connect once and don't call DestroyClient iteratively) those times reduce to 40µs and 110µs respectively.

Sounds promising!

And thanks for adding the SHOpenFolderAndSelectItems in a separate thread handling! Although I still need to add the workaround to the mod for Everything 1.4.

[void]

You beat me to it!

Everything 1.5.0.1384a adds support for the Everything 1.5 SDK.

There's still a few things to do here.
I need to build the DLLs and and in support for the old SDKs.
For now, you can build the SDK yourself if you would like to have a play.



Everything 1.5.0.1384a will now open paths in a background thread.

To open paths in the UI thread:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  open
• Select: open_folder_and_selected_items_thread
• Set the value to: false
• Click OK.

[m417z]

@Kilmatead I saw that you quietly added support for the Everything 1.5 SDK. Great work once again at minimizing the SDK to only contain the necessary parts. I shamelessly copied it once again it added about 1,000 lines to the mod, and now Everything SDKs are almost half the code.

Here's the updated mod (not officially released yet).

I did some basic measurements on a slow VM: Previously, it took 7.082 ms per query. Now, it takes 0.3777 ms per query. That's almost 20x speedup, and it feels truly instant now. Trying an extreme case, sorting C:\Windows\WinSxS (17,750 folders) takes about 8 seconds now, while it took an eternity (about 2 min) before.

BTW at first I was confused why it doesn't work, only to discover that the alpha download page doesn't link to the latest version for some reason.

[Kilmatead]

@Kilmatead I saw that you quietly added support for the Everything 1.5 SDK. Great work once again at minimizing the SDK to only contain the necessary parts. I shamelessly copied it once again it added about 1,000 lines to the mod, and now Everything SDKs are almost half the code.

Well, not so quietly - I added a post to the end of the thread (rambling nonsense about Madame Curie and the coincidence that the half-life of polonium-214 "just so happens to be" the same speed as the plugin now - 164μs ), but you must have checked the link while I was writing it. I was going to mention it here just after but you found it too fast. Glad it was helpful - obviously ES handles stuff internally as UTF-8 and void's meticulous conversions are most if it, so there wasn't much else to do but cut the superfluous stuff. Half the plugin size is just in supporting the legacy copydata method now.

I notice you dropped my ES_QUERY_NO_INDEX condition... what do you do with reparse points and/or unindexed locations - just default to your manual-approach and hope the user doesn't notice? I was happy enough to at least get to the point of having "reliably empty" 0-size folders.

Trying an extreme case, sorting C:\Windows\WinSxS (17,750 folders) takes about 8 seconds now, while it took an eternity (about 2 min) before.

It's interesting that you still seem to be slower than mine - must be your mutexes are greedy (as mine is just single-threaded). I too did the WinSxS test and mine came down from 41 seconds to 4 seconds, which is in line with expectations (about a thousand fewer folders than yours too for some reason). Anyway, since QueryPerformanceCounter is consistent, I'm satisfied.

As I mentioned previously, I did some testing with leaving the pipe "open" for multiple calls, but if one or the other EXE were to lose the connection it's very difficult to reliably reconnect... Everything just seems to become a sulky teenager and hide in its room, no matter if I close the lost pipe or not - it's not interested in kicking into server-mode unless restarted itself. So, the conservative method wins, even if it's a few ticks slower. You could try leaving it open just for the time it takes to read the subfolders of the main folder, then close it for safety until the user browses to another location, and then do it again. Seems fiddlesome though. I can't really do that as file-managers often have "collections" of folders from disparate locations which must be parsed sequentially.

I also found trying to connect to his "unnamed instances" somewhat hit-and-miss - when the comments in the SDK "say" a servername may be NULL, I found using L"1.5a" more reliable. I couldn't see anything in the code to explain it... but my testing has never been rigorously scientific and we're only privy to half the equation.

...doesn't link to the latest version for some reason.

It does if you click on the "changes" link that's there and scroll up - it's the forum's main 1.5 page with the download plus changelogs. I didn't even know that 1.5 had its own landing page, I always just lurk on the forum page to see the endless requests and bug reports. Exactly how void finds the time to read the forum and programme his toys, I have no idea - he may not actually sleep.

[m417z]

I was going to mention it here just after but you found it too fast

It seems to be a pure coincidence, I randomly went to download the latest version of your zip, not thinking I'd find the new SDK in there

I notice you dropped my ES_QUERY_NO_INDEX condition... what do you do with reparse points and/or unindexed locations - just default to your manual-approach and hope the user doesn't notice?

I was returning whatever Everything returned, zero or -1. Just before you posted your comment (another coincidence) I added your check of reparse points. I'm fine with -1 values if indexing is disabled. It actually helps to diagnose the problem when a user says all values are -1 (RTFM and enable folder indexing).

So, the conservative method wins, even if it's a few ticks slower

Totally, it'd rather not add hundreds of fragile code lines for a speedup that's imperceivable in the vast majority of cases.

[Skodd]

great project, didn't know about WindHawk. It's a shame I started using DIrectory Opus.

[m417z]

I just released a mod update with SDK3 support, and shortly after that got a bug report from a user. It looks like there's a difference in handling subfolders of folder junctions in the new SDK. The new SDK returns -1 in this case, while the old SDK returns zero.

To reproduce, create a folder junction, for example:

Code: Select all

C:\Users\User\Desktop>mklink /j jtest C:\Users\User
Junction created for jtest <<===>> C:\Users\User

Then query the size of a subfolder in this folder junction, for example: C:\Users\User\Desktop\jtest\Desktop.

Is the new behavior an intended change?

Now, thinking about it, it's probably better for the mod to just never show -1. Even if this change is unintended, there can be a user who intentionally doesn't index some folders/drives, and showing -1 values in their face is not pretty.

BTW @Kilmatead in your code, you seem to check for -1 values only for the new SDK. Does that mean that when using the old SDK, -1 values are actually displayed for folders which aren't indexed?

[Kilmatead]

BTW @Kilmatead in your code, you seem to check for -1 values only for the new SDK. Does that mean that when using the old SDK, -1 values are actually displayed for folders which aren't indexed?

Read my initial posting way back at the start of this thread and you'll see I made a request of void for a better indicator of "truly" empty folders as opposed to merely unindexed folders, as SDK2 returned 0 for any and all faults (folder not found, folder empty, folder reparse, etc) and only had a general "indexing option disabled" indicator which was essentially useless - except for, in your own eloquent words, dealing with the RTFM crowd.

void's compromise was to return -1 (or ULL max) for errors and 0 for actual 0-size. This allows writing a GetReparseTargetW function (using DeviceIoControl/FSCTL_GET_REPARSE_POINT) to get the link's target and then submit that more accurate path to ES for querying. And this worked fine... I actually did all the code for it, but removed it in the end simply because while it reported the size of the destination properly, if the user subsequently browsed into that link the resulting paths were invalid as far as ES was concerned (unless I cared to walk the path and check segments for reparse attributes, etc), so it was only half-successful, and 0's were returned, giving essentially misleading results. I don't like half-arsed solutions to things, so I just delegated any folder that was not a legitimately empty folder (0-size) to the file-manager and let it do it the slow way.

This way you get the choice of what to send back to the user... a vaguely misleading result like "well, it might be empty", or you can just say "out to lunch", or you can deal with it properly... (including extended errors like links to temporarily unavailable objects, etc).

(If you like the idea of manually resolving reparse and junction points my well-tested take on the GetReparseTargetW idea may be found in the source for the original proof-of-concept plugin I made before I worked with the author of x2 to get it properly integrated.)

I don't know why ES handles such links differently, but many programmes also have different takes on how to deal with them (image-backup applications, etc), so I decided a long time ago to just handle them myself if necessary in anything I wrote. A painful lesson, but a good one. In this case (as described above), I decided against it, but the jury's still out...

The long and short of it is that SDK2 never returned -1, so yeah that change in SDK3 is intentional (and all the better for it, in my opinion - thank you, void).

Sorry for the extended explanation... I tend to blather on a bit when hungry...

[m417z]

only had a general "indexing option disabled" indicator which was essentially useless
[...]
void's compromise was to return -1 (or ULL max) for errors and 0 for actual 0-size.

Thanks, now I have a clearer picture. So in the old SDK, -1 is only returned if folder indexing is disabled globally, which works well for the RTFM use case.

The new SDK, on the other hand, returns -1 for anything that's not an actual size - indexing is disabled globally, a specific folder is excluded, an error, etc. It makes more sense, although I no longer have the free RTFM-crowd detector. I can always query whether indexing is disabled globally and show -1 myself, but I don't think it's worth is. Also, if I remember correctly, 1.5a enables folder size indexing by default, so having it disabled should be much less common.

This allows (if desired) writing a GetReparseTargetW function (using something like DeviceIoControl/FSCTL_GET_REPARSE_POINT) to get the link's target and then submit that more accurate path to ES for querying. And this worked fine... I actually did all the code for it, but removed it in the end simply because while it reported the size of the destination properly, if the user subsequently browsed into that link the resulting paths were invalid as far as ES was concerned (unless I cared to walk the path and check segments for reparse attributes, etc), so it was only half-successful, and 0's were returned, giving essentially misleading results.

Actually, I had a similar quick journey. I wrote a simple function based on existing code and some StackOverflow answers material. It worked for a basic test (the code is below), but subfolders didn't resolve, and then I realized I'd have to walk the path, etc., as you described (and handle possible loops too). For now, I decided to just show no size for junction folders.

Code: Select all

// Code based on:
// https://github.com/transmission/transmission/blob/f2aeb11b0733957d8d77d7038daa3ae88442dd5b/libtransmission/file-win32.cc
std::wstring NativePathToPath(std::wstring_view wide_path) {
    constexpr WCHAR NativeLocalPathPrefix[] = L"\\\\?\\";
    constexpr WCHAR NativeUncPathPrefix[] = L"\\\\?\\UNC\\";

    if (wide_path.starts_with(NativeUncPathPrefix)) {
        wide_path.remove_prefix(ARRAYSIZE(NativeUncPathPrefix) - 1);
        std::wstring result{wide_path};
        result.insert(0, L"\\");
        return result;
    }

    if (wide_path.starts_with(NativeLocalPathPrefix)) {
        wide_path.remove_prefix(ARRAYSIZE(NativeLocalPathPrefix) - 1);
        return std::wstring{wide_path};
    }

    return std::wstring{wide_path};
}

std::wstring ReparseResolve(PCWSTR path) {
    if (auto const handle = CreateFileW(
            path, FILE_READ_EA,
            FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, nullptr,
            OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, nullptr);
        handle != INVALID_HANDLE_VALUE) {
        if (auto const wide_ret_size =
                GetFinalPathNameByHandleW(handle, nullptr, 0, 0);
            wide_ret_size != 0) {
            auto wide_ret = std::wstring{};
            wide_ret.resize(wide_ret_size);
            if (GetFinalPathNameByHandleW(handle, std::data(wide_ret),
                                          wide_ret_size,
                                          0) == wide_ret_size - 1) {
                // `wide_ret_size` includes the terminating '\0'; remove it from
                // `wide_ret`
                wide_ret.resize(std::size(wide_ret) - 1);
                return NativePathToPath(wide_ret);
            }
        }

        CloseHandle(handle);
    }

    return std::wstring{};
}

[Kilmatead]

For now, I decided to just show no size for junction folders.

Exactly. It feels like an imperfect solution, but the alternative is what you previously called "hundreds of fragile code lines".

There is, to be fair, a method of getting ES to index reparse points as individual volumes which does actually solve our problems (it allows the relative paths to resolve), except it needs to be done specifically for each individual link, which is completely impractical for users who create/move/destroy many such links regularly.

Since your mod also includes the "old code" for computing the sizes manually, you could always just default to that for any links, no? (Slower, yeah, but some might prefer it to 0... or does that mess up your recent sorting-fixes? Plus, there's always going to be some heckler in the audience who creates a link to C:\Windows and then complains about how long it takes...)

[m417z]

Exactly. It feels like an imperfect solution, but the alternative leads to... what you previously called "hundreds of fragile code lines".

Actually, after some more experimentation, I seems that just using GetFinalPathNameByHandle works well enough. It resolves junctions in the middle of the path, as well as junctions to junctions. I didn't test loops, but I guess it just fails in this case. Pseudo-code:

Code: Select all

int64_t size;
unsigned result = es_get_size(folder_path, &size);

if (result == ES_QUERY_NO_INDEX) {
  string resolved_folder_path = GetFinalPathNameByHandle(folder_path);
  if (resolved_folder_path.succeeded && resolved_folder_path.str != folder_path) {
    result = es_get_size(resolved_folder_path.str, &size);
  }
}

Real code.

One small pessimization with this solution is that if ES_QUERY_NO_INDEX is returned for a justified reason, e.g. the folder is excluded in Everything. I don't know that, so I'll try to resolve it unnecessarily. But I think it's acceptable.

Since your mod also includes the "old code" for computing the sizes manually, you could always just default to that, no? (Slower, yeah, but some might prefer it to 0... or, come to think of it, does that mess up your recent more tactical sorting successes?)

It really doesn't work all that well. The whole UI is frozen until calculation is done. If there are many subfolders and/or the disk is slow, the UI can become frozen for a long time, while the disk is spinning. So I'd rather not resort to that.

[Kilmatead]

...just using GetFinalPathNameByHandle works well enough.

It's a poor-man's GetReparseTarget since it fails for any paths which may be temporarily unavailable (detached drive, etc), and can't provide the reparse buffer contents (the true path) to compensate. Though as that doesn't tend to much matter for non-power-users, it has potential.