How is Everything so efficient?

[planeW]

Hello!

I've been using Everything for so many years now and as a software developer it always felt like this is too good to be true and its developer must be cheating somehow

On the one hand Everything is incredibly fast at searching, which indicate that the in-memory data structure for the database is likely a sorted array (or multiple array chunks) for its fast lookup times, cache friendliness and great scaling with multiple threads. Maybe the array items also store the index or memory address of their parent folders, for fast path building and to save memory.

But on the other hand Everything also seems to be highly dynamic, in that there are constantly files and folders being removed or added to the database, which wouldn't work that well with an array, since inserting items sorted or deleting them involves a lot of moving memory around and presumably updating lots of references to items.

I also thought that maybe those changes would be stored in different arrays, like a database diff, but I suppose this would impact the search performance too much.

So does someone know how Everything is so fast and dynamic at the same time? Because I'm always hesitant to use arrays with lots of entries once adding and removing items becomes a common operation. Or maybe I'm just underestimating how fast computers nowadays are.

Thanks!

[rollsch]

Would love to know more about this. The program is incredibly fast.

[Thy Grand Voidinesss]

Even accessing its Options is instant [which when compare to FreeCommander's Settings makes it look lightspeed fast]

[defc]

I think one of the keys is this is still in c++.

I actually don't even know if it uses an in-memory db or custom data structures, they must be getting flushed to disk pretty frequently too, plus it has to keep updating them with Windows fs events. I bet the current view stores a list a of affected nodes and then if there is an update, it can check if any of those nodes are impacted so they can be refreshed.

This kind of efficient low level programming by a dedicated dev is not very common these days with everyone having moved on to JS/Python/Web. Another example of an efficient app I use daily is xplorer2 (zabkat.com) which is a must have as a file explorer. I believe there is an integration too.

[ChrisGreaves]

... it always felt like this is too good to be true and its developer must be cheating somehow ...

I agree. I wrote myself a content-search of MSWord documents, and wrote it in Word97/VBA. Blew (back then) Windows File Explorer out of the water.

I think that David, the developer is very bright and is a very good programmer.

But I suspect (and do not know) that the main advantage comes from NOT inheriting core code from the days of MSDOS and 1.2MB 5 1/4" floppy disk ideas. This Article is a starting point, but perhaps someone with a better memory than I can dig out the folklore that Bill Gates once said something like "ordinary people won't understand sub-directories".

I have seen enough corporations that carry a basic, biut stale, philosophy for dozens of years.
I suspect that some core ideas have held true right through the development of Windows, and perhaps insistence on backwards compatibility acts as a hobble on progress.

I suspect that David has not been hindered by old ideas from the 1980s; he has had to deal with those old ideas, as hurdles, but they have not been an obstacle to his progress.

A recent example is the introduction of SSDs. Out the door goes defragmentation. In the door comes even faster content searching (for all applications, not just Everything (and as I write that I am shaken by just how weird that sounds!))

I hope that helps us travel along the magic path ...
Cheers, Chris

[NotNull]

Parts of Everything are written in assembler.

[meteorquake]

Aside from the points mentioned, my impression is that a lot of slowness and bloat in the programming world comes from using 3rd-party libraries and frameworks particularly as regards speed for tasks that are repeated rapidly. In most cases not only is coding such things specifically going to lead to much faster execution, but in many cases it's much quicker to do so than to learn how the library works, and of course more fun. d

[rollsch]

When you see how convoluted and bloated modern frameworks it's easy to see how some asm and c code could be phenomenally fast especially if directly going to the kernel Apis at the lowest possible level.

I'd love to see some of the coding tricks David is most proud of posted here. I'm sure everyone could learn a thing or two.

[void]

Everything 1.3 and earlier had a lot of assembly.
Everything 1.4 is 100% C
Everything 1.5 is 100% C

A lot of the performance improvements in Everything 1.5 come from tight loops and keeping important things in CPU cache.

[Thy Grand Voidinesss]

[...]
core code from the days of MSDOS and 1.2MB 5 1/4" floppy disk ideas. This Article
[...]

How archaic and annoyingly constraining this looks like

I must have totally forgot about such limitations from the times of me being a kid - or never had run into those limits; with the latter one being more feasible as hardware existence determines digital consciousness

[ChrisGreaves]

I must have totally forgot about such limitations ...

limitations? Limitations ?!!???
Good Grief what on earth are you talking about?
In The Good Old Days there were NO folders, just a large leather briefcase that could hold two boxes of punched cards (so about 4,000 cards)
In The Good Old Days, using a column as two six-bit bytes, we could fit 160 bytes on a punched card, so 640,000 bytes on two boxes of cards, so think about ten pounds. A 1.44 MB floppy weighed what? Half an ounce?
"Long filename support" we had, as long as you could hold a 2,000 card deck tight enough to write across the top with a marker pen.
Sigh!
Kids nowadays!
Cheers, Chris

[NotNull]

just a large leather briefcase that could hold two boxes of punched cards

A briefcase. Now that's a good idea. The times someone tripped and dropped the box of punched cards and then had to put the code in the right order again ... Even with markings on the side that took a while.

1.44 MB disks were designed to fit in a chest pocket of a shirt. Problem solved.

[myearwood]

You may not remember the times when dBase III was popular. It was so slow. Along came Fox Software and gave a massive speed boost. Microsoft eventually bought them. The greatest programmers are very rare. Everything is excellent.

[meteorquake]

I do remember the days of the BBC Micro as we had one... every byte and clock tick counted, so things were programmed cleanly and efficiently.

The more we put faster chips and more memory into computers, the more inefficient and wasteful programs become. To say "Hello World" in the past would have taken a few bytes, nowadays a 'standard effort' might take 50 MB or even far more...

Well I've just googled that and a top post is "Why does Hello World Program take 80MB in JavaFX? "

d

[ChrisGreaves]

We should remember too that development of ideas is super-fast today; I mean from Marketing to Sales in 12 months.

That puts pressure on for fast development and short-lifetime of a programming solution.
And THAT suggests using modules of pre-fabricated code to accomplish core tasks. These modules are libraries of procedures written and tested in minimal time to contribute - as building blocks - to the business application.
From this we find that we can develop a computer program in a week, but to do that we need to use less-than-efficient-execution of generalized modules.

I have a library of Word VBA procedures "UW.DOT" with 900+ procedures. Bulky and slow, but I can churn out a complete MSWord VBA application in less than a day.
Cheers, Chris

[voidv]

Well I've just googled that and a top post is "Why does Hello World Program take 80MB in JavaFX? "

d

it remind me of another programming language which the media trying to promote to the public called 'rust'
a hello world program that is a few byte in size when develop in 'C' and it is 4MB in size when develop using 'rust'
why would one line of code 'hello world' need to take up 4MB of your diskspace for the binary.

[Darz]

Everything 1.3 and earlier had a lot of assembly.
Everything 1.4 is 100% C
Everything 1.5 is 100% C

A lot of the performance improvements in Everything 1.5 come from tight loops and keeping important things in CPU cache.

Hello !
Are you planning on publishing a mini technical documentation on how you manage to make searching almost instantaneous?

We're currently working on a global search in software developed in C#. It's already quite powerful, but you have to type the keywords in the right order. For example, if I type "re bin", it finds "recycle bin", but if I type "bin re", it doesn't find it.

If we adapt the regex so that the order doesn't matter, it has too great an impact on performance. Yet in Everything Search, it's almost instantaneous!
We are already keeping things in cache. I can't believe that our performance issue is only due to the overhead of .NET, you must have a particular data structure and search method in this structure that allows you to make this so efficient