"Search as you type" is needlessly clogging the CPU

[msys]

Hello,

I have a problem with the 'interactivity' of the option 'Search as you type'. I have ~6 million files indexed (with sizes and all the other things) so each search takes a second or longer - which is very impressive - but whenever I try to type some word into the search bar, Everything never waits for me to finish typing it. Instead, it tries to search after about 3rd or 4th character (while I'm still typing) and after that, after each next consecutive character. If the word is longer, like 10 characters long, it tends to turn into an annoying slideshow: Everything eats letters from the keyboard buffer - one per about 1s - then searches for the result of letters shown so far...

Is there any option to specify the delay Everything should wait after I stop typing before searching? I've tried to find one in the ini file but to no avail.

If there isn't any, I'd strongly suggest to add one.

Turning off the option 'Search as you type' is even more inconvenient for me, as I am then forced to press 'Enter' to perform each search - which in itself is not convenient. Additionally - if I want to modify the search terms afterwards, I have to move the focus from the results to the search box.

Regards,
Michał

[void]

Setting max_threads may help in this situation.

This will limit Everything to the specified maximum number of threads. If you set this so you have a free thread for the UI and OS it may help UI / system performance.

Please try setting this value to the number of logical CPUs in your system - 1
For example, if you have 8 logical CPUs, set max_threads to 7.

To set max_threads to 7:
In Everything, type in the following search and press ENTER:
/max_threads=7

If this doesn't help, please try setting max_threads to a lower number (eg: logical CPUs / 2).

What type of CPU do you have? it sounds like the UI thread is not getting equal time slices.
I might need to add an option to increase the priority of the UI thread when a search is active..

[msys]

Hi,

Thank you, this helped a little. I have there a i7-4800MQ, so that would be 4 cores. Setting max_threads to 3 didn't help at all but putting in 2 did it.

Obviously, now searching is much slower but at least it is interactive.

For now it'll do.

Adding an option for boosting any thread(s) could be fatal in some situations imho. I consider Everything a background service and this would kind of break that unspoken rule: this way Everything can dominate the whole CPU regardless of what it is doing. I for one would not be happy with that, as I am often - for example - encoding a lot of video streams as they come in in real time and this can potentially break such an encode by putting Everything in front of everything else - no pun intended - for quite a few seconds.

Actually, now that I think of this, Everything already broke some pipelines quite a few times on this machine while starting up. If I remember correctly, the stage when it is 'sorting results' uses up all the CPU and 'kicks out' other time intensive tasks out for almost a minute... so, in this case, the option for lowering the priority of such 'sorting threads' would be more useful than boosting anything.

So, please consider adding the aforementioned option in anyway. That'll allow for searching in full speed while still being properly interactive (and without the need for tweaking threads).

Or alternatively, maybe another option like 'Search xxx milliseconds after you're done typing': without the need for the 'enter' key and without stealing the focus after searching, so that it'll be clear what is happening.

All the best and thanks again,
Michał

[Janus]

@msys

A couple of thoughts for you.

Set hide results when search is empty {Tools=>Options=>Results}, this helps by keeping the initial screen clear.
I have ~8.5M files indexed, and it makes a difference for me.

The second note is one I spent some time running down not just with everything, but other programs as well.
Turn off all the eye candy you can.

For Win7 {Advanced system settings=>system properties=>performance=>visual effects}.
I have no idea where to find it in W8 or later, or if m$ will even let you have permission to change it in them. It isn't like they admit you actually own your computer anymore.

Part of what is happening is that eye candy is giving you problems.
This happens because the display window is linked to a memory structure, the results, from everything.
Every time something changes in it, any change detectable by the display system, the entire everything UI is redrawn from scratch.

The result is that if the cartoon like effects are still being rendered when the next change hits, that render is finished, then another render is immediately called, and they can queue up.
Windows has no way to filter out these redundant or overlapping redraws, so the UI gets laggy, like your desktop suddenly has a high ping.

I use explorer++ in detail mode, and I seldom have any lag issues in it or in the everything UI either one.
My everything(x86) memory usage hovers at about ~675Mb.

Though I would suggest an activity light be added.
If everything is doing an update, then light up the middle of the magnifying lens like a network activity light.

The only time I get large everything lags now is when I extract or remove large projects.
With many projects having 10K+ files, it can get annoying, but everything is still so much better than what M$ calls search, there is no comparison.


Janus.

[msys]

Hi Janus,

Thanks for the advice.

@msys
Set hide results when search is empty {Tools=>Options=>Results}, this helps by keeping the initial screen clear.
I have ~8.5M files indexed, and it makes a difference for me.

I tried that and it does not change things much for me. The list is obviously huge but it gets generated long before I open up the main window (from the tray), so it is just 'ready'.

The second note is one I spent some time running down not just with everything, but other programs as well.
Turn off all the eye candy you can.

For Win7 {Advanced system settings=>system properties=>performance=>visual effects}.
I have no idea where to find it in W8 or later, or if m$ will even let you have permission to change it in them. It isn't like they admit you actually own your computer anymore.

I'm still on Win7 and I intend on staying as long as possible (and then escape to Linux , but unfortunately no tool is ever close to Everything there ) and the funny thing is you've stumbled upon one of my main reasons for that: the removal from Win8 of a so called 'Windows Classic Theme' which you obtain by deselecting the 'visual effects' you mentioned. This mode is so much more productive (screen estate-wise at least) than all the Aero toys etc. but the average Joe wasn't interested in that, so they cut it away...

Part of what is happening is that eye candy is giving you problems.
[...]

You are probably right but what if the algorithm for interaction would be (as I've suggested in the previous post) to not do anything until the user stops changing the search (text) field? Like restart the timer on any event related to that field and only start searching after such timer reaches 1 second or 2 (user-configurable). Wouldn't that solve the whole problem regardless of how many cores there are or how busy or fast the system is overall?

... or am I missing something there?

Regards,
Michał

[therube]

If (part) of your issue is the results displaying as you're typing you're query, you can disable that until you actually hit a <CR>.

Tools | Options | UI -> Search as you type (uncheck)

[void]

Please try Everything 1.4.1.897.

I've added YieldProcessor for every 1024 (512 for x64) file/folder comparison. This should make Everything give up its time slice more often.
However it does come at a cost, from my testing it is very small ~roughly 1ms for every 25 million files (12.5 million files for x64).

To reset max_threads to the default setting:
In Everything, type in the following search and press ENTER:
/max_threads=0

And yes, I do want to avoid changing thread priorities as I've found this makes it worse, either Everything will have too much priority (Everything lags system) or not enough (searches do not complete).

If YieldProcessor helps, I'll add it to sorting and updating the database too.

[Janus]

@msys

You may this funny, but that is part of the reason I will also not be going past Win7.
W8 and later is not windows.
W10 is just a graphics terminal for your microsoft cloud account.
The advertising and usage profiling also offends me in many contexts, not the least of which is that I actually own my computer.
The security concerns of having your usage profiled and reported via mechanism you do not control is the sort of thing repressive regimes do, not legitimate businesses that care about the customer.
It is obvious that we are a product to M$ now, not its customers.

I run Win7 using "classic shell" using the classic theme with all of the eye candy disabled.
Explorer++, 7+ taskbar tweaker, void tools everything, network activity indicator, and some others.
I do like the ability to reorder tings on the taskbar, and that they NEVER GROUP OR COMBINE!!!
I even got my quick start back, though it is a folder on the desktop now, but it still looks and works the same beside the start button.
And I can drag stuff to the start button to add it to the start menu where I want it.

The scatter shot approach of splattering related settings like tornado debris offends me.
2K/XP desktop just worked, they should have just left it alone.
I liked a single context menu on the desktop taking me to everything having to do with the desktop.
Now it is all scattered and incoherent, and what they did to network connections is just awful.

No version of Linux or any of its desktops are really any better than the garbage M$ is putting out now.
They also scatter things at random, and have no real gui anything, just gui front ends for the cmd line tools.

When ReactOS goes beta, I will be switching to it.


@void
In the mean time, one thing that may help Everything is a ~500ms pause after a key stroke to begin search.
That way no search happens while you are still typing.
Yet the moment you stop, the results are just there, minus the flicker.
Just a thought.


Janus.

[msys]

Please try Everything 1.4.1.897.
[...]

Unfortunately, with v897 the behavior is exactly the same. The slideshow, etc.

Are you somehow against the idea I'm suggesting (among Janus, apparently - see the quote below)?

@void
In the mean time, one thing that may help Everything is a ~500ms pause after a key stroke to begin search.
That way no search happens while you are still typing.
Yet the moment you stop, the results are just there, minus the flicker.
Just a thought.

Yup, that's exactly what I've suggested since starting this topic, with the addition of being able to configure the length of this pause.

[...]
When ReactOS goes beta, I will be switching to it.

I agree with most of what you've said there but are you saying the UI in ReactOS got any better? I'd have to check but frankly that's hard to believe. Also, switching to ReactOS will not be that easy - even when it will be stable (and that's a big when ), there are MS lawyers that are already putting clauses into runtime libraries' licences that are forbidding you from running them under anything not produced by MS (this includes Wine and ReactOS). Without runtime libraries nothing will work obviously, and rewriting these from scratch will take forever. So using ReactOS properly will be either illegal or impossible

If (part) of your issue is the results displaying as you're typing you're query, you can disable that until you actually hit a <CR>.
Tools | Options | UI -> Search as you type (uncheck)

I know about that option, I've even mentioned it in my first post 'just in case', see the following quote from it:

Turning off the option 'Search as you type' is even more inconvenient for me, as I am then forced to press 'Enter' to perform each search - which in itself is not convenient. Additionally - if I want to modify the search terms afterwards, I have to move the focus from the results to the search box.

Regards,
Michał

[Janus]

@msys

Not a lawyer, but from years of study, and trying to state things simply.
Part of what has kept the ReactOS project going so slow is blackbox reproduction of core windows functionality.
M$ already tried poisoning the tree, and failed.
On the reuse and API fronts both.

The clauses relating to purchased rather than leased/cloud usage fall under the first sale doctrine.
You can also look up the shrink-wrap license which has been rebuked in court multiple times.
They lost this fight, in its multitude of forms, many years ago.
This is part of the reason so many corporate software houses are forcing users onto cloud usage.

A sale implies an exchange of goods, which is a transfer of ownership, and control.
Once ownership changes hands, usage of the goods is beyond the producers purview and responsibility both.
First sale creates a safe harbor for the producer, by taking it out of their sphere of control.

This is the reason behind the strong community/social presence of leased software.
The monitoring such as done in W10, or of community sites of leased software, is done for corporate legal safety.
While they maintain control of their software, they have also lost safeharbor.
So deliberate misuse of their software can theoretically blow back on them, though the odds are low.
Yet as long they actively monitor and report, they get best effort protection which is worth a lot.

This has to do with ReactOS in a direct way.
The API for windows is open, and it is being reproduced.
The dotnet libraries that underpin VS, have also been open sourced in order to undermine the MONO project, which M$ tried to shutdown, but were instead shot down in court.
The redist support libraries of VS can not be restricted on the purchased version, and so long as the exact same files are used on the community version, they can not be restricted either.
While the community/leased version of VS can restricted to an M$ produced OS.
The purchased or standalone version cannot be so restricted.

So the core redist libraries are safe.
Then there is the DB stuff.
Which admittedly does carry better protections, but protections which are untested in court.
I have never had anyone ask to convert to sql server, but I have converted a lot of stuff from it to mysql.
ODBC is a programming mess since it denies low level access to the data.
No one uses it if they have a choice, linux does a better job if the DB is large enough to put on a lan, or server, or online.
Which is part of why I am looking forward to ReactOS, since those linux solutions can then be used on a windows based platform.
I have used xampp on windows, it far more useful than the M$ equivalent, and multiplatform.

The uncertainty of who owns the OS and the computer itself are part of what is driving PC sales down.
Not the whole thing of course, but from the corporate side.
If you do not own the OS, then how can you guarantee what that OS does?
If it leaks performance data, what else does it leak?
Can the OS owner you are leasing from control what you do with the hardware you own?
Or do you in fact actually own the functionality of the hardware?
For this, look at non oem printer refills and the whole coffee cartridge thing.
You bought the printer or coffee maker, yet the OEM tried to control what you could do with it past fist sale.
They failed, as they should have, and gloriously so.
Yet had those printers or coffee makers been leased like so much software is today, the cases would not have even made it to court.
A cloud account is a lease, it is NOT! a sale, just a right of usage.
Like renting a car, you get to use it, but they set the rules.

ReactOS bypasses all of these concerns because they have already done the hard part as they have gone.
One of the main strengths of Linux is the OS itself is auditable.
You can check the binary and source of every single file if you like.
People who take security seriously like that sort of thing.

Once ReactOS goes beta, M$'s version of windows is dead.
With dotnet going open source to undercut MONO, the entire VS compiled eco system is safe.
MONO was implementing dotnet for linux, for mac, and back into windows, which M$ undercut by opensourcing after legal efforts to stop them failed.
The only thing I am hoping for is a useful/usable replacement for office, which has become as much garbage as newer windows.

As for ReactOS itself.
I have managed to get Celestia and Explorer++ both running in it.
My own private forks of both, on my own copy of tweaked copy ReactOS, but I got them to work, more or less.

I know of no reason that Everything won't run on ReactOS just like it does on windows.
I look forward to finding out once the NTFS drivers are more stable.


Janus.

[void]

Unfortunately, with v897 the behavior is exactly the same. The slideshow, etc.

I'm surprised 897 didn't help.
Did you exit Everything completely before running 897? please check 897 is shown in Help -> About Everything.

Please try Everything 898, I've increased the yielding frequency to once every file/folder comparison.

Are you somehow against the idea I'm suggesting (among Janus, apparently - see the quote below)?

It's a good idea, I have it on my TODO list for Everything 1.5.

[Janus]

@void

I feel sorry for the table holding the print out of your to do list for 1.5 , have you worn your printer out yet?


Janus.

[msys]

I'm surprised 897 didn't help.
Did you exit Everything completely before running 897? please check 897 is shown in Help -> About Everything.

Please try Everything 898, I've increased the yielding frequency to once every file/folder comparison.
[...]

I've double checked and yes, I am using the correct version.

There was no noticeable difference with v897, but with v898 there is one... for the worse. Now the slideshow is about 2x - 3x slower . Still, setting /max_threads=2 helps with this, like in v895 I've used before that.

@Janus
Thanks for the explanation, this is mighty interesting (well, it should be for everyone valuing their privacy, there must be some long-term way out of this madness). Could you be so kind to elaborate on where exactly (in which countries) has MS lost the legal cases you mentioned? Links would be enough for me if you prefer. Groklaw is no more, so it is not that easy to find. Much obliged

Regards,
Michał

[NotNull]

@msys
I run Win7 using "classic shell" using the classic theme with all of the eye candy disabled.

Contrary of what you expect, turning ON Aero might even improve your performance.
That's because in that case large portions of the screen redrawing, etc can be off-loaded to the GPU.
But it will depend on your hardware. You could satrt by trying Aero (again) with a lot of the eye candy disabled.

[msys]

Contrary of what you expect, turning ON Aero might even improve your performance.
[...]

I am fully aware of that. Still, it's usability/productivity I'm after first and performance later, as the problems with the latter can always be mitigated by upgrading the hardware, while problems with the former are not so easily solved...

The problems with 'visual styles' (or however these are marketed nowadays) are apparent a few seconds after turning these on: there is less space on the taskbar for... well, tasks (windows/shortcuts) and each window/shortcut is itself bigger. I've found no way to shrink these to the sizes of the classic mode. So I'm limited in the number of opened windows/programs before the taskbar overflows. This is a big deal for me, as I qualify as at least a 'power user' (running about 40-50 programs constantly across multiple monitors). Also, I'm losing this precious screen estate in exchange for having everything more distracting/colorful and neither pretty nor more informative (at least in my opinion).

tldr; even if enabling Aero would somehow improve the performance, it will seriously hinder just about everything else I care about.

Regards,
Michał

[Janus]

Based solely on my own experience.

Speeding up the hardware and software does nothing for you, if it clogs the wetware.

Snazzy borders are for people who can not naturally multitask.
The gaudy borders, fuzzy edges, blank real estate etc, create visual separations that provide wide area visual context cues.
While they are a large help to people who lack multitasking skills.
They are nothing but a headache and or nausea inducing annoyance and hindrance to those of us who have them.

I can appreciate the thinking that goes into the design of modern design of UIs.
However, I also feel like the lack of choice about using that eye candy serves to drag those who can do more, down to the level of those who can not.
If the eye candy works for you, then you should use it.
If it does not, then you should not be forced to use it.
There is no one size fits all solution.
We are individuals, so we should all be free to use what works for us.

Of course I am also of the opinion that the simpler solution is most often the correct one.
I am also aware that I am in the minority in today's world.


Janus.

[void]

Thanks for trying Everything 898 and your feedback.

Could you please try Everything 899.

I've removed YieldProcessor and added Sleep(0) to every 1024 comparisons.
This is a little more extreme and am hoping to see some improvement in UI responsiveness.

Please make sure you reset max_threads to the default of 0 for the best performance:
In Everything, type in the following search and press ENTER:
/max_threads=0

I suspect yielding and sleep will have no impact as the issue lies with page faults, paging or memory performance.
Could you please check Page Faults before and after performing a search:

• Open the Task Manager (Ctrl + Shift + ESC)
• Click the Processes tab (Details tab on Windows 8+)
• Select the Everything process
• From the View menu, click Select Columns.
• Check Page Faults.
• Click OK.
• Take note of the number of page faults.
• Type in the following search slowly, so each character is registered:
  details
• What is the number of pages faults before and after the search?

How much RAM do you have? how much RAM is Everything using?

[msys]

Thanks for trying Everything 898 and your feedback.
Could you please try Everything 899.
[...]

I've tried v899 and what do you know, the problem is gone! ... even when max_threads=0.

So, sleep(0) somehow helped.

Thank you!

For the sake of completeness, I'm answering all of your questions anyway:

• After a fresh start of Everything and loading its database, Page Faults are at about 220 000 and the searches have minimal to no impact on its value - it very slowly rises by itself (there is always a lot going on on the system and Everything might just be busy appending current changes to the database).
• I have 32GB of RAM installed and usable on this system.
• Everything is currently using about 820MB (Private Working Set).

Regards,
Michał

[msys]

One other thing - Everything v899 still spikes the CPU usage to 100% while I'm typing some long text in the search field (while it should wait until after I've stopped typing for that), but now this field is fully interactive, so it is a big improvement anyway.

All the best,
Michał

[void]

Thanks for testing 899.

Looks like the lack of thread yielding was the issue.
The page faults are normal.

Everything 899 will still use 100% CPU.
Sleep 0 simply yields the thread every 1024 comparisons to give other threads with the same priority a chance to use the CPU.

YieldProcessor should be the same as Sleep 0 for most intel CPUs. However in this case it doesn't appear to work as intended.

There is a cost with Sleep 0, it is roughly 1ms for every 7million files (1ms for every 3.5 million with Everything x64).
I'll be testing Sleep 0 and most likely keeping it in all future versions of Everything.

I've also implemented an option to set a search delay and am currently testing it for Everything 1.5.