[Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
-
- Posts: 41
- Joined: Thu Sep 27, 2018 4:46 pm
[Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
It consumed all 64GB RAM.
Crashed at 128GB with an error I've not seen before: .\src\mem.c(1124): mem_alloc(): Fatal error: out of memory 137438953472
(137438953472 = 128Gibibyte).
& somehow Virtual Memory Committed managed to reach 237,094MB. I was refactoring my bookmarks & macros & probably had 6-7 different search windows open plus the bookmarks, setting, & filters windows. It was definitely being abused.
I've had, I believe, the same crash happen enough times on my old PC I've learned long ago to close & reopen Everything every 30min-ish whenever creating/experimenting with macros, functions, bookmarks, & filters. The time elapsed doesn't actually matter. It can happen anytime. But changes
to bookmarks, filters, settings, etc. don't save until everything is closed & it sucks to lose a half hour of whatever I was changing.
On this new PC I neglected to force my changes to save bcus with substantially more power & ram, I thought maybe it wouldn't happen.
I was wrong.
The error on the new PC is a little different but seems the same I've always encountered - just taken to the extreme bcus of the additional resources. On the old PC there was 32GB RAM & 32GB set pagefile limit. I'd know when Everything was getting close to crashing bcus the whole PC would slow down. The new PC didn't really slow down. I did notice Explorer was slow to render & some of my chrome tabs refreshed, but I didn't think anything of it because it was still responsive.
I don't consider this too big of an issue since I'm abusing it It also only happens once in a blue moon (like when I'm abusing it).
If you're interested & have time I can supply the 1MB kernel.dmp, 10MB mini.dmp, & possibly the 2GB compressed full dump (123GB when extracted).
-------------
EDIT: Usually it would just crash and generate an event in the EventViewer & not save any of my changes. This was the first time it was a graceful shutdown with an error dialog. I clicked OK shortly after and it managed to still save the various csv files & database which was a nice surprise. No event was generated for the EventViewer.
Crashed at 128GB with an error I've not seen before: .\src\mem.c(1124): mem_alloc(): Fatal error: out of memory 137438953472
(137438953472 = 128Gibibyte).
& somehow Virtual Memory Committed managed to reach 237,094MB. I was refactoring my bookmarks & macros & probably had 6-7 different search windows open plus the bookmarks, setting, & filters windows. It was definitely being abused.
I've had, I believe, the same crash happen enough times on my old PC I've learned long ago to close & reopen Everything every 30min-ish whenever creating/experimenting with macros, functions, bookmarks, & filters. The time elapsed doesn't actually matter. It can happen anytime. But changes
to bookmarks, filters, settings, etc. don't save until everything is closed & it sucks to lose a half hour of whatever I was changing.
On this new PC I neglected to force my changes to save bcus with substantially more power & ram, I thought maybe it wouldn't happen.
I was wrong.
The error on the new PC is a little different but seems the same I've always encountered - just taken to the extreme bcus of the additional resources. On the old PC there was 32GB RAM & 32GB set pagefile limit. I'd know when Everything was getting close to crashing bcus the whole PC would slow down. The new PC didn't really slow down. I did notice Explorer was slow to render & some of my chrome tabs refreshed, but I didn't think anything of it because it was still responsive.
I don't consider this too big of an issue since I'm abusing it It also only happens once in a blue moon (like when I'm abusing it).
If you're interested & have time I can supply the 1MB kernel.dmp, 10MB mini.dmp, & possibly the 2GB compressed full dump (123GB when extracted).
-------------
EDIT: Usually it would just crash and generate an event in the EventViewer & not save any of my changes. This was the first time it was a graceful shutdown with an error dialog. I clicked OK shortly after and it managed to still save the various csv files & database which was a nice surprise. No event was generated for the EventViewer.
Last edited by DerekZiemba on Mon Aug 28, 2023 2:40 pm, edited 1 time in total.
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Thank you for the issue report DerekZiemba,
Please send the mini crash dump to support@voidtools.com
Privacy
Please send the mini crash dump to support@voidtools.com
Privacy
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
(You mean the 120GB version he already has isn't good enough .)
Now, as far as "Modules", does that imply that those various items have injected themselves into the Everything process space?
Revo, Icaros, Power Toys, Dropbox, MSWebp, Onedrive, Onenote...?
So the statics always load, regardless, & the dynamics only load if called?
I'll note that I have something called "IE" in mine. Gross.
. (Win7.)
Now, as far as "Modules", does that imply that those various items have injected themselves into the Everything process space?
Revo, Icaros, Power Toys, Dropbox, MSWebp, Onedrive, Onenote...?
So the statics always load, regardless, & the dynamics only load if called?
I'll note that I have something called "IE" in mine. Gross.
. (Win7.)
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Most likely shell extensions. And if I *had* to guess: some of them don't "play nice" and leak memory or write to the wrong places in memory.
(Shell extensions can be disabled inside Everything, but the dumps might reveal which one is/are the culprit )
-
- Posts: 41
- Joined: Thu Sep 27, 2018 4:46 pm
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
> (Shell extensions can be disabled inside Everything, but the dumps might reveal which one is/are the culprit )
I emailed the mini & kernel dumps to support@voidtools.com shortly after void requested.
> Now, as far as "Modules", does that imply that those various items have injected themselves into the Everything process space?
Yeah I included that in the screenshot for that reason. However I had to "Hide System" & "Hide knowndll images" or the list goes on forever. I only now just realized System Informer has a way to export the list. So attached is the exported lists of what's currently loaded, unfortunately when it crashed I didn't know this was an option.
----------------
Well shoot. Can't attach .csv or .md files & can't put it in the post bcus "Your message contains 285495 characters. The maximum number of allowed characters is 60000."
Guess I'll put it on github
(Note: If you scroll up to the raw formatted version of the file, it visually shows which module loaded which in a tree hierarchy)
I emailed the mini & kernel dumps to support@voidtools.com shortly after void requested.
> Now, as far as "Modules", does that imply that those various items have injected themselves into the Everything process space?
Yeah I included that in the screenshot for that reason. However I had to "Hide System" & "Hide knowndll images" or the list goes on forever. I only now just realized System Informer has a way to export the list. So attached is the exported lists of what's currently loaded, unfortunately when it crashed I didn't know this was an option.
----------------
Well shoot. Can't attach .csv or .md files & can't put it in the post bcus "Your message contains 285495 characters. The maximum number of allowed characters is 60000."
Guess I'll put it on github
(Note: If you scroll up to the raw formatted version of the file, it visually shows which module loaded which in a tree hierarchy)
Last edited by void on Fri Aug 25, 2023 2:10 am, edited 1 time in total.
Reason: removed links
Reason: removed links
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Thank you for the mini crash dump DerekZiemba,
I am still looking into the issue.
A buffer is corrupted.
The crash occurred when building the search from your filter macro while inside a list editor in the options window.
Could you please send your Filters-1.5.csv from %APPDATA%\Everything to support@voidtools.com
I am still looking into the issue.
A buffer is corrupted.
The crash occurred when building the search from your filter macro while inside a list editor in the options window.
Could you please send your Filters-1.5.csv from %APPDATA%\Everything to support@voidtools.com
-
- Posts: 41
- Joined: Thu Sep 27, 2018 4:46 pm
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Oh wow, I only just realized my changes were saved! It appears like everything's there, right up to the crash, even the "Search History-1.5a.csv"
So I've attached all of it to that gist.
--------------------
> The crash occurred when building the search from your filter macro while inside a list editor in the options window.
I was primarily working with the Bookmarks feature so I'm a bit surprised it was the filters that caused it. The backup of `%APPDATA%\Everything` I have available on the new PC is from before I discovered bookmarks, so I was trying to get away from that & instead use bookmark macros.
Back then I was sticking a bunch of custom defined functions in the "_" filter, then referring to the `_:` macro in every other filter - as you'll see.
The Bookmarks are very much a work in progress. Trying to figure out what I can & cannot do. BTW, is there a way to do the `<ancestor:folder:attributes:HDI> "ungoogled*\"` on lines 49 to 66 of `search-history-1-5a-csv` type thing I was attempting? The goal is to filter all child files of an ancestor directory if any parent directory is marked hidden & not indexed, even if the child file has normal attributes.
So I've attached all of it to that gist.
--------------------
> The crash occurred when building the search from your filter macro while inside a list editor in the options window.
I was primarily working with the Bookmarks feature so I'm a bit surprised it was the filters that caused it. The backup of `%APPDATA%\Everything` I have available on the new PC is from before I discovered bookmarks, so I was trying to get away from that & instead use bookmark macros.
Back then I was sticking a bunch of custom defined functions in the "_" filter, then referring to the `_:` macro in every other filter - as you'll see.
The Bookmarks are very much a work in progress. Trying to figure out what I can & cannot do. BTW, is there a way to do the `<ancestor:folder:attributes:HDI> "ungoogled*\"` on lines 49 to 66 of `search-history-1-5a-csv` type thing I was attempting? The goal is to filter all child files of an ancestor directory if any parent directory is marked hidden & not indexed, even if the child file has normal attributes.
Last edited by void on Fri Aug 25, 2023 2:09 am, edited 1 time in total.
Reason: removed links
Reason: removed links
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Thank you for the filters and bookmarks.
Bookmark macros eat all characters until a white space.
There's a few filter/bookmark macro references that explode because the parameters eaten are massive.
I'm working on improving the syntax and enforcing a hard limit of 8MB for macro expansion.
For now, please go through all your filters/bookmarks and check the search.
Please make sure all references to a nested filter/bookmark macro have a trailing space.
For example:
Name: EXEbin:<X>
Search: EXE_: <<X:>|<ignore-whitespace:Path:X:>|<appversion:X:>|<appname:X:>|<ignore-whitespace:ignore-punc:Company:X:>>
should be:
Search: EXE_: << X: >|< ignore-whitespace:Path:X: >|< appversion:X: >|< appname:X: >|< ignore-whitespace:ignore-punc:Company:X: >>
Search for:
folder:attributes:HDI "ungoogled*\"
Select all your folders and copy all the filenames to the clipboard (Ctrl + Shift + C)
Change the search to ancestorfilelist1:
Hold down Ctrl and click the ancestorfilelist1: text in the search box.
Paste your filenames (Ctrl + V)
Click OK.
Bookmark macros eat all characters until a white space.
There's a few filter/bookmark macro references that explode because the parameters eaten are massive.
I'm working on improving the syntax and enforcing a hard limit of 8MB for macro expansion.
For now, please go through all your filters/bookmarks and check the search.
Please make sure all references to a nested filter/bookmark macro have a trailing space.
For example:
Name: EXEbin:<X>
Search: EXE_: <<X:>|<ignore-whitespace:Path:X:>|<appversion:X:>|<appname:X:>|<ignore-whitespace:ignore-punc:Company:X:>>
should be:
Search: EXE_: << X: >|< ignore-whitespace:Path:X: >|< appversion:X: >|< appname:X: >|< ignore-whitespace:ignore-punc:Company:X: >>
This can be done with File List Slots.The Bookmarks are very much a work in progress. Trying to figure out what I can & cannot do. BTW, is there a way to do the `<ancestor:folder:attributes:HDI> "ungoogled*\"`
Search for:
folder:attributes:HDI "ungoogled*\"
Select all your folders and copy all the filenames to the clipboard (Ctrl + Shift + C)
Change the search to ancestorfilelist1:
Hold down Ctrl and click the ancestorfilelist1: text in the search box.
Paste your filenames (Ctrl + V)
Click OK.
-
- Posts: 41
- Joined: Thu Sep 27, 2018 4:46 pm
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
those bookmarks are expanding to something like an 8mb query??? That blows my mind. It's magic that it can still be so fast.
--------
So that ungoogled thing was just an example. There's all kinds of other similar folders like it. In that case its an ungoogled portable chromium build & the folder just sits in my userprofile along with a bunch of other similar things such as other portable apps. Chrome in particular generates all kinds of files i don't care about.
Instead of manually targeting each folder like that everytime i do a search or by manually creating what i now imagine is a huge filter (that would break anytime a new similar thing is introduced), i thought maybe i could create a filter or bookmark/macro that'd automatically exclude anything like it. Excluding new things would be a simple as right clicking a parent folder i don't care much about, then In the windows property dialog setting hidden & not content indexed. The Everything filter when selected would then be able to not show that item and any of its descendants. I wouldn't need to edit an existing filter or add a new one.
Currently the only way i know to do that is when building the index by checking the "don't index hidden" or "don't index system files" checkboxs. Normally i don't want to see those types of files as it's all just noise. But occasionally i need to get at them (system files, hidden files, or other temp like, cached, resource, or generated files) & I'm not about to rebuild the index to do so. I just want my normal "Home" filter to exclude that type of stuff or any of the backed up stuff. Makes getting at them simply switching the filter or using a keybind.
---------
I'm pretty much trying to use the search syntax like a scripting language & am abusing the features i know about to try and get there. Feels like I'm jamming the cylinder into the triangle hole of that kids game & the only tool i know i have is a hammer.
Are there any plans to introduce maybe a scripting language? Lua perhaps?
---------
Curious why you removed the gist links. I didn't see anything in them that i feel violates my privacy. But in any case, I'll delete the gist, just in case.
--------
So that ungoogled thing was just an example. There's all kinds of other similar folders like it. In that case its an ungoogled portable chromium build & the folder just sits in my userprofile along with a bunch of other similar things such as other portable apps. Chrome in particular generates all kinds of files i don't care about.
Instead of manually targeting each folder like that everytime i do a search or by manually creating what i now imagine is a huge filter (that would break anytime a new similar thing is introduced), i thought maybe i could create a filter or bookmark/macro that'd automatically exclude anything like it. Excluding new things would be a simple as right clicking a parent folder i don't care much about, then In the windows property dialog setting hidden & not content indexed. The Everything filter when selected would then be able to not show that item and any of its descendants. I wouldn't need to edit an existing filter or add a new one.
Currently the only way i know to do that is when building the index by checking the "don't index hidden" or "don't index system files" checkboxs. Normally i don't want to see those types of files as it's all just noise. But occasionally i need to get at them (system files, hidden files, or other temp like, cached, resource, or generated files) & I'm not about to rebuild the index to do so. I just want my normal "Home" filter to exclude that type of stuff or any of the backed up stuff. Makes getting at them simply switching the filter or using a keybind.
---------
I'm pretty much trying to use the search syntax like a scripting language & am abusing the features i know about to try and get there. Feels like I'm jamming the cylinder into the triangle hole of that kids game & the only tool i know i have is a hammer.
Are there any plans to introduce maybe a scripting language? Lua perhaps?
---------
Curious why you removed the gist links. I didn't see anything in them that i feel violates my privacy. But in any case, I'll delete the gist, just in case.
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
According to void:DerekZiemba wrote: ↑Fri Aug 25, 2023 12:55 pm Curious why you removed the gist links. I didn't see anything in them that i feel violates my privacy.
A full dump would contain all your filenames. (hundreds of MB)
A mini crash dump will only contain a small snapshot which includes the stack. (<10 MB)
The stack may contain a couple filenames from your system that Everything may currently be processing.
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
I will consider an ancestorattribute: search function.
(this function would require attribute indexing -kindof like childattributes: )
Thank you for the suggestion.
Lua support is on my TODO list.
Currently bookmark macros are very basic and not really designed for nested calls.
voidtools does not collect any user data.
User data such as Search History and Everything settings are removed from the voidtools forums.
This information could be used to uniquely identify you. (Volume GUIDs are unique)
(this function would require attribute indexing -kindof like childattributes: )
Thank you for the suggestion.
Lua support is on my TODO list.
Currently bookmark macros are very basic and not really designed for nested calls.
voidtools does not collect any user data.
User data such as Search History and Everything settings are removed from the voidtools forums.
This information could be used to uniquely identify you. (Volume GUIDs are unique)
-
- Posts: 41
- Joined: Thu Sep 27, 2018 4:46 pm
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
I've triggered it again... This time I have noticed my PC slow down significantly.
The standard windows task manager can't deal with this apparently - it's froze solid.
I imagine the leak must be all zero's or something because Physical Memory isn't really affected. In fact SoftwareInformer say's it's only using pretty much normal amounts as far as Private bytes & WorkingSet go. It is however showing 135GB Private Bytes allocated but doesn't appear to know which process to attribute it to.
Below are the the new bookmarks I was creating. I was working on the "ANNOYING" one when this occurred. It happened right after I put in the `#{: expression #}:` grouping brackets because I couldn't make heads or tails of the `<>`'s anymore. Now that I think of it, I think that type of grouping is what got me last time too.
--------------
While typing this, before I could get a minidump, I just lost SystemInformer. Oddly Everything seems to still be working fine...
Luckily ProcessExplorer successfully launched & I was able to grab another mini dump. I've attached it as "Everything64.zip. ------------
EDIT:
And now I lost vscode too. Odd error. Says too many files open??? Might be relevant to something Everything has done. Everything seems to be working fine tho. I just exited it & confirmed in ProcessExplorer that it had exited, but it has not freed the RAM.
The standard windows task manager can't deal with this apparently - it's froze solid.
I imagine the leak must be all zero's or something because Physical Memory isn't really affected. In fact SoftwareInformer say's it's only using pretty much normal amounts as far as Private bytes & WorkingSet go. It is however showing 135GB Private Bytes allocated but doesn't appear to know which process to attribute it to.
Below are the the new bookmarks I was creating. I was working on the "ANNOYING" one when this occurred. It happened right after I put in the `#{: expression #}:` grouping brackets because I couldn't make heads or tails of the `<>`'s anymore. Now that I think of it, I think that type of grouping is what got me last time too.
"HOME:" , 0, "", , , , , , , , , , "WINDOWS: !SYS: !HIDDEN: !ANNOYING:" , , "", "", 0, , , , , 0, "HOME" , , ""
"WINDOWS:", 0, "", , , , , , , , , , " !< LINUX: >" , , "", "", 0, , , , , 0, "WINDOWS" , , ""
"LINUX:" , 0, "", , , , , , , , , , "<""Linux:\""|""C:\WSA""|""C:\**\wsl\""> " , , "", "", 0, , , , , 0, "LINUX" , , ""
"SYS:" , 0, "", , , , , , , , , , "< attrib:<S|HI|HSA|RHS|RHA|RHD|HDA|HDL> >|< owner:<""SYSTEM""|""TrustedInstaller""|""SERVICE""> >|< ""C:\ProgramData\Package Cache\"" >|< ""C:\Windows\"" ext:evtx >" , , "", "", 0, , , , , 0, "SYS" , , ""
"HIDDEN" , 0, "", , , , , , , , , , "< attrib:H >|< regex:path:""(?-i)\\\.[a-z]+\\"" >" , , "", "", 0, , , , , 0, "HIDDEN" , , ""
"ANNOYING", 0, "", , , , , , , , , , "!<""node_modules\""|""packages\""|""\cache\""|#{: t<""""|""e"">mp\**cache\ #}:|#{: \<""""|c|ca|app|html|dx|d3ds|dawn|font|gpu|""npm-""|""NV_""|offline|package|startup|web>cache<""""|2|d|s|data>\ #}:|""Users\*\AppData\**\User Data\""|""*google*chrom*\User Data\"" >", , "", "", 0, , , , , 0, "ANNOYING", , ""
While typing this, before I could get a minidump, I just lost SystemInformer. Oddly Everything seems to still be working fine...
Luckily ProcessExplorer successfully launched & I was able to grab another mini dump. I've attached it as "Everything64.zip. ------------
EDIT:
And now I lost vscode too. Odd error. Says too many files open??? Might be relevant to something Everything has done. Everything seems to be working fine tho. I just exited it & confirmed in ProcessExplorer that it had exited, but it has not freed the RAM.
Last edited by void on Sat Sep 02, 2023 10:14 am, edited 1 time in total.
Reason: removed dmp
Reason: removed dmp
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Thank you for the dmp DerekZiemba,
The dmp was just Everything running normally.
Everything 1.5.0.1356a makes a lot of changes here:
The preprocessor has been simplified.
This will likely break things...
Everything will now do two passes on the search.
The first pass will expand any #[function-name: ... #]: functions.
The search is then broken into search terms.
The second pass will expand any [function-name:...] functions in each search term.
Bookmark and filter searches will apply the same process.
Any [function-name:...] functions inside parameters passed to macros are now expanded and treated as quoted text.
There is now a 8MB limit on macro expansion.
To set the expansion limit:
Everything 1.5.0.1356a fixes an issue with complex subexpressions.
You might have been seeing hangs caused by this issue.
Previous versions would hang on searches like: name:<<abc 123> | <foo bar> | <baz qux>>
Highlighting performance is also improved with complex subexpressions.
The NOT operator (!) will now work inside subexpressions. for example: name:<!<<abc 123> | <foo bar> | <baz qux>>>
I highly recommend using spaces around < > and | in your filter/bookmark searches as future versions will likely eat the < > and |
For example:
case:< name:"abc 123" ext:mp3;jpg >
case:< ext:mp3;jpg name:$param: >
Please let me know if you see any hangs with this version.
Please let me know if any of your bookmark/filter macros are no longer working.
The dmp was just Everything running normally.
Everything 1.5.0.1356a makes a lot of changes here:
The preprocessor has been simplified.
This will likely break things...
Everything will now do two passes on the search.
The first pass will expand any #[function-name: ... #]: functions.
The search is then broken into search terms.
The second pass will expand any [function-name:...] functions in each search term.
Bookmark and filter searches will apply the same process.
Any [function-name:...] functions inside parameters passed to macros are now expanded and treated as quoted text.
There is now a 8MB limit on macro expansion.
To set the expansion limit:
- In Everything 1.5, from the Tools menu, click Options.
- Click the Advanced tab on the left.
- To the right of Show settings containing, search for:
max - Select search_max_expand_size.
- Set the value to: 8388608
(where 8388608 is the desired limit in bytes) - Click OK.
Everything 1.5.0.1356a fixes an issue with complex subexpressions.
You might have been seeing hangs caused by this issue.
Previous versions would hang on searches like: name:<<abc 123> | <foo bar> | <baz qux>>
Highlighting performance is also improved with complex subexpressions.
The NOT operator (!) will now work inside subexpressions. for example: name:<!<<abc 123> | <foo bar> | <baz qux>>>
I highly recommend using spaces around < > and | in your filter/bookmark searches as future versions will likely eat the < > and |
For example:
case:< name:"abc 123" ext:mp3;jpg >
case:< ext:mp3;jpg name:$param: >
Please let me know if you see any hangs with this version.
Please let me know if any of your bookmark/filter macros are no longer working.
-
- Posts: 41
- Joined: Thu Sep 27, 2018 4:46 pm
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Thank you for Everything. I'll give that a try.
Hopefully it doesn't break too bad... Will it give me some kind of indication if I've exceeded the 8MB macro expansion limit? I may try setting it lower to identify any potential problematic macros. Just blows my mind they may be expanding that much.
------------------
Anyway, I threw a $20 your way. Always impressed by your support, work, and commitment.
One day, could syntax highlighting be implemented (also multiline queries... a man can dream) ? Ideally to color matching braces & hint at invalid syntax. Also to potentially identify what's a:
- function
- preprocessor function
- search modifier
- property
- filter macro
- bookmark macro
- some other user defined macro/function (are there any others?)
- string (or what's being treated as a string)
- number
- operator (as in size:>10b) so the `>` would be colorized as an operator to avoid confusion with brace matching
------------------
EDIT:
EDIT 2:
I went through & reread most of the docs I referenced above. It makes a little more sense after reading the additional comments below the initial posts. But I still have some kind of mental block that's preventing me from entirely grasping std search FNs(especially vs std properties/column names), PreProcessor FNs, modifiers, user defined FNs, MACROs, & how to and when to tie them all together. What makes sense now is PreProcessorFNs return a value & SearchFNs filter results.
What I struggled with was piping output from a PreProcessorFN that contained nested SearchFNs or Properties to another SearchFN & putting all that together in a MACRO in a way that is valid syntax, then using that MACRO from the search box. But now I think I'm realizing that's not possible bcus SearchFNs are for searching and always affect the result list. I guess I thought I could pass the results of a SearchFN to a PreProcessorFN (basically behind the scenes would built an intermediate result list) that returned something I could then pass to a SearchFN to filter the visible result list. Part of the confusion stems from columnnames, properties, & SearchFNs resembling & basically being referenced/called the same way.
I also found the dollar `$` `$param` syntax is used in Column Formulas. But still, how would you pass a parameter to a column formula? Is that somehow possible?
Hopefully it doesn't break too bad... Will it give me some kind of indication if I've exceeded the 8MB macro expansion limit? I may try setting it lower to identify any potential problematic macros. Just blows my mind they may be expanding that much.
Hmm. I always thought it did. Guess that explains why some things haven't worked as expected.The NOT operator (!) will now work inside subexpressions. for example: name:<!<<abc 123> | <foo bar> | <baz qux>>>
Already reworked all my macros (Filters & Bookmarks) after you last mentioned this. Some things that have always stumped me as to why it wasn't working, now work!I highly recommend using spaces around < > and | in your filter/bookmark searches as future versions will likely eat the < > and |
I'm actually not sure if I use that form very much. But I'm also confused, and have always been, over what I'm actually using or doing. In most cases I'm not sure if what I'm using:The first pass will expand any #[function-name: ... #]: functions.
The search is then broken into search terms.
The second pass will expand any [function-name:...] functions in each search term.
- Is a regular built in Search Function
- Is a built in Search Preprocessor Function? (confusion on how it differs from above)
- Not a function, but just a Search Modifier?
- Is none of the above, but just a regular Property?
- Is the built in thing I'm using any of the above, or is it actually a macro? What actually constitutes a macro?
- Is anything built in considered a macro? Or do the things listed above avoid the expansion issue?
- Are macros only user defined? With the term only being used to reference things like: Filters & Bookmarks?
- How does `[define:definition]` fit into all of this? The syntax to call it is different (requiring a # hashtag) & documentation refers to it as a user defined function.
- Is this effectively just a Macro? Or is it a Function & somehow fundamentally different? How do they differ?
- Does it differ from macros defined using `/define name=search`?
- Does a user defined function (of either the `[define:definition]` or `/define name=search` variant) undergo expansion the same way macros (as in Filters & Bookmarks) do?
------------------
Anyway, I threw a $20 your way. Always impressed by your support, work, and commitment.
One day, could syntax highlighting be implemented (also multiline queries... a man can dream) ? Ideally to color matching braces & hint at invalid syntax. Also to potentially identify what's a:
- function
- preprocessor function
- search modifier
- property
- filter macro
- bookmark macro
- some other user defined macro/function (are there any others?)
- string (or what's being treated as a string)
- number
- operator (as in size:>10b) so the `>` would be colorized as an operator to avoid confusion with brace matching
------------------
EDIT:
I just noticed the `$param`. In what scenario do we get parameters and can use them in that form?case:< ext:mp3;jpg name:$param: >
- In `[define:definition]` form, params are `#arg:`. The `$` form (ie: `$columnHeading`) is used to reference column names (see viewtopic.php?f=12&t=10099#ifdef & #iflen below it) as well as (I think) metafields like $result-count:, $folder-result-count:, etc. & special variables such as $column0, $column1, ....etc.
- In macros, AFAIK you can't really operate on the passed in argument & it can only be passed to other things.
For example (that also showcases roughly what I envision a syntax highlighting feature would do), referencing the param is done like so:
case:< ext:mp3;jpg name:< param: > >
As far as I know, there is no $param: like functionality available outside of I think $column1:, $column2:, ...etc.
- Regarding the syntax highlighting / semantic formatting example:
- Spaces that are syntactically significant are brighter than the background color. However, spaces that are not significant to the syntax, for example any that'd be enclosed in quotes ie: "C:\Program Files (x86)" , would be treated same as any other enclosed token like characters such as : \ ( ) ..
➢ Spaces that are just eaten would not be colored unless touching a neighboring space that does matter - also there could be a single line of pixels @ normal background color separating them to visualize they're not necessary.
➢ The quote color is the same as mp3 & jpg which are interpreted as a string, but that actual quoted content should be slightly brighter so it doesn't all blur together / keeps tokens & identifiers obvious. - Notice space between ext: & name: clauses is helpfully slightly brighter & wider (em space U+2003) to make it obvious that space in particular is interpreted as a logical AND.
➢ Would be a nice QOL improvement.
➢ It'd need to be brighter than other spaces to differentiate from the "touching a neighboring space that does matter" spaces. - The <> & <> braces are almost the same color as the function/modifier/macro they belong to. In this case they're slightly brighter. This is so if another clause is nested that belongs to the same colored syntax type, the nested members brackets could instead be a darker shade. As more nesting takes place, the 1st level braces would get brighter yet, the 2nd level darker, and the 3rd level somewhere in between, etc.
➢ In my example they were not made obviously brighter (depending on the quality of your monitor, they're probably too close for gaming/6bit+FRC monitors to differentiate) because it'd only become necessary above 1 level nesting.
- Spaces that are syntactically significant are brighter than the background color. However, spaces that are not significant to the syntax, for example any that'd be enclosed in quotes ie: "C:\Program Files (x86)" , would be treated same as any other enclosed token like characters such as : \ ( ) ..
- Regarding the syntax highlighting / semantic formatting example:
EDIT 2:
I went through & reread most of the docs I referenced above. It makes a little more sense after reading the additional comments below the initial posts. But I still have some kind of mental block that's preventing me from entirely grasping std search FNs(especially vs std properties/column names), PreProcessor FNs, modifiers, user defined FNs, MACROs, & how to and when to tie them all together. What makes sense now is PreProcessorFNs return a value & SearchFNs filter results.
What I struggled with was piping output from a PreProcessorFN that contained nested SearchFNs or Properties to another SearchFN & putting all that together in a MACRO in a way that is valid syntax, then using that MACRO from the search box. But now I think I'm realizing that's not possible bcus SearchFNs are for searching and always affect the result list. I guess I thought I could pass the results of a SearchFN to a PreProcessorFN (basically behind the scenes would built an intermediate result list) that returned something I could then pass to a SearchFN to filter the visible result list. Part of the confusion stems from columnnames, properties, & SearchFNs resembling & basically being referenced/called the same way.
I also found the dollar `$` `$param` syntax is used in Column Formulas. But still, how would you pass a parameter to a column formula? Is that somehow possible?
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Thank you for your donation and support DerekZiemba,
Perhaps I need to reword a few things.
Here is the order in which Everything parses your search:
Preprocessor always looks like:
#[function:...#]:
-or-
#function:
These are expanded before anything else.
This was the only method available in the first preprocessor implementation.
With the first preprocessor implementation, #[quote:...#]: was designed to convert text to a search term.
Search modifiers are always before search functions / macros:
modifier:function:parameters
Macros:
macro:parameter
Macros override search functions.
Can expand into multiple search functions.
Search functions always look like:
function:parameter
property-name: is treated the same as search-function:
Almost all properties have an associated search function with the same name.
For example:
date-modified:
The search function parameter can include the following preprocessor style:
[function:...]
This style was recently added.
You no longer need to use the awkward #[quote:...#]: style.
The expanded text is treated as quoted text.
I will refer to this style as the "termprocessor".
Parameters can also include references to your macros.
Expanded macros are treated as quoted text.
There are shortcuts for the preprocessor and termprocessor.
A good example of this is the clipboard preprocessor function:
#clipboard: is the same as #[clipboard:#]:
clipboard: is the same as [clipboard:]
#clipboard: will treat spaces as AND
clipboard: will treat spaces literally
clipboard: preprocessor.
There are conflicting search functions and termprocessor functions.
For example, the year: search function and the year: termprocessor function.
Search functions are always used first, so:
year:year:"2023-09-09"
is the same as:
year:[year:"2023-09-09"]
=> year:2023
(The year: search function is then called with 2023)
year: preprocessor.
year: search function.
Since there's no clipboard: search function, clipboard: on its own is treated as a termprocessor function.
Eval scripts always include the following (anywhere in the term):
$property:
For example:
len($stem:)%3==0
=> match filenames where the stem length is a multiple of 3.
In short:
#[function:...#]: = preprocessor.
#function: = preprocessor.
[function:] = termprocessor.
$property: = eval script.
function: = modifier > macro > search function > termprocessor.
This is different to a filter macro, bookmark macro or just a macro.
The preprocessor does not have access to bookmarks/filters/macros.
Only use #define: if you want to access these defines from the preprocessor.
The default filters contain a few macros, for example: exe:
(Besides the default filters macros which can be deleted or changed)
You can reference your defines with either #[define-name:#]: or #define-name: or [define-name:]
You can call your defined functions with either #[define-name:...#]: or #define-name:<parameters> #define-name:parameters or [define-name:parameters]
Defines can be called or referenced from the preprocessor.
The preprocessor does not have access to filters/bookmarks/macros.
/define creates a macro.
It has nothing to do with the preprocessor define.
Depends on how they are called.
#[mydefine:#]: or #mydefine: will expand spaces as AND.
[mydefine:] will expand as quoted text.
mymacro: outside a search function will expand spaces as AND
mymacro: inside a search function will expand as quoted text.
I recommend using macros over preprocessor defines.
I will consider search syntax highlighting.
Thank you for the suggestion.
Previously, macros would have to define the parameter name.
Now you can reference the parameter without defining a parameter name.
A filter macro example:
Name: Upper Case Example
Search: case:[upper:paramname:]
Macro: up<paramname>
up:abc
=>
case:ABC
The preferred syntax is now:
Name: Upper Case Example
Search: case:[upper:$param:]
Macro: up
up:abc
=>
case:ABC
Preprocessor defines must define the parameter names.
Preprocessor defines can have more than one parameter.
Once with the preprocessor and again with the termprocessor.
(exactly the same as if you had typed the macro search into the search box)
$param: is only available inside the macro search.
I do want to add support for LUA at some stage.
However, I am really trying to avoid having the user enter in syntactically correct searches.
The Everything search syntax is flexible to allow searches like: size:>128mb (don't treat > as a closing the search group) -no one will want to escape the >
Each search function has it's own syntax.
Generally, most search functions all following the basic search syntax standard.
Space is always AND.
I am considering more aliases for operators.
for example:
::and::
::or::
::not::
::group-start::
::group-end::
This might make it easier to build complex macros.
Column Formulas are different again.
Column Formulas are C expressions which evaluate to a string or number.
Column Formulas are evaluated for each result -after the search.
Use $property: to reference the property values for the current result.
You can use column-a - column-f to define custom values and reference them with $a: - $f:
column-a - column-f can also be set from your search.
You can also use regex group captures and reference them with $0: - $9:
Things have changed quite a bit from the original preprocessor implementation.In most cases I'm not sure if what I'm using:
Perhaps I need to reword a few things.
Here is the order in which Everything parses your search:
Preprocessor always looks like:
#[function:...#]:
-or-
#function:
These are expanded before anything else.
This was the only method available in the first preprocessor implementation.
With the first preprocessor implementation, #[quote:...#]: was designed to convert text to a search term.
Search modifiers are always before search functions / macros:
modifier:function:parameters
Macros:
macro:parameter
Macros override search functions.
Can expand into multiple search functions.
Search functions always look like:
function:parameter
property-name: is treated the same as search-function:
Almost all properties have an associated search function with the same name.
For example:
date-modified:
The search function parameter can include the following preprocessor style:
[function:...]
This style was recently added.
You no longer need to use the awkward #[quote:...#]: style.
The expanded text is treated as quoted text.
I will refer to this style as the "termprocessor".
Parameters can also include references to your macros.
Expanded macros are treated as quoted text.
There are shortcuts for the preprocessor and termprocessor.
A good example of this is the clipboard preprocessor function:
#clipboard: is the same as #[clipboard:#]:
clipboard: is the same as [clipboard:]
#clipboard: will treat spaces as AND
clipboard: will treat spaces literally
clipboard: preprocessor.
There are conflicting search functions and termprocessor functions.
For example, the year: search function and the year: termprocessor function.
Search functions are always used first, so:
year:year:"2023-09-09"
is the same as:
year:[year:"2023-09-09"]
=> year:2023
(The year: search function is then called with 2023)
year: preprocessor.
year: search function.
Since there's no clipboard: search function, clipboard: on its own is treated as a termprocessor function.
Eval scripts always include the following (anywhere in the term):
$property:
For example:
len($stem:)%3==0
=> match filenames where the stem length is a multiple of 3.
In short:
#[function:...#]: = preprocessor.
#function: = preprocessor.
[function:] = termprocessor.
$property: = eval script.
function: = modifier > macro > search function > termprocessor.
#[define:...#]: or #define: or [define:] is a preprocessor define.Is the built in thing I'm using any of the above, or is it actually a macro? What actually constitutes a macro?
This is different to a filter macro, bookmark macro or just a macro.
The preprocessor does not have access to bookmarks/filters/macros.
Only use #define: if you want to access these defines from the preprocessor.
No, there are no macros if you delete all your filters.Is anything built in considered a macro? Or do the things listed above avoid the expansion issue?
The default filters contain a few macros, for example: exe:
Yes.Are macros only user defined? With the term only being used to reference things like: Filters & Bookmarks?
(Besides the default filters macros which can be deleted or changed)
Defines can be a simple string or complex functions.How does `[define:definition]` fit into all of this? The syntax to call it is different (requiring a # hashtag) & documentation refers to it as a user defined function.
You can reference your defines with either #[define-name:#]: or #define-name: or [define-name:]
You can call your defined functions with either #[define-name:...#]: or #define-name:<parameters> #define-name:parameters or [define-name:parameters]
Defines are fundamentally different to macros.Is this effectively just a Macro? Or is it a Function & somehow fundamentally different? How do they differ?
Defines can be called or referenced from the preprocessor.
The preprocessor does not have access to filters/bookmarks/macros.
My poor choice of naming here.Does it differ from macros defined using `/define name=search`?
/define creates a macro.
It has nothing to do with the preprocessor define.
Yes and no.Does a user defined function (of either the `[define:definition]` or `/define name=search` variant) undergo expansion the same way macros (as in Filters & Bookmarks) do?
Depends on how they are called.
#[mydefine:#]: or #mydefine: will expand spaces as AND.
[mydefine:] will expand as quoted text.
mymacro: outside a search function will expand spaces as AND
mymacro: inside a search function will expand as quoted text.
I recommend using macros over preprocessor defines.
I will consider search syntax highlighting.
Thank you for the suggestion.
Support for $param: was very recently added. (1355a)Just noticed the `$param`. In what scenario do we get parameters and can use them in that form?
Previously, macros would have to define the parameter name.
Now you can reference the parameter without defining a parameter name.
A filter macro example:
Name: Upper Case Example
Search: case:[upper:paramname:]
Macro: up<paramname>
up:abc
=>
case:ABC
The preferred syntax is now:
Name: Upper Case Example
Search: case:[upper:$param:]
Macro: up
up:abc
=>
case:ABC
Preprocessor defines must define the parameter names.
Preprocessor defines can have more than one parameter.
The search in macros are expanded twice.In macros, AFAIK you can't really operate on the passed in argument & it can only be passed to other things.
Once with the preprocessor and again with the termprocessor.
(exactly the same as if you had typed the macro search into the search box)
$param: is only available inside the macro search.
I do want to add support for LUA at some stage.
However, I am really trying to avoid having the user enter in syntactically correct searches.
The Everything search syntax is flexible to allow searches like: size:>128mb (don't treat > as a closing the search group) -no one will want to escape the >
Each search function has it's own syntax.
Generally, most search functions all following the basic search syntax standard.
Space is always AND.
I am considering more aliases for operators.
for example:
::and::
::or::
::not::
::group-start::
::group-end::
This might make it easier to build complex macros.
Column Formulas are different again.
Column Formulas are C expressions which evaluate to a string or number.
Column Formulas are evaluated for each result -after the search.
Use $property: to reference the property values for the current result.
You can use column-a - column-f to define custom values and reference them with $a: - $f:
column-a - column-f can also be set from your search.
You can also use regex group captures and reference them with $0: - $9:
Re: [Out of memory error]: Crashed @ 128GB & 237GB virtual memory committed
Everything 1.5.0.1359a adds a ancestor-attributes: search function.
ancestor-attritbutes: requires attribute indexing.
To enable attribute indexing:
Please note: all volume roots (for example: C: ) have the Hidden and System attributes set.
Everything will ignore these attributes for volume roots when using ancestor-attribute:
Example usage:
ancestor-attr:HDI
ancestor-attritbutes: requires attribute indexing.
To enable attribute indexing:
- In Everything, from the Tools menu, click Options.
- Click the Indexes tab on the left.
- Check Index attributes.
- Click OK.
Please note: all volume roots (for example: C: ) have the Hidden and System attributes set.
Everything will ignore these attributes for volume roots when using ancestor-attribute:
Example usage:
ancestor-attr:HDI