ES piping output is much slower than -export-* options

[Nnnes]

Everything's command line interface seems to be orders of magnitude slower when using a pipe vs. when using "-export-txt". For example, on my machine,

es "C:\Windows" > list.txt

takes 18.07 seconds, while

es "C:\Windows" -export-txt list.txt

takes 0.27 seconds. Is this a bug?

• I am hoping to run a query with a similar number of results (~200k) and use its output as part of a script I am writing. Did I miss some way to speed it up? (I could fall back to writing to and reading from a temporary file if I need to - or maybe try to write my own command line tool using the SDK.)
• The speed of the shell/pipe itself might have some effect, but it does not fully account for the slowdown: subsequently running "TYPE list.txt > list2.txt" takes 4.39 seconds.
• Using the "-txt" option has little if any effect on speed.
• Other formats might have a similar problem to an even greater degree. "-export-csv list.csv" takes 1.46 seconds; I stopped "-csv > list.csv" after over a minute and the file wasn't even a tenth of its expected final size.
• My test times are from PowerShell, but the command was also slow when I tried calling it and reading its output in a couple different programming languages.
• I am using ES 1.1.0.9.

[therube]

Confirmed.
Interesting.

Code: Select all

TimeThis :  Command Line :  es e:\windows > elistout

TimeThis :    Start Time :  Tue Jan 09 12:51:51 2024
TimeThis :      End Time :  Tue Jan 09 12:51:55 2024
TimeThis :  Elapsed Time :  00:00:03.843

Code: Select all

C:\out>timethis es e:\windows -export-txt eexport

TimeThis :  Command Line :  es e:\windows -export-txt eexport
TimeThis :    Start Time :  Tue Jan 09 12:53:17 2024
TimeThis :      End Time :  Tue Jan 09 12:53:17 2024
TimeThis :  Elapsed Time :  00:00:00.238

As it is, I most often redirect (or even Tee, which I haven't checked but I'll assume is rather expensive in itself).
I might have to think about exporting. (Though in my case, what's a little bit of time .)


(I've been meaning for some time now to post similar regarding -file-list, I think it is, vs. pasting a string of multiple OR's.)


?

Why does my program run faster when I redirect stdout?
Is using Redirection to print to file in C++ faster than using the convectional cout?

[void]

The slowness is from reallocating a buffer to convert the result unicode text to the active console code page.



ES-1.1.0.27 improves performance.
With this version you should see about the same performance with -export-txt out.txt and > out.txt

I've also added a -cp 65001 command line option to specify the code page when redirecting output.
Change 65001 to any code page you like.
65001 is UTF-8.
Omit -cp to use the current console code page. (usually 850)

-export-txt will always save as UTF-8.

[Nnnes]

Wow, thanks for the quick update!

Big improvements on my machine - both 32-bit and 64-bit 1.1.0.27 now take ~4.5x more time with "-txt > list.txt"/"-txt -cp 65001 > list.txt" compared to "-export-txt list.txt" rather than ~70x. (in cmd, 32: ~230ms/~970ms, 64: ~200ms/~900ms) Definitely usable for my needs.

When using a file redirector, PowerShell seems to be converting the piped output back to UTF-16 regardless of which code page I set, and I think slowing it down in the process. Not sure if anything can be done about this inside the program; just have to use e.g. "| Out-File -Encoding utf8 list.txt" instead of "> list.txt". (Super User: redirecting output in powershell produces UTF-16 encoded text)

...Now that I'm testing it a bit more, I can see that PowerShell is responsible for a significant portion of the extra time taken for the old version too. Running my original command with ES 1.1.0.9 is taking 13.83s in cmd vs my initial measurement of 18.07s in PowerShell. And the difference looks a lot like my TYPE command's execution time. Hmm...

[Nnnes]

Another factor that could affect performance is the specific implementation and efficiency of the command line tool you are using.

You don't say.

It's possible that the tool is optimized for writing output to a file using the "-export-txt" option, but not as efficient when processing data through a pipe. This can explain the significant performance difference you are experiencing.

You don't say.

To speed up your query and improve performance, you could consider writing the output to a temporary file and then reading from that file in your script. This approach can help bypass the potential slowdown associated with using a pipe.

...I already said.

Additionally, you mentioned the idea of writing your own command line tool using the SDK, which could also be a viable solution if you require further optimization and control over the processing speed.

Yes, that's why I mentioned it.

This one's quite a miss for your LLM. In the future, I would consider reading over its output to check whether it has actually given you any useful information, or any information at all that isn't already in your prompt, before copying and pasting. Cheers!