Index Journal Suggestions

[adamantine]

(i use win-server 2003 r2 sp2, 32-bit, 32 gb ram) (Everything-1.5.0.1238a.x86 portable)

thank you for the very promising version 1.5 (at the moment i may use it for the tests only)
the 'index journal' feature is probably the most interesting for me (along with the 'content indexing' feature)

speaking about 'index journal':
- since 2010 i had been constantly using the 'cenvorox file monitor' program (version 3.2.4.2) (made in 2006) (i still have to use it)
- not a single attempt (to substitute it with something more convenient and modern) was successful in these 11 years
- the 'index journal' is a ray of hope for sure
but right now there are quite a lot of negative aspects/questions/ideas that i would like to mention:

1) in my case the 'Export...' button allows to save a CSV-file with this content only:

ID,Date,Action,Name,Path,New Name,New Path,Size,Date Modified,Date Created,Date Accessed,Attributes,Parent Date Modified,New Parent Date Modified

this is the 1st line (the 2nd line is empty) (there are 2 lines only, while the 'index journal' window displays hundreds/thousands of ID's)

2) will it be possible (later) to choose which columns to show and which not to show?
for example, i certainly don't need these: Date Accessed, Attributes; and most likely several others too
while the horizontal space is quite valuable there


3) will it be possible (later) to export to usual .txt-file? (not only csv)

this is the pattern/example that i suggest to consider for such txt-file (which in fact will serve us as a log):

1   2021.03.14  09.42.25   File Rename   D:\Programs\my-test-9.txt   D:\Programs\my-test-10.txt
2   2021.03.14  09.42.25   Folder Modify   D:\Programs

but it even could be implemented in a lot better way (more convenient to look through a long txt-file whose content looks like a huge log; because in majority of cases the content of 1 line will fit that line horizontally):

1   2021.03.14  15.01.05   D:\Programs\my-test-9.txt   Rename From
2   2021.03.14  15.01.05   D:\Programs\my-test-10.txt   Rename To
3   2021.03.14  15.01.05   D:\Programs   Modify

the rest of example could look like this:

4   2021.03.14  17.42.25   D:\Programs\my-test-11.txt   Add
5   2021.03.14  20.03.00   D:\Programs\my-test-9.txt   Delete
6   2021.03.15  00.42.25   D:\Programs\my-test-10.txt   Delete
7   2021.03.15  00.42.26   D:\Tests\my-test-10.txt   Add
8   2021.03.15  03.00.59   D:\Programs\my-test-12.txt   Add

(yes, i think that all the time/date-related symbols have to be highly customizable: order, user-defined delimiters/spaces, presence of seconds...)

here ID's 6 and 7 are technically the 'move' action (this is just an alternative way to display the ID's, thus not using the 'move' definition at all)
the idea is based on the long experience which i have with the 'cenvorox file monitor' program

thanks to it, i also have the saved history of all my file/folder events since august of 2010: it occupies 1 gb in txt-files but they are not archived
(as you understand, they may occupy much less in a rar/zip-format)
but 'cenvorox' is not convenient in many ways: it constantly requires manual tidying up, it demands to be a service, it demands to have 1/2 processes in a loaded state, it makes impossible to rename any root folder in any drive (to actually rename any of it, 'cenvorox' may be completely unloaded but this is the additional and slightly tedious action)


4) so there is a very important feature that may be added to 'index journal':

a. possibility to manage/save/keep all the file-system-events (often it it necessary to return to some point and see exactly in tiniest details what was happening with this or that file/folder, even if it was long time ago)
b. i suggest to implement some convenient system that will allow to keep absolutely all ID's in automatically created (easy readable) txt-files
c. this system/feature will be aimed at advanced users, and it has to be very customizable
there should be a possibility to choose the folder where those txt-files will be created by ET, according to user-defined settings like:

- how many maximum ID's a txt-file may contain (for instance, we choose 20.000, then a new txt will be auto-created after each 20.000 ID's)
(my personal choice would have been 200.000 or even much more)
- what is the name pattern for such txt-file? for instance, i would have decided to see it this way: "2021.03.14  19.42.25.txt"

right after the auto-creation of a consecutive txt-file, the 'index journal' clears its content (which was already copied to a just saved txt-file)
but of course, the clearing must be done by ET only when the 'index journal' window is not opened

5) the upper 'Action' line could be improved
there is a lot of free space there to the right side, and thus the drop-down list is a pure non-ergonomic evil in this case
all its 11 entries could be made available directly as small and accurate buttons: horizontally placed, 1 by 1, with a corresponding inscriptions like: all actions, folder create, folder delete...


6) idea: if some line is selected (in the result field of 'index journal'), press on F2 may give this whole line the 'inline rename' state
(this will allow to select any part of a path/name/date... and send it to clipboard: this option could become a nice time-saver in many situations)

7) now the title bar is 'Index Journal'. it may become something like this: Index Journal - 3217
so it will immediately prompt a user: how many file/folder events are displayed right now in a long list (no need to scroll it down)
it may be useful if ID's sort is ascending (0 on top) which is my case always
(the '3217' value should not be auto-changed during the current session of the 'Index Journal' window)
(this suggestion is quite arguable and should be treated as least important)

8) something like the 'Import...' button may be considered too
those automatically created/named txt-files will be used with this new 'Import...' setting
(here the main sense is probably in the fast search/filter that is already implemented in the 'index journal' window)
but importing should not re-write the current (not saved so far) view in the 'index journal' window
so seemingly there should be some additional button like 'Switch To Current Events' (near the 'Import...' button)

[void]

Thank you for your feedback adamantine,

The Index Journal was created so you could monitor changes to your file systems.
I had never intended for it to log changes to disk. However...

1) in my case the 'Export...' button allows to save a CSV-file with this content only:
this is the 1st line (the 2nd line is empty) (there are 2 lines only, while the 'index journal' window displays hundreds/thousands of ID's)

I swear I tested this! It was a very recent sort optimization that broke it..

Please try the latest Everything 1.5 Alpha.

Version 1.5.0.1242a fixes an issue with exporting the index journal when no search is active.

2) will it be possible (later) to choose which columns to show and which not to show?
for example, i certainly don't need these: Date Accessed, Attributes; and most likely several others too
while the horizontal space is quite valuable there

Added to my TODO list.

3) will it be possible (later) to export to usual .txt-file? (not only csv)

Done, please try the latest Everything 1.5 Alpha.

Please let me know what you think, what needs to change.
Is there a standard log format going around?
Do I need to export the Journal Item ID for each line?

-I can't split the journal up internally to Rename To and Rename From as this needs to be a single event for the Everything Server. However, I do split them into separate To/From lines for the txt export.

(yes, i think that all the time/date-related symbols have to be highly customizable: order, user-defined delimiters/spaces, presence of seconds...)

I'm using rfc 3339 for the time format. Is there any reason to use something else?
Do you need more precision than seconds? -I can go to 100 nanoseconds.

a. possibility to manage/save/keep all the file-system-events (often it it necessary to return to some point and see exactly in tiniest details what was happening with this or that file/folder, even if it was long time ago)

I've added a journal_log_file ini option to specify a log file on disk to the latest Everything 1.5 Alpha.
Set this to the filename of your log file, eg: D:\Logs\Everything Index Journal.txt
I still need to add log file rolling... added to my TODO list.

b. i suggest to implement some convenient system that will allow to keep absolutely all ID's in automatically created (easy readable) txt-files

Keep item IDs, noted.
Currently, if you change your index, the Journal will be reset and the next item ID will start at 0.
I will look into keeping the journal through index setting changes.
I can't do this due to the Everything Server, clients use the Index Journal to synchronize their local indexes.
An index setting change may just have to force log rolling.

c. this system/feature will be aimed at advanced users, and it has to be very customizable
there should be a possibility to choose the folder where those txt-files will be created by ET, according to user-defined settings like:

- how many maximum ID's a txt-file may contain (for instance, we choose 20.000, then a new txt will be auto-created after each 20.000 ID's)
(my personal choice would have been 200.000 or even much more)

Added to my TODO list: max line count, max size limits for log rolling

- what is the name pattern for such txt-file? for instance, i would have decided to see it this way: "2021.03.14 19.42.25.txt"

Added to my TODO list: auto name for log file (eg: date), daily logs.

right after the auto-creation of a consecutive txt-file, the 'index journal' clears its content (which was already copied to a just saved txt-file)
but of course, the clearing must be done by ET only when the 'index journal' window is not opened

This would be handled with log file roll over.
I am consider another tool to do this... I will need to create an API to pull this data out of Everything.

5) the upper 'Action' line could be improved
there is a lot of free space there to the right side, and thus the drop-down list is a pure non-ergonomic evil in this case
all its 11 entries could be made available directly as small and accurate buttons: horizontally placed, 1 by 1, with a corresponding inscriptions like: all actions, folder create, folder delete...

I'll consider checkboxes instead of the dropdown combobox.

6) idea: if some line is selected (in the result field of 'index journal'), press on F2 may give this whole line the 'inline rename' state
(this will allow to select any part of a path/name/date... and send it to clipboard: this option could become a nice time-saver in many situations)

This already exists, unless I'm missing something? -It's not inline though, it brings up a rename dialog.
There's also Ctrl + C to copy the path of the currently selected items.
Right clicking gives more options.

7) now the title bar is 'Index Journal'. it may become something like this: Index Journal - 3217
so it will immediately prompt a user: how many file/folder events are displayed right now in a long list (no need to scroll it down)
it may be useful if ID's sort is ascending (0 on top) which is my case always
(the '3217' value should not be auto-changed during the current session of the 'Index Journal' window)
(this suggestion is quite arguable and should be treated as least important)

I wanted this too. I was thinking of a status bar, the window caption will work for now..
Done in the latest Everything 1.5 Alpha.

8) something like the 'Import...' button may be considered too
those automatically created/named txt-files will be used with this new 'Import...' setting
(here the main sense is probably in the fast search/filter that is already implemented in the 'index journal' window)
but importing should not re-write the current (not saved so far) view in the 'index journal' window
so seemingly there should be some additional button like 'Switch To Current Events' (near the 'Import...' button)

I'll consider it.

Thank you for your helpful suggestions.
I hope I can make the Index Journal work for you.

[mtu]

i'm testing the alpha and appreciating this feature as an Avafind refugee this reminds me of the must-have feature(for me) that it used to have, the Scout Bot. it enabled a user to monitor the system for created and modified folders and files and it is great to see such a feature finding its way to Everything, Thank you.

i'd like to offer my suggestions on this feature

- would it be possible to enable the Index Journal window to be dockable ? it would be great to have it always visible and attached to the main window
- i'd also like to be able to create custom Actions. let's say i'd like to view only deleted files and folders, currently we can only view one or the other but with the ability to see what Actions that we want to see we can mix and match the Actions that would suit our needs.

version 1.5 is shaping up to be an exciting release and hopefully i can fully let go of the memories of the user experience i had with Avafind(specifiacally the Scout Bot) all those years ago and go all in with Everything.

[NotNull]

- would it be possible to enable the Index Journal window to be dockable ? it would be great to have it always visible and attached to the main window

A different approach to that:
When you configred Everything to run in the background (Menu:Tools > Options > UI ), you can even close the Everything Search window.
The Index Journal will stay opened.

To open a new search window, double-click the Everything system-tray icon (for example; there are more ways).

[mtu]

whew, that was fast! appreciate you getting back to me.

yes i've noticed that the Index Journal remains open when Everything's main window is closed but i'd like to simulate the Scout Bot feature from Avafind. here the Scout Bot showing recent filesystem changes was to the right of the search results.

Ava Find.JPG (55.52 KiB) Viewed 178340 times

[NotNull]

For the record, @void is the developer of Everything.
I am just someone who helps here answering questions and have no influence on how Everything evolves (which is a good thing! )

Some time ago I played around with Ava Find. I did like the ScoutBot, but also think the Index Journal will be much more powerful in the end (I like teh extra information)

Your suggestion to select multiple actions will certainly help with that (missed that too a couple of times).
The user interface is still under heavy development - it is an alpha release after all - so we will have to wait what @void has to say if and when this gets implemented (there is still a lot to do .. )


FWIW: I liked Ava Find's way to present filenames (in two lines; for others, see here for an example)

[mtu]

For the record, @void is the developer of Everything.
I am just someone who helps here answering questions and have no influence on how Everything evolves (which is a good thing! )

have seen some of your posts around the forum and suspected as much but your input/help is still very much appreciated it is good to have your queries answered in a matter of hours and not days, if ever.

Some time ago I played around with Ava Find. I did like the ScoutBot, but also think the Index Journal will be much more powerful in the end (I like teh extra information)

yes already the Index Journal already offers more than the Scout Bot which is why i'm excited about this feature and the release of version 1.5.

Your suggestion to select multiple actions will certainly help with that (missed that too a couple of times).
The user interface is still under heavy development - it is an alpha release after all - so we will have to wait what @void has to say if and when this gets implemented (there is still a lot to do .. )

let's hope he takes it on board as this is the perfect time to forward these suggestions before the door is closed near to final release.

FWIW: I liked Ava Find's way to present filenames (in two lines; for others, see here for an example)

had forgotten how convenient it was to have information presented that way, it saves horizontal space and looks a bit better. hopefully this two line display makes its way into everything one day.

[NotNull]

let's hope he takes it on board as this is the perfect time to forward these suggestions before the door is closed near to final release.

Good point!

I find it really amazing that on the third day after the very first release of an alpha version, we are already talking about suggestions for new functionality. The forum should be full of bug reports!
Which means that @void has done a really excellent job here (but undoubtedly more bugs will still come to light ...)

[adamantine]

void, i'm grateful for the detailed answer and for your serious approach to various aspects
even if some of them are not that big and may seem not that important

1 of my main goals is to never lose a single line/event from the history
so now i'm trying to understand for myself: will it be possible at all to switch to the 'index journal'?
and if 'yes', then: how exactly should i maintain this whole new system? (without the 'cenvorox file monitor' program)
which means: i need to be sure that ET won't get me into some real trouble like this:

Currently, if you change your index, the Journal will be reset and the next item ID will start at 0

i didn't knew about this aspect while writing my previous message in this thread

but at the moment the 'index journal' is still in the process of implementation
and some most essential things are not ready yet
so that's why in my posts here i mention various small questions/ideas too
(though i'm much more worried about the main things like: correct exporting and further convenient reading of the saved txt-file)

export to txt-file is successful now (the yesterday's csv-bug is gone)
(i don't think that we need to be bothered about 'standard log format' because the txt-format probably should be enough)


1) i just noticed that 'index journal' has severe trouble with the date-time:
they are displayed according to the moment when the 'index journal' was opened (not when the events were actually happening)

this behavior can be seen if you load the 'index journal' after some period of time: and see at the dates-times of the just added (to the list) ID's
they all will have equal date-time (the exact date-time of the moment when you opened the 'index journal')

and even more: the exported txt-file also has these numerous equal dates-times
while it is clear that they do not reflect the real state of things:
the real events were happening earlier (and in very different hours/minutes/seconds)


2) (inside the exported txt-file) all notes like 'File Move From', 'File Modify'... are strangely placed/arranged
sometimes the gap looks like 2-3 empty symbols (and sometimes 8 or more)
i propose to change it so that there were always exactly 9 spaces before such notes (namely spaces, not some 1 huge 'tab' symbol)
while 6 spaces may be placed between date-time and path

currently all lines in this txt-file begin with the date: i ask to add the number to start of each line
3 spaces may be placed between the number and the date

speaking about the view of date-time. now it looks like this:
in the main window of 'index journal': 2021.03.16 00.58
while in the exported txt: 2021-03-16 00:58:03
is it possible to have these 2 various versions look identically? exactly like this: 2021.03.16   00.58.03

so here's the example of the desired display (in the exported txt):
1   2021.03.15   22.58.03      D:\Images\2021.03.15\example-8.png         File Create
2   2021.03.15   23.04.58      D:\Images\2021.03.15\example-8.png         File Delete
3   2021.03.16   01.02.21      D:\Images\2021.03.16\example-number-9.png         File Rename From
4   2021.03.16   01.02.21      D:\Images\2021.03.16\example-number-10.png         File Rename To

so, to sum up:
3 spaces between number of line and date
3 spaces between date and time (if possible)
6 spaces between time and path
9 spaces between path-file and action

all these small aspects will help a lot because this txt should be created in an easy readable/searchable way


3) a small idea: new button 'pause' (in the main window of 'index journal')
sometimes it may become useful when concentrating on some details inside the numerous lines of 'index journal'
(without any kind of distraction like: constantly resizing vertical scroll, constantly changing number in the title bar)
(and it seems that further press on 'pause' should load all the "missing" events to the list)
(so this is the pause for the view only: generally the info about 'file system events' should not be lost)

4) a tiny question: now ID's are starting with 0. is it possible to have them always starting from 1?

5) some people may like to use the drop-down list (not the checkboxes)
so maybe it's better to implement it as an ini-setting (if it is technically possible):
option-1: current state (drop-down list only)
option-2: 11 checkboxes only
option-3: both the drop-down list and 11 checkboxes

and as was mentioned above in the other post: some additional and combined actions probably may be useful
for example: Folder+File Create, Folder+File Delete...
i propose to name them this way: Items Create, Items Delete...


6) i'm also thinking about the following aspect:
my main ET is 1.4.1.963 and i always use it with these 4 columns only: name, extension, size, date modified
as you can see, there is no 'path' column (because it's combined with the 'name' column)
there's a corresponding ini-option to do it (i became slightly happier when i found it by chance)

i had been using this approach/view for a year or 2 (and i find it very-very convenient)
in no way i'm going to return to the standard 'path column + name column' system
so maybe later we should consider/discuss something like this for the 'index journal' too

(i apologize for possible mistakes and for possible not very logical moments in my message:
there are too many nuances here, while i have to describe them all not in my native language)

[void]

Thanks for the feedback adamantine,

I've simplified the logging so I have something working for now in Everything 1.5.0.1243a.
More options to come...

To enabling journal logging:

• In Everything, from the Tools menu, under the Debug submenu, click Config
• This will open your Everything configuration file in Notepad.
• Change the following lines:
  journal_log=0
  to:
  journal_log=1
• Save changes and close Notepad.
• Accept the prompt in Everything to reload your configuration

When journal logging is enabled, Everything will create the following file:
%LOCALAPPDATA%\Everything\index-journal-2021-03-16.log
-where 2021-03-16 is the current date.

A new log will be created for each day. (this will probably need to be monthly)

The contents of the log file look like:

Journal-ID Item-ID Date Filename File/Folder Action

For example:
01d71987c50842f9 999 2021-03-14 09:42:25 D:\Programs\my-test-10.txt File Delete

The Journal-ID with an Item-ID will give you a reference to a unique change (Since the Journal-ID can change and the Item-ID will reset to 1).
The Date is now the USN time-stamp, not the Everything log time.

2) (inside the exported txt-file) all notes like 'File Move From', 'File Modify'... are strangely placed/arranged

I'm using TABs so it is easier to parse the data. I will consider spaces.
If I use spaces, I would put the action before the filename, eg:

1 2021.03.15 22.58.03 File Create D:\Images\2021.03.15\example-8.png

The action could be space padded so the filename always starts on the same column.

speaking about the view of date-time. now it looks like this:...is it possible to have these 2 various versions look identically? exactly like this: 2021.03.16 00.58.03

I'm using RFC 3339 for the date format for easy parsing.
2021.03.16 00.58.03 is your local format.
I will consider using the local format.
Will you ever parse or analyze these files later? how will you read them?

3) a small idea: new button 'pause' (in the main window of 'index journal')

You can pause the journal from main Everything search window -> Index -> Pause updating.
I will consider adding this option to the Index Journal somewhere..

4) a tiny question: now ID's are starting with 0. is it possible to have them always starting from 1?

Done, I'm still using indexes internally, so ID = internal_index + 1

i propose to name them this way: Items Create, Items Delete...

I'll consider this, thanks for the suggestions.

[adamantine]

1) now ID's are starting with 1, thank you

2) at the first glance, the "severe trouble with the date-time" seemed to be gone in the new '1243'-alpha-version (i'm not completely sure about it)

but when i again loaded ET (after several hours), i saw that many entries are arranged very strangely (inside the 'date' column). example:

1425 2021.03.16 17.58 File Create some-file-1
1426 2021.03.16 17.58 File Delete some-file-2
1427 2021.03.16 16.04 File Create some-file-3
1428 2021.03.16 16.05 File Modify some-file-4

the earlier events (from 16.04 and 16.05) are placed after the newer events (which is not normal/acceptable)


3) if i had the possibility, then i would have changed the following things:

a. remove the notes like "01d71a6b2ddb91ca", "01d71987c50842f9"
(horizontal space is very important in the exported txt-file: often the path-name may be quite long) (moreover, these notes look awful)
i'm trying to understand: is 'a reference to a unique change' that necessary?

b. not use "%LOCALAPPDATA%\Everything\"
(all related folders/files should be placed in the ET-folder) (it's the portable application after all, at least in my case)

c. not use the .log extension (the .txt extension is enough)

d. bring back the 'File Rename From', 'File Rename To' instead of these new variants: 'File Rename-From', 'File Rename-To'

i even propose to consider these simplified variants: Rename From, Rename To, Create, Delete...
(without specifying if it's a Folder or a File)
(in the overwhelming majority of cases we already have a dot and an extension if it's a File, and thus we may distinguish a File from a Folder)

(i also think that it's better to use 'Add' instead of 'Create')

this will probably need to be monthly

'monthly' is of course a long way better than 'each day'. but 'weekly' seems to be the optimal choice
(or we may return to the idea about limits: '20.000 or 200.000' from the 1st message of this thread)

I will consider spaces

ok. i hope to see them someday
(generally, the current display of the exported txt-file is really not good for me: i hope that none of us will give up and that this part of ET will be functioning right eventually. i'm not proud of it, but i was very close to giving up after i saw this new alpha-version)

Will you ever parse or analyze these files later? how will you read them?

speaking honestly, there's nothing serious/interesting here in my case. i will just open these exported txt-files from time to time (often) (sometimes very often) and analyze some part of it visually
(i also may edit them or copy something from them to clipboard)
(i also search inside such history-files a lot: using my Notepad2 or using the content-search of ET)

so sometimes it is kind of a reference, while in other cases it can also be a way to get the info that helps me to repair some mistakes, accidentally made by me or by some program or by some script or maybe even by the OS

I will consider adding this option to the Index Journal somewhere..

i propose to consider adding it as a button with the unpressed/pressed state
(Index Journal should remember a state of the button across the sessions of ET)

[void]

Thank you for your feedback adamantine, and for persisting with Everything 1.5a..

I've made some changes to log file layout in Everything 1.5.0.1245 as I now understand you are only going to view these logs (not analyze/parse them):
I'm using 3 spaces now.
Please let me know if this layout is an improvement, how is the 'action' column positioning?
I've also added the option to toggle the Index Journal columns and I've disabled some of the unnecessary ones by default.
There's still a lot to do here, I hoping I'm making some progress..

2) at the first glance, the "severe trouble with the date-time" seemed to be gone in the new '1243'-alpha-version (i'm not completely sure about it)

I'm looking into this issue. Are these files on different drives? Everything will update one drive at a time, so it is possible you'll see a bunch of updates for one drive, and then another drive. Keeping Everything running in the background should help prevent this as there will be less updates to process.

a. remove the notes like "01d71a6b2ddb91ca", "01d71987c50842f9"

This is the Journal ID. Combined with the item ID, this will give you a unique reference to an index change.
I've removed this Journal ID column. However, keep in mind the ID could reset to one due an index setting change.
I will look into keeping a unique ID across index setting changes.

a reference to a unique change' that necessary?

For viewing the logs, no.
For analyzing the logs, yes.
The Journal ID could go in the filename.

b. not use "%LOCALAPPDATA%\Everything\"

With the latest update, Everything 1.5 will now store the logs in the "logs" folder in the same location as your Everything for the portable version.

c. not use the .log extension (the .txt extension is enough)

I will try .txt
FYI the -1.5 in the filename is just the instance name, this will be removed in beta/release.

d. bring back the 'File Rename From', 'File Rename To' instead of these new variants: 'File Rename-From', 'File Rename-To'

Done, I was trying to make it parse-able, but now I know this is not necessary.

(without specifying if it's a Folder or a File)

Done.

'monthly' is of course a long way better than 'each day'. but 'weekly' seems to be the optimal choice

I've abandon the monthly idea, the logs will grow way too large and will be unviewable.
I'll explore storing single day logs in a weekly or monthly subfolder.

(or we may return to the idea about limits: '20.000 or 200.000' from the 1st message of this thread)

It's on my TODO list.
Thank you for your suggestions.

[adamantine]

here's the description of my situation (just in case):
a) i have 0 ssd. currently i have 5 hdd (all ntfs) (just 5 logical drives: c, d, e, f, g) (totally 600.000 objects: folders and files)
b) i never use: ET service, ETP/FTP server, HTTP server
c) i never use: installed ET (portable only)
d) i never have ET running in background (luckily, i don't even have the notification area in my OS anymore)
here i mean: the "run in background" setting is always disabled

e) i have the "create a new window when running Everything" setting always enabled
f) i constantly have 0 or 1 or 2 or even more instances of ET opened (with various searches in each of them)
g) when i finish some current task in 1 instance of ET, i usually press Num5. which means Alt+F4 in my main ahk-script
h) when i need to search for something in ET (while other instances of ET are busy or absent), i press the / button (near the right shift)
which means "run ET" in my main ahk-script

Are these files on different drives?

yes


in 1.5.0.1245 i created an exported txt-file. there were 232 actions. it turned out that the txt contained 244 lines:
the last line is empty (which is bearable but not good), while there were also defective/duplicated numbers of lines. examples:

4   2021-03-19 21:41:31   Create        F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini.tmp
5   2021-03-19 21:41:31   Rename From   F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini.tmp
5   2021-03-19 21:41:31   Rename To     F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini
6   2021-03-19 21:41:32   Create        C:\Program Files\Cenvo-Rox-File-Monitor\data\cr_file_events.cdb-journal

30   2021-03-19 21:43:57   Delete        F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini
31   2021-03-19 21:43:57   Rename From   F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini.tmp
31   2021-03-19 21:43:57   Rename To     F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini
32   2021-03-19 21:43:57   Create        D:\Images\2021.03.19\2021.03.19   21.43.57.png

78   2021-03-19 21:46:50   Delete        F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini
79   2021-03-19 21:46:50   Rename From   F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini.tmp
79   2021-03-19 21:46:50   Rename To     F:\New-4\Everything-1.5.0.1245a.x86\Everything-1.5a.ini
80   2021-03-19 21:46:50   Create        C:\Program Files\Cenvo-Rox-File-Monitor\data\cr_file_events.cdb-journal

as i understand, these duplications are only related to the "Rename From"/"Rename To"-actions (for some reason)
(added later: the duplications were already present in the 1.5.0.1243 version)


here are my thoughts concerning the general appearance of the exported txt-file:
it looks not good if the "action" column is placed before the "path-name" column
it is also harder to visually compare adjacent paths/names (which sometimes may be quite similar), while for example i have to find a small difference
so i ask to bring back the previous approach (when the "action" column was after the "path-name" column)

currently the situation with spaces is not optimal:
- 3 spaces (between number of line and date) look good

- 1 space (between date and time) (instead of 3 spaces) looks not good (but kind of bearable)
not equal separators (hyphens in date, colons in time) look not good (but kind of bearable)
just 4 dots (instead of 2 hyphens and 2 colons) would have been the better variant
but frankly speaking, even the current display of date-time is already better than in the "cenvo-rox" program

- 3 spaces (between time and next column) are quite normal
(i was previously asking for 6 spaces but this is the aspect where i'm completely ready for a compromise)

- i noticed that the quantity of spaces is very different between various "actions" and the "path-name" column:
8 spaces after Delete, 8 spaces after Create, 3 spaces after Rename From, 5 spaces after Rename To, 8 spaces after Modify
this issue may be solved this way:

the "action" column becomes the last column
6 spaces are added between "path-name" and any type of "action"
if it's technically hard to implement 6 spaces here, then let it be 3 spaces (but it's visually not convenient here: 3 is too little)
(i was previously asking for 9 spaces but this is the aspect where i'm also ready for a compromise: 6 will be quite good, 3 will be kind of bearable)

I'll explore storing single day logs in a weekly or monthly subfolder

actually, it's not that easy for me to define the best solution right now
briefly i'll try to describe my long-term practice with that "cenvo-rox" program:

(approximately once a week) (or sometimes less frequently) i manually export all actions to the txt-file
and right after that i clear those very events actions in the program's GUI (using 1 simple setting in the main menu of the program)
sometimes the txt-file may contain ~50.000 actions (lines), sometimes it's 200.000 or 300.000
(added later: in case of some error or some my mistake, i may manually export all actions to the txt-file substantially more often than just once a week)

in the program's GUI: the quantity of actions is always static, so i never lose a single action
1 of the main problems is: the rarer i perform such cleaning, the longer i wait for the program's GUI to load all the numerous lines (with actions)
(added later: 1 line contains 1 action)
after the load: i see the appearing of a complete list of actions that were done right after my previous manual export to a txt-file

[void]

Thank you for your feedback adamantine,

a) i have 0 ssd. currently i have 5 hdd (all ntfs) (just 5 logical drives: c, d, e, f, g) (totally 600.000 objects: folders and files)

Thank you for sharing your setup.
Would it be preferred to a have separate logs for each drive?
-This would keep Dates in sequence.

My current thoughts are to make a separate program to log the USN Journal to suit your needs.

in 1.5.0.1245 i created an exported txt-file. there were 232 actions. it turned out that the txt contained 244 lines:
the last line is empty (which is bearable but not good), while there were also defective/duplicated numbers of lines.

I could use a unique Log ID for each line. However, these would not match the IDs in the Index Journal UI.
I am assuming you would need the Log ID to match the ID in the Index Journal UI. -Please let me know if this is not the case.
The Index Journal must have a single ID for a complete rename operation (old name and new name) for Everything Server synchronization.

it looks not good if the "action" column is placed before the "path-name" column
it is also harder to visually compare adjacent paths/names (which sometimes may be quite similar), while for example i have to find a small difference

I assumed you are using a mono-spaced font for reading the logs.
What about using a 3 letter appreviation, eg: ADD/DEL/RNT (rename to)/RNF (rename from)/MVT (move to)/MVF (move from)

- 1 space (between date and time) (instead of 3 spaces) looks not good (but kind of bearable)

I need to add custom date/time formatting code to do this, its on my TODO list.
The current version is using rfc 3339. The plan is to use your local format, my local time formatting code currently does not support seconds so it's on my TODO list.

the "action" column becomes the last column

I find this hard to read, maybe using [DELETE] after the filename with square brackets might help.

(approximately once a week) (or sometimes less frequently) i manually export all actions to the txt-file

Thanks for sharing your cenvo-rox usage.
I'll look into some sort of simple mechanism to call Everything daily-weekly to update a log.

[adamantine]

Would it be preferred to a have separate logs for each drive?

no, just 1 log is needed (otherwise, i'll constantly have 5 times bigger concern/worry)

I am assuming you would need the Log ID to match the ID in the Index Journal UI

yes, this assumption is correct. i would like to add a few thoughts here:
- this match seems to be a very natural and useful thing
- but if it's technically very difficult to achieve, i may try to test some future implementation where the Log ID's are totally absent
(this could be the last resort)


these days i'm much more worried about the fact that the 'index journal' is maybe not very reliable
(even if we speak about it's main functionality)
here's the example of what i mean:
- in the evening: i perform some file/folder-actions
- in the night: ET is completely absent in the taskbar (both the regular 1.4-version and the 1.5-alpha)
- in the morning: i load the 1.5-alpha, open its 'index journal', and see that only some part (of my evening file/folder-actions) is present
- simultaneously i also notice that all not numerous 'night'-actions (auto-saves of my opened firefox, for example) are totally absent
(while they are present in the 'cenvo-rox' program)

- so i see a huge period of time (many hours) which is not reflected/reported in the 'index journal' in any way
(while some actions were certainly performed during that period of time)
- in the end of the 'index journal' i see several very fresh actions (belonging to the minutes when i loaded the current instance of 1.5-aplha)

(i'll continue to observe this strange behavior of the 'index journal')

The Index Journal must have a single ID for a complete rename operation

yes, i understood that
(i'm trying to comprehend if those duplicated numbers are very big evil
i don't know, maybe i'll manage to get used to them somehow)

I assumed you are using a mono-spaced font for reading the logs

here my knowledge is only enough to say that i use Notepad2 with these 'default font' settings:
font - tahoma, font style - regular, size - 12, script - cyrillic
but i suspect that this info is not important at all in this thread

What about using a 3 letter appreviation, eg: ADD/DEL/RNT (rename to)/RNF (rename from)/MVT (move to)/MVF (move from)

i feel that it's definitely not for me (i also never liked any kind of abbreviations)

I find this hard to read

i can understand this. what about trying it this way:
the "action" column is placed before the 'path-name' column (like you prefer)
but at the same time the beginning of the 'path-name' column is arranged so that all paths begin very-very equally (vertically)

there will be rare but obvious exceptions in cases when there is passage from 9 to 10, 99 to 100, 999 to 1000...
(these exceptions will not exist if we really decide to get rid of the Log ID's)

my quick calculations show that we need to add 12 spaces after any action:
Create, Modify, Delete, Rename From, Rename To, Move From, Move To
or if the spaces are not good enough to accurately arrange the paths vertically: maybe we should try again your initial approach (TABs instead of spaces before the 'path-name' column)

maybe using [DELETE] after the filename with square brackets might help

i don't think that square brackets (or any other brackets) (or the action displayed in the upper case) are a good idea

[void]

Thank you for your feedback adamantine,

these days i'm much more worried about the fact that the 'index journal' is maybe not very reliable

I will look into this.
It could be Everything performing a fresh reindex -it would be useful to log the journal ID to know for sure. -Check the Last build date from Tools -> Debug -> Statistics -Is this build date recent? -What is the last reported rebuild reason?

- so i see a huge period of time (many hours) which is not reflected/reported in the 'index journal' in any way

Was there actions in the journal with a date before this missing time period?



I've added a journal_log_format option to Everything 1.5.0.1249a.

Please try the following format:

• In Everything, type in the following search and press ENTER:

  Code: Select all

  /journal_log_format=#split:#id:   #date:<YYYY.MM.DD   hh.mm.ss>   #filename:      #action:
  

  Type in the following search and press ENTER:
  /restart

Syntax is:
#split: = use two lines for rename / move
#id:<width> = item ID
#date:<format> = date in rfc 3339 or in the specified format. For example: <YYYY-MM-DD hh:mm:ss.SSS>
#action:<width> = action text.
#filename:<width> = old filename (old and new if using split)
#newfilename:<width> = new filename.

<width> (optional) = pad text to width characters. Prefix with - to left align. For example, show action left aligned to 6 characters:
#action:<-6>

[adamantine]

It could be Everything performing a fresh reindex

i may be wrong but i don't think that this is the reason

Is this build date recent?

info from Tools -> Debug -> Statistics:
Last build date: 2021.03.24 05.16

info from index journal:
1   2021.03.24 05.16   File Create         Path: D:\Images\2021.03.24         Name: 2021.03.24   05.16.33.png

info from exported txt:
1   2021.03.24   05.16.33   D:\Images\2021.03.24\2021.03.24   05.16.33.png      Create

What is the last reported rebuild reason?

the 'Last rebuild reason:' line is empty

Was there actions in the journal with a date before this missing time period?

yes, many of them. another very fresh example:
i still keep the earlier 1245-alpha (it had been untouched by me for many-many hours: but i don't remember the exact intermission)
so i just opened it and saw the following situation: there are 103.810 actions

the actions from 1 to 103.696 belong to this period of time: from 2021.03.19 21.41 to 2021.03.23 19.10

the action 103.697: 2021.03.24 01.37 (here i have the obvious absence of numerous actions that were actually done after 2021.03.23 19.10)
the action 103.698: 2021.03.24 01.38
the action 103.699: 2021.03.24 05.11 (here i also have the obvious absence of numerous actions that were actually done)

the last action 103.810: 2021.03.24 07.06
and even in the very recent 2 hours: there were numerous actions which are totally absent here (above the last action)

(these aspects are slightly complicated for me so please ask more questions in case my info is not very clear)

journal_log_format option

thank you for adding it
from the beginning i was thinking about something like this but didn't pay a substantial attention to such possible approach
the setting adds an important flexibility and enlarges the user friendliness

(at the moment, i failed to find any meaningful use for the <width> setting
but that's not a big deal. maybe i'll look into it later)


i noticed that with these settings in the ini-file:

journal_log=1
journal_log_directory=F:\New-4\Everything-1.5.0.1249a\my-jl

the automatically created txt-file (inside the just created 'my-jl' folder) gets modified literally every second
(as long as 1.5-alpha is opened)
(there's something wrong here. besides that, this is absolutely not the behavior that any hdd/ssd likes, i think)


in the main window of index journal:
a) sort by date doesn't work normally
i mean that sort by date should take into consideration the 'time' as well (but it doesn't)
so currently: sort by date seems to be senseless (both ascending and descending)
because there are cases when, for example:
2021.03.24 06.03 is present before the 2021.03.24 05.46
while later there is another 2021.03.24 06.03 that is placed after the above-mentioned 2021.03.24 05.46

b) the 'time' should have seconds, in my opinion
c) there should be slightly bigger visual delimiter (between date and time) than just 1 space
yes, this is important in the main window (of index journal) too
3 spaces could be a best variant (maybe 2 is normal but certainly not 1)

[void]

i may be wrong but i don't think that this is the reason

Can you confirm there's a gap of missing events where the item ID is still in sequence.
For example:

Item ID / Date / Action / Filename
100 / 2021.03.24 05.16.33 Delete / foobar.txt
... gap of missing changes for 2 hours ...
101 / 2021.03.24 07.16.33 Delete / foobar2.txt

Is the Item ID consecutive?
Please make sure the action combo is set to all actions and there is no active search.
It could be a monitoring issue which I am still investigating..

the automatically created txt-file (inside the just created 'my-jl' folder) gets modified literally every second
(as long as 1.5-alpha is opened)
(there's something wrong here. besides that, this is absolutely not the behavior that any hdd/ssd likes, i think)

Everything will constantly stream to this log file (WriteFile)
Data is not flushed to disk and should be buffered.

Is the log file constantly reported in your Index Journal as Modified? -If so, it may indicate an issue with closing and reopening of the same file over and over.

a) sort by date doesn't work normally

The date sort is currently the same as the ID sort.
I haven't updated since changing the date to use the USN Journal date.
...Added to my TODO list.

b) the 'time' should have seconds, in my opinion

Added to my TODO list: Use long time format for index journal.

To set Everything to show seconds in the time format:

• In Everything, type in the following search and press ENTER:

  Code: Select all

  /time_format=HH.mm.ss

Note: this format is used for all times in Everything.

Time Format.

c) there should be slightly bigger visual delimiter (between date and time) than just 1 space
yes, this is important in the main window (of index journal) too
3 spaces could be a best variant (maybe 2 is normal but certainly not 1)

Everything uses your Windows Region and Language settings to format the date and time.
To override your region and language format:

• In Everything, type in the following search and press ENTER:

  Code: Select all

  /date_format=yyyy.MM.dd  

  Note the double space at the end.

Note: this format is used for all dates in Everything.

Date Format.

Thank you for your suggestions.

[adamantine]

i'm not sure that the following screenshots help us but nevertheless here they are just in case:
1) the id's 1-3780 are absent for some reason:

2021.03.27 16.45.02.png (190.57 KiB) Viewed 176415 times

2) another example of that not normal order in the date column:

2021.03.27 16.40.22.png (195.98 KiB) Viewed 176415 times

3) those screenshots are from yesterday, while this is the just created screenshot:

2021.03.28 06.57.27.png (187.36 KiB) Viewed 176415 times

Can you confirm there's a gap of missing events where the item ID is still in sequence

it's quite hard to find any useful info (including the 'gap of missing events') because of not adequate sort in the date column
but anyway, it seems that i can confirm there's a gap of missing events


moreover, currently (under such circumstances) there's not much sense in the id column for me (and unfortunately i cannot even disable it)


in case i'm doing something wrong: is there any relatively simple algorithm that i may try to perform?
(so that you got more useful answers) (and so that this whole situation stopped being too complicated for me)

for example:
- download the newest alpha-version, run its exe-file
- disable the 'run in background' setting in the options, close this alpha-version
- open its newly created main ini-file
- make this exact change in 'the line ...' to ...
- make this exact change in 'the other line ...' to ...
- close the main ini-file, load this ET-alpha again, open its index journal, ...

but such algorithm will probably have sense only after some improvements (related to the date column) are implemented:
in some further alpha-version

[void]

Thank you for your feedback adamantine,

The screenshots are helpful, thank you.

I have added index journal date sorting to Everything 1.5.0.1252a.
Does this version miss any changes?

moreover, currently (under such circumstances) there's not much sense in the id column for me (and unfortunately i cannot even disable it)

You can remove the #id: entry from your log format and resize this column to 1 pixel.
I will consider an option to hide the ID column.

in case i'm doing something wrong: is there any relatively simple algorithm that i may try to perform?

• Completely exit out of Everything (From the File menu, click Exit).
• I assume you are not using the Everything Service and you are running Everything as an admin -please correct me if I am wrong
• Replace your Everything.exe with the latest Everything 1.5 Alpha.
• Launch Everything.
• Everything should remember your run-in-background setting.
• Everything should remember your log-format.
  -Please let me know if this is not the case.

You can quickly opening your Everything.ini from Tools -> Options -> Config
Make any changes
Save and close notepad
Accept the prompt in Everything to reload your config.

Some ini settings you can change directly in the Everything search bar, for example:

• In Everything, type in the following search and press ENTER:

  Code: Select all

  /journal_log_format=#split:#date:<YYYY.MM.DD   hh.mm.ss>   #filename:      #action:
  

Thank you for your suggestions.

[adamantine]

Does this version miss any changes?

the 'file modify' action was certainly absent in index journal once (this action was related to a textual cue-file)
i suppose that more or less the same thing happened more than just once

You can remove the #id: entry from your log format and resize this column to 1 pixel

resizing is not remembered (even within the current session of ET-alpha)

I will consider an option to hide the ID column

this option could be useful

I assume you are not using the Everything Service and you are running Everything as an admin

i'm not using the Everything Service, i'm running Everything as an admin (windows-2003 with only 1 user)

Replace your Everything.exe

i didn't understood this part of your message: i have a regular 1.4-ET and also the lastest 1.5-alpha (1252)
so i'm not sure that i know how to comment these 2 items:
"Everything should remember your run-in-background setting"
"Everything should remember your log-format"

You can quickly opening your Everything.ini from Tools -> Options -> Config

Some ini settings you can change directly in the Everything search bar

i do such things in a more simple and intuitive way: i always edit the ini-file when ET is completely unloaded

are there any exact steps that i must do after getting a new alpha-version? (and before checking if its index journal works normally)

are these ini-lines correct?:
journal=1
journal_max_size=1640873984
journal_log=1
journal_log_directory=

are there any other ini-lines that i should edit? (so that index journal could become a substitution for that cenvo-rox program)


(added later):
index journal is sorted by date. but i noticed that content of the file index-journal-2021-04-04-1.5a.txt has wrong order of lines
look at the order of time (15.24.50 after 20.27.03) (15.24.42 after 20.27.00) in the following example:
2465   2021.04.04   20.27.03   C:\Program Files\Cenvo-Rox\data\cr_file_events.cdb-journal      Create
2466   2021.04.04   20.27.03   C:\Program Files\Cenvo-Rox\data\cr_file_events.cdb-journal      Delete
2467   2021.04.04   15.24.50   D:\Images\2021.04.04\2021.04.04   15.24.50.png      Create
2468   2021.04.04   15.25.00   D:\Images\2021.04.04\2021.04.04   15.25.00.png      Create
...
3313   2021.04.04   20.26.50   D:\Images\2021.04.04\2021.04.04   20.26.50.png      Create
3314   2021.04.04   20.27.00   D:\Images\2021.04.04\2021.04.04   20.27.00.png      Create
3315   2021.04.04   15.24.42   F:\New-4\Everything-1.5.0.1252a\Everything-1.5a.db.tmp      Create
3316   2021.04.04   15.24.42   F:\New-4\Everything-1.5.0.1252a\Everything-1.5a.db      Delete
3317   2021.04.04   15.24.42   F:\New-4\Everything-1.5.0.1252a\Everything-1.5a.db.tmp      Rename From
3317   2021.04.04   15.24.42   F:\New-4\Everything-1.5.0.1252a\Everything-1.5a.db      Rename To
3318   2021.04.04   15.24.42   F:\New-4\Everything-1.5.0.1252a\index-journal-2021-04-04-1.5a.txt      Modify

[adamantine]

this is the very fresh example of a situation when the 'file modify' action is absent many times (the action is related to a textual file 1.cue):

1) everything is good in cenvo-rox program (all 11 actually happened 'file modify' actions are present):

18:34:00, 18:34:00,   20:22:19, 20:22:19,   20:26:24, 20:26:24,   20:30:39, 20:30:39
20:34:39, 20:34:39,   20:40:25, 20:40:25,   20:45:08, 20:45:08,   20:53:34, 20:53:34
20:57:50, 20:57:50,   21:02:04, 21:02:04,   21:12:43, 21:12:43

i don't know why all these actions are duplicated (but i remember: this is not very unusual behavior for cenvo-rox)
at least these actions are not lost (unlike ET-alpha)


2) while index journal only has this 1 line (same file 1.cue):
18:34:00   (file modify)

[void]

Thank you for your reply adamantine,

resizing is not remembered (even within the current session of ET-alpha)

Everything will reset the column width to the default if you set the column width to 0 pixels to avoid columns from being "lost".
Instead of resizing the column to 0-pixels, try resizing the column to 1-pixel.
Does Everything remember the almost invisible column width?

i didn't understood this part of your message: i have a regular 1.4-ET and also the lastest 1.5-alpha (1252)

Are the Everything.exe 1.4 and Everything.exe 1.5 files in the same folder? -If so, please try moving them to separate folders. Otherwise, they might fight over the same Everything.ini.

-Please make sure Store settings and data is disabled for Everything 1.5 under Tools -> Options -> General.
-This will store your settings in Everything-1.5a.ini in the same location as your Everything 1.5 exe.

are there any exact steps that i must do after getting a new alpha-version?

Since you don't have the service installed, all you need to do is make sure Everything is not running (Right click the Everything System tray icon and click Exit) and overwrite your Everything.exe with the latest version.

are these ini-lines correct?:
journal=1
journal_max_size=1640873984
journal_log=1
journal_log_directory=

Yes, these are fine.
The journal_max_size is a little large (1564 MB).
Set journal_max_size to the maximum journal size in bytes.
For example, to set journal_max_size to 100MB:
journal_max_size=104857600

are there any other ini-lines that i should edit? (so that index journal could become a substitution for that cenvo-rox program)

Just the journal_log_format setting at this stage.
I plan to add more in a future release.

index journal is sorted by date. but i noticed that content of the file index-journal-2021-04-04-1.5a.txt has wrong order of lines

The lines written to the log are currently always in ID order.

this is the very fresh example of a situation when the 'file modify' action is absent many times (the action is related to a textual file 1.cue):

Is there a program running that has a handle open to this file?
-If so, Everything will only receive one "modify" notification when the file is first modified and another when all the file handles are closed.

Was Everything running in the background during this time?
If not, Everything may only log the last "modify" action, ignoring the earlier ones. (this is probably not the case here as only the earlier time is logged)

[adamantine]

try resizing the column to 1-pixel ... Does Everything remember the almost invisible column width?

i tried 1 pixel - it works (but looks ugly)
it's better for me to wait for the implementation of this: "I will consider an option to hide the ID column"

moreover, the current situation (in index journal) leads me to a thought that the 'export' button may become useless for me too
because ET-alpha automatically adds the actions to a file like this: index-journal-2021-04-09-1.5a.txt (such file is created once a date)
which seems to be more or less appropriate behavior (at least at this stage)
i have not got used to it yet. so i will continue to think about it (and experiment with it)

and seemingly, the very important thing right now is to wait for the change in this aspect:
"The lines written to the log are currently always in ID order"
i hope that it's possible to change this approach to a better variant:
the lines written to the log are always in the order which is present in main window of index journal
(but the order by date could be quite enough: when the log starts from the old actions and continues with the newer actions)

Are the Everything.exe 1.4 and Everything.exe 1.5 files in the same folder?

of course no (they are even in the different drives)

make sure Store settings and data is disabled for Everything 1.5 under Tools -> Options -> General

it is always disabled

make sure Everything is not running

i always try to pay attention to things like this

Right click the Everything System tray icon

i don't have the system tray (thanks to the 'group policy'). and it feels great :)

a bit of off-topic: i cannot find a reliable (and visually ideal) way to get rid of the windows start button
currently i use the 'StartKiller' utility which does its job wonderfully (the 3 previously uploaded screen-shots show my task-bar completely)
but StartKiller requires an exe-process to be constantly running
i wish i had a (sophisticated) piece of the ahk-code to achieve the same effect (but without usage of this utility)

journal_max_size=1640873984 ... The journal_max_size is a little large (1564 MB)

after i experimentally entered something like 999999999999, ET-alpha itself gave me this number: 1640873984
so i thought that this is the largest number that's possible
this aspect is not that important for me but anyway: are you sure that 1640873984 is not the largest option?

Is there a program running that has a handle open to this file?

it seems that the answer is 'yes'
the program is notepad2, and that 1.cue file was constantly opened
during several hours the workflow was:
edit something in the file, save it, the file stays opened, (slightly later) edit something in the file, save it...

so i guess that cenvo-rox can handle (understand) such modifications, while ET-alpha can't do it
is it because cenvo-rox has its service installed? is it because cenvo-rox has its process constantly running? (CRFileMonSvr.exe)

but these questions are not that essential to me because it looks like i will have to get used to this:
"Everything will only receive one "modify" notification when the file is first modified"
or to forget about a dream to substitute the cenvo-rox (in case i eventually fail to get used to such numerous new things/changes)


at first, these your words ("and another when all the file handles are closed") were not clear to me
because in that '11 actions' example: there was just 1 action (not 2 actions) related to the 1.cue file (in ET-alpha)
but later i realized that the file is still not closed
so after it will be closed, i may expect the confusing observation:
the ET-alpha displays the just added 'modify' action (for a file which was actually modified several days ago)
it looks like i will have to live with such behavior of ET, or forget about the whole idea of substitution (cenvo-rox > ET 1.5)
since i really don't want to forget, i plan to continue considering all the possibilities to switch to ET 1.5 (when it becomes beta-version)

Was Everything running in the background during this time?

no

[void]

Thank you for the feedback adamantine,

after i experimentally entered something like 999999999999, ET-alpha itself gave me this number: 1640873984
so i thought that this is the largest number that's possible
this aspect is not that important for me but anyway: are you sure that 1640873984 is not the largest option?

The largest supported value is:
4294966272

so i guess that cenvo-rox can handle (understand) such modifications, while ET-alpha can't do it
is it because cenvo-rox has its service installed? is it because cenvo-rox has its process constantly running? (CRFileMonSvr.exe)

Everything will use the USN Journal to monitor changes to NTFS volumes.
My guess is CRFileMonSvr uses ReadDirectoryChanges.

The USN Journal will record two log entries:
one when the file is first modified and
another one when all file handles are closed.

ReadDirectoryChanges will periodically send notifications about changes to files.

If you would like to use ReadDirectoryChanges instead of the USN Journal, use folder indexing instead of NTFS indexing.
However, you would need to keep Everything running. Otherwise, it will miss changes.

[adamantine]

i noticed 2 negative aspects. example of the first aspect:

- i had not been opening the ET-alpha for the previous several days
- the recent log was in the file index-journal-2021-04-11-1.5a.txt:
7 500 lines, 835 KB (modified: 2021.04.11   03.11.45)
start of first line: 318856   2021.04.10   22.16.20
start of last line: 326018   2021.04.11   03.11.45

- a few hours ago i opened the ET-alpha
- the new file index-journal-2021-04-14-1.5a.txt appeared:
183 000 lines, ~17-18 MB (i deleted many-many files yesterday but as i understand: it's not important in this example)
start of first line: 326019   2021.04.11   03.11.45
start of the line which is before the last line: 505379   2021.04.14   01.23.54
but the last line was truncated. it just looked like this:
505380   

anyway, this truncation turned out to be harmless
because the file index-journal-2021-04-14-1.5a.txt was correctly continued after i closed the ET-alpha
(i mean that the line 505380 got its full content, and the subsequent lines were added correctly too)

i don't know the reason why such truncation happened
it's a bad thing of course but at least nothing was lost


but there's the second negative aspect, and it's much more essential
without a doubt i may confirm that there were several missing 'modify' actions
they are again related to textual cue-files

but this time the log-file (and index journal too) do not contain a single line concerning the 2 cue-files (at least 2):
- the first 'modify' action is absent for both files (this is the first modification of a cue-file)
- the second 'modify' action is absent for both files too (this is the 'modify' action which should be added when all the file handles are closed)

- these 2 cue-files have equal name (1.cue), and the cenvo-rox displays:
2021.04.14   02:27:49   1.cue   Modify
2021.04.14   02:28:06   1.cue   Modify

while the ET-alpha has the complete absence of these 'modify' actions
(these 2 files were closed somewhere around 02:29:00)
(i checked the log and the index journal somewhere around 12:00:00)

use folder indexing instead of NTFS indexing

at the moment i'm totally sure that i will always use NTFS indexing
moreover, my initial interest (for the upcoming new 'content indexing' feature) seems to be gone
because i realized that i always need to have any search result in a most actual state
and if i understand correctly, the 'content indexing' feature cannot guarantee it

[horst.epp]

...
moreover, my initial interest (for the upcoming new 'content indexing' feature) seems to be gone
because i realized that i always need to have any search result in a most actual state
and if i understand correctly, the 'content indexing' feature cannot guarantee it

I use content indexing a lot and whenever I update some of the documents which are indexed
the results are almost instant available if I search for content.
So what is your specific problem with not actual content index ?

[adamantine]

(off-topic starts)

horst.epp
i have not checked the "content indexing" feature extensively yet
that's why i wrote "if i understand correctly". which means "i'm not sure"

the reason for that thought is: i used to think that index is something that adds speed but lacks relevance
if this is not the case with the ET's new "content indexing" feature, then probably it's a positive news for me

but anyway it does not look like an ideal feature for me just because of this thing:
"whenever I update some of the documents which are indexed"
these words literally mean to me:
- i always have to remember what is indexed and what is not (and it's impossible to do, generally)
- i always have to remember when it was indexed
- all new files will be automatically out of this feature
(what if i create several hundreds of files during a day? of course, they all will not be auto-indexed, right?)


generally speaking, this is 1 of the biggest problems for me (when using the OS and the ET program):
1) very often i have to manually re-calculate the size of all folders (20 TB) in all my 5 hdd's
because otherwise i will have to deal with some kind of extra-slowness in some/many moments of my file management
to avoid it, from time to time i perform such re-calculation when the OS is in idle state
(windows server 2003 is a good OS, so there's always a chance to have it in idle state, unlike lousy win-10/win-8...)

sometimes the whole 20 TB are re-calculated within ~3-5 seconds, sometimes it takes ~1-18 minutes
i still don't know what is causing such a huge difference
for example, if i download 5-6 GB, it's an almost total guarantee that the next re-calculation will not be fast
while the downloaded 2-3 GB do not affect the speed of re-calculation usually
i suppose that there's something like an inner system limit (related to some 'file/folder' system index)

the bad thing is that i cannot predict when i should launch the re-calculation, and when i should not do it
so i launch it either just in case or when i encounter an obvious slowness
(for example, when opening a folder with hundreds of sub-folders/files)

- re-calculation is always needed right after the load of OS
- sometimes re-calculation may not be needed during 10/20 or even 30/40 hours
- sometimes re-calculation may be needed after just 1 hour (or even after smaller amount of time)
- re-calculation is a thing that i would like to forget for good (but i can't)

to re-calculate, i use the following template (it's bookmarked in my ET 1.4):
infolder:C:|infolder:D:|infolder:E:|infolder:F:|infolder:G: !attrib:H
then i manually lasso-select all folders and press the keyboard button "Pause" (it means Alt+Enter in my system)
instead of lasso-selection, i may press "Tab" and "F4" (F4 means Ctrl+A in my system)

to speed everything up, i may divide the lasso-selection into 3 parts: drives C, D, E (then huge drive F) (then huge drive G)
this new approach gives me 3 times faster re-calculation (since all 3 parts are nearly equal: ~6-7 TB)


2) quite often i have to manually load the following template (it's also bookmarked in my ET 1.4):
G:\PO\|D:\FM\|F:\NA\|     ...[many other folders]...     E:\LP\|C:\OM\|G:\PM\     ext:cue;txt     content:"×"
the × symbol is totally absent everywhere in my file system
it is necessary to do such "empty" search so that later (during some minutes/hours) i had some "non empty" content search in an instant way

if somebody knows a better approach/approaches, please let me know
(i'm not considering solid state drives, that's understood) (because i care about safety of files)

(off-topic ends)

(thinking out loud) i'm eagerly waiting for this: "The lines written to the log are in date order (from old to new)"
because currently it's difficult to test/view the content of log-files

[horst.epp]

(off-topic starts)

horst.epp
i have not checked the "content indexing" feature extensively yet
that's why i wrote "if i understand correctly". which means "i'm not sure"

the reason for that thought is: i used to think that index is something that adds speed but lacks relevance
if this is not the case with the ET's new "content indexing" feature, then probably it's a positive news for me

but anyway it does not look like an ideal feature for me just because of this thing:
"whenever I update some of the documents which are indexed"
these words literally mean to me:
- i always have to remember what is indexed and what is not (and it's impossible to do, generally)
- i always have to remember when it was indexed
- all new files will be automatically out of this feature
(what if i create several hundreds of files during a day? of course, they all will not be auto-indexed, right?)

You don't have to remember what is indexed and also new files are of course indexed.
I think you should spend some time to learn the basics of content indexing.
You just specify the path of the folders and what type of files should be indexed.
The rest is of course fully automatic and you have nothing to remember.

For viewing of log files I would not use Everything but one of the available log file viewers
which do a tail operation on such files, WinTail for example
https://sourceforge.net/projects/wintai ... t/download

[adamantine]

spend some time to learn the basics of content indexing

spending the time ended very quickly:

.\src\mem.c(1049): mem_alloc(): Fatal error: out of memory 4294967295

right now it's not a huge problem since i'm not planning to use the 1252-alpha version in the future
but anyway, nothing helps to restore the normal functioning of this alpha version:

- including the disabling of the "content indexing" feature in options
(ET still allows me to run the main menu items)
(ET does not allow me to perform any search: it just indicates "Loading..." in status bar)
(that error-window is present all the time) (if i close the error-window, ET crashes)

- including the editing of these lines in ini-file:
content_include_only_folders
content_include_only_files=*.cue;*.txt
content_max_size=0
journal_max_size=4294966272

For viewing of log files I would not use Everything

for viewing of log files i use notepad2 (Everything is for creating of log files)

[tuska]

.\src\mem.c(1049): mem_alloc(): Fatal error: out of memory 4294967295

Are you still using x86.portable?
If so, I would try x64.portable.

[adamantine]

Are you still using x86.portable?

yes. there was info about it (this thread, the first message, the first line), and nothing was changed since then

the special (but kind of sad) aspect is that i nearly always have very-very much of free ram
(my OS is 32-bit but it also supports PAE so it's possible to have at least 32 gb of ram)

If so, I would try x64.portable

i never tried x64 (because i don't use the x64 OS)
but i just tried to launch it, and here's the result:

The image file F:\New-4\Everything-1.5.0.1253a.x64\Everything64.exe is valid, but is for a machine type other than the current machine

[horst.epp]

Are you still using x86.portable?

yes. there was info about it (this thread, the first message, the first line), and nothing was changed since then

the special (but kind of sad) aspect is that i nearly always have very-very much of free ram
(my OS is 32-bit but it also supports PAE so it's possible to have at least 32 gb of ram)

If so, I would try x64.portable

i never tried x64 (because i don't use the x64 OS)
but i just tried to launch it, and here's the result:

The image file F:\New-4\Everything-1.5.0.1253a.x64\Everything64.exe is valid, but is for a machine type other than the current machine

Sorry but of course an x64 app can't run in a 32bit OS.
Also PAE doesn't help in this case for the memory requirements of the Everything database.
Its wasting your time and also Voids to search for all type of problems in this environment.

[void]

Thank you for your feedback adamantine,

but the last line was truncated. it just looked like this:
505380

Sounds like Everything crashed? with no crash dialog? was Everything terminated?

without a doubt i may confirm that there were several missing 'modify' actions

If you search for this file in Everything, does it have the correct date modified timestamp?

Debug logs might help:

• In Everything, from the Tools menu, under the Debug submenu, click Config.
• Change the following line:
  debug_log=0
  to:
  debug_log=1
• Save changes and exit Notepad.
• Accept the prompt in Everything to reload your configuration.
• The next time you notice Everything missing a change, could you please check your %TEMP%\Everything Debug Log.txt for any mention of the missing file. There should be several lines starting with USN <action> <file.cue>

If Everything is crashing, the last few lines in the debug log might help determine the issue.

Debug logs can grow very large.
To disable debug logging:

• In Everything, from the Tools menu, under the Debug submenu, click Config.
• Change the following line:
  debug_log=1
  to:
  debug_log=0
• Save changes and exit Notepad.
• Accept the prompt in Everything to reload your configuration.

spending the time ended very quickly:

.\src\mem.c(1049): mem_alloc(): Fatal error: out of memory 4294967295

This is a bug.

Could you please send me a mini crash dump when this dialog is shown.

[adamantine]

Sounds like Everything crashed?

certainly no

with no crash dialog?

there were no crash dialogs

was Everything terminated?

no, ET was working as it should
besides that, there was 1 more time when similar truncation happened:

1) a line contained more info than just id
2) the text was cutted somewhere in the middle of the line (somewhere inside a word)
3) i noticed this 2nd occurence during several minutes/hours (after the 1st occurence took place)
(i don't remember the exact time anymore - it was a few days ago)

If you search for this file in Everything, does it have the correct date modified timestamp?

yes, those 2 files have the correct date modified timestamps
as i mentioned earlier, cenvo-rox correctly displays that timestamps:
2021.04.14 02:27:49
2021.04.14 02:28:06
my regular ET 1.4 also displays these very timestamps

The next time you notice Everything missing a change, could you please check your %TEMP%\Everything Debug Log.txt

ok, i'll check it if i notice Everything missing a change. but i cannot promise any news here because:

- nobody knows how to reproduce that missing (so i may either notice it once again or not)
- the 1252-alpha was deleted after those numerous attempts to restore its working state (it became not usable at all)
- i haven't tried any newer alpha-versions because there's nothing interesting in them for me
- i plan to try reproducing the bug in some further alpha-version, and post here the info in case of successful reproduction

[adamantine]

i tried to customize the options and the content of ini-file (of a new alpha-1255) to a state which was present in my previous alpha-1252
the result is: that fatal error does not appear


i still cannot continue testing the index jounal because of completely useless content of its log-file/files (like index-journal-2021-04-23-1.5a.txt)
it sorts all actions by id which makes no sense if there are 2 (or more) drives
(cenvo-rox has no problems here but it is a problem for ET for some reason)
(so the only way out is to use sort by date, not by id: otherwise any logging is nearly unreadable/unusable)

all actions/timestaps (of all drives) are added to the log-file interchangeably
which makes no sense if there is a wish to understand something in those thousands of lines (of the textual log-file)

the solution is: a new option allowing to choose the 'sort by date' instead of current 'sort by id'
somehow i thought that such an option is easy to implement. maybe i was wrong
anyway i will be hoping to see it in the next alpha

[void]

Thanks for the feedback adamantine,

the solution is: a new option allowing to choose the 'sort by date' instead of current 'sort by id'
somehow i thought that such an option is easy to implement. maybe i was wrong
anyway i will be hoping to see it in the next alpha

It's difficult to do if Everything is not running.
I would have to buffer every single change since Everything was last run, sort by date and then add to the journal.
For this to be feasible you would need to run Everything in the background.

Everything missing a change

Everything will not add changes to the journal if the database is not changed.

I have added a journal_add_all ini setting to Everything 1.5.0.1257a.
When enabled, Everything will always add all actions to the journal, even when there is no change to the database.

To enable this setting:

• In Everything, type in the following search and press ENTER:
  /journal_add_all=1
• If successful, you should see journal_add_all=1 in the status bar after a few seconds.

Please let me know if Everything is missing any changes in your index journal with this setting enabled.

[adamantine]

Please let me know if Everything is missing any changes in your index journal with this setting enabled

ok (i enabled this setting)

It's difficult to do if Everything is not running

i understand. thank you for the attention to this thread
maybe this is the end of index-journal/ET-1.5 for me (unless some new good solution appears)

because such current logging has no value/sense (i'm speaking about the order in the time-column):

1519   2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Create
1520   2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Modify
1521   2021.04.30   15.44.14   E:\SP\C\folder1a      Rename From
1521   2021.04.30   15.44.14   E:\SP\C\folder2bb      Rename To
1522   2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Create
1523   2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Modify

it's bearable in case of just 6 lines (taken for this example)
but it's not acceptable/readable/usable in case of thousands of lines

[adamantine]

1) i decided to use the ET 1.5 because of:
- new 'content indexing' option (too fast to be just neglected in favor of good old but slow ET 1.4)
- new useful columns: length, audio bits per sample, audio sample rate, audio channels, full path length, child count
- the 'options' sub-window (and other sub-windows too) is now resizable which means 'much more convenient'
- index journal. it could have been a killer feature for me but its current state is not that great/promising
anyway i have some additional thoughts about it:

i'm considering a possibility to use the index journal despite all related negative aspects
(the absence of 'sort by date' is the strongest chagrin today)
now i have this ini-line:
journal_log_format=#split:#id:   #date:<YYYY.MM.DD   hh.mm.ss>   #filename:      #action:
the experimental idea is to remove the #id part (unfortunately it turned out to be useless in any case)

then the log file (it has the txt extension) contains something like this:

2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Create
2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Modify
2021.04.30   15.44.14   E:\SP\C\folder1a      Rename From
2021.04.30   15.44.14   E:\SP\C\folder2bb      Rename To
2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Create
2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Modify

such textual content is still not usable (as i already mentioned in my previous message)
but notepad2 has the 'sort lines' main menu item
there it's possible to choose the 'sort ascending' setting
so all selected lines may be manually sorted ascending (as if they are sorted by date-time):

2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Create
2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Modify
2021.04.30   15.44.14   E:\SP\C\folder1a      Rename From
2021.04.30   15.44.14   E:\SP\C\folder2bb      Rename To
2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Create
2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Modify

it's not a good solution but at least it's something

is ET capable of doing it automatically? (always sort ascending all the lines of a log file)
if yes, could this be added as an ini-tweak?

this is not a good solution, particularly because the names of actions may be sorted improperly
(in case when these actions happened during the same second)
- for example Rename To may appear before the Rename From
- this disadvantage is big but maybe it's bearable, i'm not sure yet
- is adding the milliseconds a nice/partial way out? for instance:

2021.04.30   15.44.14.008   E:\SP\C\folder22      Rename From
2021.04.30   15.44.14.990   E:\SP\C\folder11      Rename To

 
2) there's an ini setting journal_log_directory=
in that folder: the txt-files are created/modified (the files like this: index-journal-2021-05-06-1.5a.txt)
i ask you to add a new ini setting. it may be named journal_log_name_format=
so that it was possible to edit this line to: journal_log_name_format=YYYY.MM.DD
and thus obtain the 2021.05.06.txt instead of index-journal-2021-05-06-1.5a.txt

 
3) it seems that my workflow will imply/demand a frequent/constant manual launch of the very recent log-file
so it's necessary to always have it as a file with the definite path-name (for example like this: D:\Programs\ET\index-journal\recent.txt)

is it possible to implement such an approach at all?
if yes, the recent.txt should be later auto-renamed by ET (when a newer log file appears)
then this newer file gets the recent.txt name
while the renamed file should get the name with a corresponding date (for example 2021.05.16.txt)

[aviasd]

but notepad2 has the 'sort lines' main menu item
there it's possible to choose the 'sort ascending' setting
so all selected lines may be manually sorted ascending (as if they are sorted by date-time):

2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Create
2021.04.30   15.44.00   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Modify
2021.04.30   15.44.14   E:\SP\C\folder1a      Rename From
2021.04.30   15.44.14   E:\SP\C\folder2bb      Rename To
2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Create
2021.04.30   15.45.40   D:\Images\2021.04.30\2021.04.30   15.45.40.png      Modify

it's not a good solution but at least it's something

It sounds like if everything would have an option to export the index journal from the command line, you could get what you want.
I.E: export the csv to a known filename and then run a powershell (or gnu sort) to sort it :

Code: Select all

Import-Csv '.\Index Journal.csv' | Sort-Object date | ft

Or

Code: Select all

Import-Csv '.\Index Journal.csv' | Sort-Object date | export-csv  -NoTypeInformation "Sorted journal.csv"

For now the index has to be exported manually.
I know it's a workaround but it's something.
Note: With powershell you could do some more advanced filtering for faster narrowing:

Code: Select all

Import-Csv '.\Index Journal.csv' | Sort-Object date | where { $_.path -match "app" -and $_.name -match "m
odify" }  | ft

Note2 When exporting the index journal (with default settings), there's an additional comma between the date column and the action column resulting in incorrect parsing of column header (example above is presented with current issue) :

ID,Date,Action,Name,Path,New Name,New Path,Size
2539533,2021-05-06 11:49:08,,File Delete,sync_config.db-wal,C:\Users\...\user_default,,,

This causes an issue with parsing the action under the correct header ( action becomes Name in the above query), and so on - all the columns for the file are shifted to the right

+1 for milliseconds logging

Note3: You can format the index journal to have milliseconds using /time_format=HH.mm.ss.SSS but that applies to *all* time formats in everything which is not granular enough. ( and there are still events happening at the same millisecond, so ID would need to be used anyway )

Note4 (Final)
You could also use powershell to have exact date sorting by casting the date column as datetime, then you can query by date range:

Code: Select all

Import-Csv '.\Index Journal.csv' | foreach { $_.date = $_.date -as [datetime]; $_ } |                    
where { $_.date -ge "05/06/2021 11:38" -and $_.date -le "05/06/2021 11:40" -and $_.path -match "app" } | 
Sort-Object date,id | ft

[adamantine]

aviasd, thank you for participating in this thread

using /time_format=HH.mm.ss.SSS but that applies to *all* time formats in everything

luckily we don't have to change the time_format=
so i continue to have it this way: time_format=HH.mm.ss

instead: i found out that adding the milliseconds is already possible
journal_log_format=#split:#date:<YYYY.MM.DD   hh.mm.ss.SSS>   #filename:      #action:
unfortunately it brings nothing good. even the hh.mm.ss.SSSSSSSS brings nothing good
several small tests showed that all Rename From/Rename To share the same hh.mm.ss.SSSSSSSS
so milliseconds are not the way out at all

maybe i will add the .SSS to the journal_log_format= (in the future)
it looks not that good (and it has not much sense), so i have no final dicision at the moment

 

so ID would need to be used anyway

(i suspect that) not in my case

export the index journal

things are partially/slightly different comparing to the beginning of this thread:
exporting is not used by me anymore (ET itself generates the new log files, and gives them a corresponding dates)

all i have to do is to launch the ET, and then close it (i may do nothing inside the ET)
this is not convenient scheme/approach but this is the only way to get the log file updated (which i periodically need, actually)

 

the command line

a powershell (or gnu sort)

if possible, i would have preferred to not use all that complicated (for me) stuff

the csv

the txt will always be used in my case

[adamantine]

this is how a part of the log file may look after the manual sort ascending (done in notepad2):

2021.05.07   01.40.38   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db      Delete
2021.05.07   01.40.38   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db      Rename To
2021.05.07   01.40.38   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Create
2021.05.07   01.40.38   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Modify
2021.05.07   01.40.38   F:\New-4\Everything-1.5.0.1257\Everything-1.5a.db.tmp      Rename From

2021.05.07   01.40.39   D:\Images\2021.05.07\2021.05.07   01.40.39.png      Create
2021.05.07   01.40.39   D:\Images\2021.05.07\2021.05.07   01.40.39.png      Modify
2021.05.07   01.40.39   F:\New-4\Everything-1.5.0.1257\index journal\index-journal-2021-05-07-1.5a.txt      Modify
2021.05.07   01.40.49   D:\Images\2021.05.07\2021.05.07   01.40.49.png      Create
2021.05.07   01.40.49   D:\Images\2021.05.07\2021.05.07   01.40.49.png      Modify

2021.05.07   01.43.16   C:\WINDOWS\Temp\~DF676D.tmp      Delete
2021.05.07   01.43.16   C:\WINDOWS\Temp\~DF676D.tmp      Modify
2021.05.07   01.43.16   C:\WINDOWS\Temp\~DF676D.tmp      Modify
2021.05.07   01.43.16   D:\Programs\Xyplorer (Portable)\Data\action.dat      Create
2021.05.07   01.43.16   D:\Programs\Xyplorer (Portable)\Data\action.dat      Delete
2021.05.07   01.43.16   D:\Programs\Xyplorer (Portable)\Data\action.dat      Modify

2021.05.07   01.44.08   C:\Documents and Settings\Administrator\Desktop\z.txt      Modify
2021.05.07   01.44.08   D:\Programs\U-Torrent (Portable)\resume.dat      Rename From
2021.05.07   01.44.08   D:\Programs\U-Torrent (Portable)\resume.dat      Rename To
2021.05.07   01.44.08   D:\Programs\U-Torrent (Portable)\resume.dat.new      Create
2021.05.07   01.44.08   D:\Programs\U-Torrent (Portable)\resume.dat.new      Modify
2021.05.07   01.44.08   D:\Programs\U-Torrent (Portable)\resume.dat.new      Rename From
2021.05.07   01.44.08   D:\Programs\U-Torrent (Portable)\resume.dat.old      Delete
2021.05.07   01.44.08   D:\Programs\U-Torrent (Portable)\resume.dat.old      Rename To

date-time appear to be perfect but the price is very high:
- a necessity to do the manual uncomfortable slow sort (of numerous lines inside the log file)
- and what is worse: an awful and confusing look/order of actions
in the example above: the actions in bold are placed in not normal order

 
the "notepad2 - main menu - sort lines - sort ascending" command works very slow
example: 20 000 textual lines are sorted during circa 30-40 seconds (my processor is i7-4930k, 6 cores)

this is the additional reason why i'm asking to add a new feature to the index journal:
a possibility to compile a list of folders/files that are excluded from index journal
both from its main window and especially from the log file

but the better implementation is to let the user choose among the 4 possibilities:
- the current option (all folders/files are included)
- a new option: exclude the chosen folders/files from main window (and from the log file)
- a new option: exclude the chosen folders/files from main window (but not from the log file)
- a new option: exclude the chosen folders/files from the log file (but not from main window)

there's already the 'temporary excludes' option but it's related to the ET main window only

[adamantine]

I have added a journal_add_all ini setting to Everything 1.5.0.1257a.
When enabled, Everything will always add all actions to the journal, even when there is no change to the database

(in the recent 2 weeks since your last reply) i haven't noticed any positive or negative change after enabling this setting
i still use the 1257-alpha

instead i noticed that the 'modify' action is really not very informative
i'm not sure if it's the new 'journal_add_all' ini-setting or not (i lean towards the view that this setting is not the reason)
but there are cases when index journal displays several 'modify' actions for several files
(while in reality those files have quite different modified state)

so the check-boxes (separately for each action in index journal) seem to be the real necessity
because currently the 'modify' action is not very useful (but rather confusing)
 
 

20 000 textual lines are sorted during circa 30-40 seconds

that was the early example of low speed (when manually sorting the lines inside notepad2)
1 further case was even more pessimistic: ~30.000 textual lines are sorted during ~90 seconds

[void]

Thanks for the feedback adamantine,

I will consider an exclude list for the Index Journal.
Thank you for your suggestion.
Please keep in mind the index journal's main purpose is to provide an audit of the Everything database.

I'm still thinking about sorting the index journal log by date.
There are some problems to overcome.. For example: how to handle adding an external drive to your index that was modified externally. (you will inevitability end up with log items with dates out of order)

A log viewer in Everything might be an option (it would provide fast date sorting)
I would like to make another program that manages creating/viewing logs.

but there are cases when index journal displays several 'modify' actions for several files
(while in reality those files have quite different modified state)

Everything will gather the file size, date modified, attributes, etc... when Everything is running and the change is detected.
This means the file size, date modified, attributes etc... will not be known until Everything is running.
In other words, this information is not stored in the USN Journal.
If there is many changes to a file, and Everything is not running, when you start Everything you will see a bunch of changes with all the same file sizes, date modified, attributes etc...

so the check-boxes (separately for each action in index journal) seem to be the real necessity

Experimenting with checkboxes for each action and a checkbox files/folders is on my TODO list.
Thanks for the suggestions.

[aviasd]

Note2 When exporting the index journal (with default settings), there's an additional comma between the date column and the action column resulting in incorrect parsing of column header (example above is presented with current issue) :

ID,Date,Action,Name,Path,New Name,New Path,Size
2539533,2021-05-06 11:49:08,,File Delete,sync_config.db-wal,C:\Users\...\user_default,,,

@void
Could you please confirm/validate the issue described above?

( To recreate, export as csv and open in excel )

[void]

Note2 When exporting the index journal (with default settings), there's an additional comma between the date column and the action column resulting in incorrect parsing of column header (example above is presented with current issue) :

This will be fixed in the next alpha update.

[void]

Everything 1.5.0.1262a removes the extra , when exporting as CSV.

[aviasd]

Everything 1.5.0.1262a removes the extra , when exporting as CSV.

Confimed.

[adamantine]

I'm still thinking about sorting the index journal log by date.
There are some problems to overcome.. For example: how to handle adding an external drive

et creates these txt-files:
...
index-journal-2021-06-26-1.5a.txt
index-journal-2021-06-27-1.5a.txt
index-journal-2021-06-28-1.5a.txt
...

is it technically possible to add a simple ini setting so that et began to sort these txt (log) files ascending?
(before saving a txt file on disk)
(for those who don't have any external drives, and don't plan to have them)
(it will be a partial solution that improves the current usage of index journal)

 

I will consider an exclude list for the Index Journal

such option seems to be simple but quite useful
it may simplify/lessen the txt-files a lot
these days: each of them occupies 2-5 mb (it's a small size but there could be hundreds of such files)
besides that: there could be some not needed folders that generate thousands of useless lines inside those txt (log) files

 

A log viewer in Everything might be an option (it would provide fast date sorting)
I would like to make another program that manages creating/viewing logs

it's hard to say now if this option is a great idea. many aspects depend on the way it's implemented
are there any news concerning this possible log viewer?

[void]

Thanks for the feedback adamantine,

Everything writes to the log on disk as the changes are detected (one volume at a time).

The log is never entirely in memory.
Sorting would require the entire log to be memory.

For now, please try a third party tool to sort the log (Does Notepad2 sort?)
I'll consider having Everything sort the logs after the end of each day.

I will consider an exclude list for the Index Journal

This is still on my TODO list.

are there any news concerning this possible log viewer?

None yet, sorry.
Everything 1.5 stability issues have my current priority.