Date-modifieddupe: or similar, but ignore x amount of time

[Herkules97]

I don't know why, but Teracopy doesn't copy folders correctly. They are often a few..microseconds? off. Whatever the last 3 numbers for NTFS are.
Take this folder..
Original date created: 2023-03-10 06:27:11:1345075
Copied date created : 2023-03-10 06:27:11:1345168
This then marks it as non-duplicate, making it hidden when trying to find duplicate data.
Maybe there is a robocopy-type program that works like Teracopy, as in isn't tediously cmd line, but I like how Teracopy works mostly. Unless it is similar enough, probably won't use it.
Better to be able to disable reading the entire time.
Alternatively finding out how to make Teracopy not fail copying time.

Alternatively to those, I could use other methods for finding duplicates, but size, name, date created and date modified combined seems to do a good enough job.
I suppose maybe size, name and a hash could work instead. Depends on how big indexing a hash is, even if I doubled RAM I already run at around 80% at all times. 20% available on current PC is still more than 30% available on previous computer..So I can still do some thing else. But not if I have to increase RAM on EBV much.

[NotNull]

I don't know how Everything handles this internally, but some tolerance might be useful anyway as different filesystems handle times diffeently.
Some filesystems have a precision? accuracy? resolution? (can't think of the right word atm) of 2 seconds, whereas your C-drive uses 100 nanosecond intervals. And some filesystems do not support daylight saving times and can be off by 1 hour.

Most of the time this is not a big deal, but time-duplicates can be a challenge indeed.


I think you should try to contact Teracopy too as this behaviour is unexpected (and unwanted IMO).
Do you use verifying after files are copied? Maybe something "happens to the files" during that stage?

[therube]

What is giving you the times that you are displaying?
What comparison program is then not comparing these directories as identical?

And if Everything, what is your Search?


(Your subject: says "modified", but your text says "created".)

[Herkules97]

What is giving you the times that you are displaying?
What comparison program is then not comparing these directories as identical?

And if Everything, what is your Search?


(Your subject: says "modified", but your text says "created".)

The times are from this program, the comparison is from this program, the search is the first "word" in the title, using this program.
Yes, I use modified then created. I did say "or similar".

[Herkules97]

I don't know how Everything handles this internally, but some tolerance might be useful anyway as different filesystems handle times diffeently.
Some filesystems have a precision? accuracy? resolution? (can't think of the right word atm) of 2 seconds, whereas your C-drive uses 100 nanosecond intervals. And some filesystems do not support daylight saving times and can be off by 1 hour.

Most of the time this is not a big deal, but time-duplicates can be a challenge indeed.


I think you should try to contact Teracopy too as this behaviour is unexpected (and unwanted IMO).
Do you use verifying after files are copied? Maybe something "happens to the files" during that stage?

Well it's only the folders, I don't use verification.
I was thinking it was possible to change the display itself if that affected it, but removing the last three numbers didn't change it.

I was thinking something like date-moddifieddupe;len:-3 or something to remove the last 3, -4 to remove last 4 from consideration and so on.

I just did a test, copied one folder to another, then copied from that to another. All 3 on different devices. The timestamp is kept.
So it seems it only alters from original folders for some reason.
Too many folders to undo it in the future, but ye I guess if Codesector has a support section I could try that.

I would still need to exclude the 3 last numbers for existing folders and there are a lot of them.

https://ideas.codesector.com/p/timestam ... t-original
I made the post, but some of the stuff on there looks to be a year or older..So maybe it will never be fixed.

[dedupeit]

Code: Select all

 add-column:a a:=$dc:/1000 dupe:columna

This takes the data created and divides it by 1000 (removing the last 3 digits of the timestamp). It sticks it in a new column A. And then it looks for dupes based on that new column. Is this what you're looking to do?

[Herkules97]

Code: Select all

 add-column:a a:=$dc:/1000 dupe:columna

This takes the data created and divides it by 1000 (removing the last 3 digits of the timestamp). It sticks it in a new column A. And then it looks for dupes based on that new column. Is this what you're looking to do?

Not sure, if it's a column would it even work? The command I use is sizedupe: namedupe: date-createddupe: date-modifieddupe:..
It's my mistake not including that, I must've written it in a different post recently and presumed it was here..Or it was in an edit I never sent as I do edit a lot. This post alone has taken 1 hour and 11 minutes .
And more I've forgotten is that the 4 dupe checks work at the same time. A lazy check showed 12,000 files with only the first 3, 11,000 files with all 4.
Just to be certain I am not using 4 of them and only sizedupe: is working. I use sizedupe first because it sorts it by size which is what I would do anyway. There is sort:size or whatever, but sizedupe: is 2-in-1.

The above command also happens to work for .efu exports too, which I make for archives.
So if I wanted to find duplicates across both live locations and archives' contents and I bothered setting up a db for that, I can use it there too.
This was more to my own message about trying hash instead..It wouldn't work, at least not for any .efu up to this point and to save space and tedium I probably don't want much more data in an .efu anyway. Especially if I want to dump all current .efu exports into a db..I did that once, but I have way more exports today than 2-3 years ago and not much more RAM available because I spent it all already.

If I haven't convinced anyone so far, then it's a bust as I've got no more to give I don't think.
It's also only an issue with folders, that I don't even de-dup. I don't know what goal I had with this post, maybe I knew something last morning that I wanted it for..Or maybe I just wanted to bring attention to a possible addition to the software if Void wants to add it.

[dedupeit]

So your goal is to find duplicates based on name, size, date created (with some tolerance) and date modified (with some tolerance). If so, then this should work.

Code: Select all

add-column:a a:=$dc:/1000 dupe:columna add-column:b b:=$dm:/1000 dupe:columnb dupe:name,size,columna,columnb

[Herkules97]

So your goal is to find duplicates based on name, size, date created (with some tolerance) and date modified (with some tolerance). If so, then this should work.

Code: Select all

add-column:a a:=$dc:/1000 dupe:columna add-column:b b:=$dm:/1000 dupe:columnb dupe:name,size,columna,columnb

Ye, it did work for one db that is smaller in size and only has folders monitored, no NTFS...But for whatever reason the one that has all the devices on it with NTFS monitoring, that I can use duplication checking for, just gives everything 9223372036.854775806?
Both the columns made are given that and it's completely unusable.
A very few select get what looks like a real timestamp, but only in column A, they still have that number in column b.
Only 5 out of the 305 in current sorting has close to a real timestamp in column A.

It also did some weird visual stuff when I used yours, I did "[foldername] folder: sort:size namedupe: add-column:a a:=$dc:/1000 dupe:columna add-column:b b:=$dm:/1000 dupe:columnb" and it didn't do oddities, but still strange timestamps in those columns.
It was like it was re-sorting 5 times? Seems like it only happened once.

But I guess if the db isn't broken or whatever, it does work. I checked with that folder monitor one and every timestamp does remove those last 3 numbers and otherwise line up..At least as much as I can read it given that it's in the 1336/1337 timestamping system, not something I can understand. But I suppose I don't have to if I am just de-duping. If it ends in 30 in column a, those are the fifth and fourth in the full created timestamp.
It also works in at least one NTFS db. So Idk..Maybe it's too big or something is missing with the monolithic db.

[therube]

I don't know why, but Teracopy doesn't copy folders correctly. They are often a few..microseconds? off. Whatever the last 3 numbers for NTFS are.
Take this folder..
Original date created: 2023-03-10 06:27:11:1345075
Copied date created : 2023-03-10 06:27:11:1345168

Sure enough!

Code: Select all

C:\out>wmic FSDIR where Name="C:\\out\\lynx\\lynx-oldssl-setup_290" list /format
:list | grep Last
LastAccessed=20250127124154.108_513-300
LastModified=20250127124154.108_513-300

C:\out>wmic FSDIR where Name="C:\\Local\\SANDBOX\\PazVideoCutter\\drive\\C\\out\
\lynx5\\lynx\\lynx-oldssl-setup_290" list /format:list | grep Last
LastAccessed=20250127124154.108_524-300
LastModified=20250127124154.108_524-300

(I manually stuck the _ in there to emphasize the diff.)

Source was outside of sandbox.
Destination was inside (if that should matter - but it did not matter to Salamander where timestamps were exactly the same).
Used Teracopy 3.17 (& what a PITA program, if you ask my opinion).

(Fastcopy did not fair any better, not even keeping directory times in sync.
[I might have written about that before? Seem to recall something where the topmost directory was not dated correctly, but subdirectories to it were - but in this case, I did not even see that?]

Salamander's Copy, copy directory, worked.)


And yet, when I did it a second time, from command line (ask me how I figured out that syntax ), it gave identical output directory timestamps (but again, wrong compared against the source).

Code: Select all

C:\out>wmic FSDIR where Name="C:\\Local\\SANDBOX\\PazVideoCutter\\drive\\C\\tera
copy317zip\\lynx\\lynx-oldssl-setup_290" list /format:list | grep Last
LastAccessed=20250127124154.108_524-300
LastModified=20250127124154.108_524-300

(Since you asked, teracopy.exe copy c:\out\lynx\ C:\Local\SANDBOX\PazVideoCutter\drive\C\teracopy317.)