Finding out broken flies and inaccessible folders

[Thy Grand Voidinesss]

Long story short [https://forums.tomshardware.com/threads ... 0.3805224/]:

Is there a way in Everything to generate a list of items being suspected of not being A-OK?


After performing

chkdsk /f /r

on my archive drive I have now 400 GB of files and sub-folders in a newly created system folder named found.000. As I have no idea where they came from, they are useless to me - as I have multiple backups of the same [main] folders, thus the same [sub]names show up multiple times. I will delete them, because I still will have many more copies from different points of time

But the bigger problem is that I do not know also, which folders are now still OK, which are missing just 1 file and which 10000. Although I could just leave them as they are, because those remaining can be unaffected thus be eligible in future to serve as source for file-by-file recovery, but I prefer not to as they take precious space while being inferior to other data-stamped backups

But the biggest problem is that some sub-folders became inaccessible. So if I could automatically locate them then I could delete their parent folders - and not only get rid of damaged / incomplete backups but also make even more room for new complete backups in the future

[NotNull]

This is what I would do (all untested, but should not break anything)

• Create a folder on a different disk than your problem disk. Lets say X:\test
• Copy Everything64.exe to that folder
• Copy a backup Everything database from before these problem to that sam folder
• Rename it to backup.db
• In your regular Everything, go to Options > Indexes
• Check which property is NOT indexed. For example Index attributes is not enabled
• Remember this for later on
• Exit your regular Everything (Menu:File > Exit)
• In File Explorer, browse to X:\test
• Type CMD.exe in the addressbar
• Press ENTER
• Paste the following command and press ENTER:

  Code: Select all

  Everything64.exe -readonly -instance readonly -db backup.db -config nul\no.ini

• When the readonly instance is started, check the statusbar
• If it doesnt say STOPPED, stop here
• In the search bar, search for what you are interested in.
  Fore examle
  C:\
  or
  C:\Data
  or
  C: !C:\Windows
• Right-click the resultlist header and select Attributes (/..)
• Select all resultlist entries (CTRL + A)
• Menu:File > Read extended properties
• Wait till the progress bar is finished

The items in the resultlist *without* an entry in the Attributes (/..) column are the ones that are no longer available on the disk.

Menu:File > Exit to end this instance.



I would also check the health of the disk (SMART data). CrystalDiskInfo is good at that. Install the portable version (but not on the problem disk). Healthstatus should be at least GOOD. I would buy a new disk otherwise.
BTW: this is a good time to buy a new disk; prices have fallen (and arestill falling) rapidly recently

[void]

I'm sorry to hear about the corrupted volume.

If you have backups of the files, the following might help identify larger files (> 1mb):

Search for:
c:\found.000\ size:>1mb

Select all results and press Ctrl + Shift + C to copy all the filenames to the clipboard.

Change the search to:
sizefilelist1: dupe:size

Hold down Ctrl and left click the sizefilelist1: text in the search box.
---This will show the file list slot 1 editor---
Paste the filenames from the clipboard and click OK.

Using matching sizes from your backup you should be able to determine the original filename.

[Thy Grand Voidinesss]

Thank you both for the ideas. But if I understood them correctly they would need a file-by-file intervention / restoration? And I reckon some items might get missed because hey became inaccessible?

With very important documents like tax forms or few existing photos of grandmother they would be doable. But when trying to check / recover for example copies of the same 20 000 songs from ~6 points of time, when having them also in 20+ other backups - it is simply not worth the time and energy. [Because what I lost / damaged are most likely some backups of various folders from the last 3 years from archive drive A. I still have such backups [with different dates] on archive drive B and the current versions]


I was thinking more around the lines of Everything trying to access all folders - and somehow marking those, which I am unable now to enter when navigating in FreeCommander or Windows Explorer. Because if I see such sub-folder then I will assume that also others got corrupted from that timestamped backup

I have tried using TreeSize Free [https://www.jam-software.com/treesize_free] but it does not seem to have such capability


And coming back to your non-applicability of your solutions to my case: the problem with them is alike too hash values. Because what good will do me an info that folder X from 2022 01 01 has now a different value than when it was secured in form of a backup at that time, if it has 100 000 items? Or how would I compare efficiently 10 000 hash values of its sub-folders and then re-act to [lets assume] 1000 mismatches?


[After: dealing with backing up of data for over 20 years, experiencing this first (that I know of) failure of my backup system since 2014, participating in another discussion about data storage (https://forums.tomshardware.com/threads ... ps.3805050) and discussing cons and pros of various approaches with ChatGPT - I know that I need to acquire another archive drive (C) and also keep it off-line and never connect it at the same with A and B, while also keep burning Blu-ray discs. But the problem is I would need to buy a 22TB SAS HDD, while wanting at least 16TB SATA3 SSD. As for the A, it is quite new and have not been even filled up, with my lost occurring to some accumulated software glitches]

[horst.epp]

I don't understand why you wasting time in case of such hardware failure.
Following normal backup procedures you have several generation of backups on different mediums.
In addition I make automatic encrypted backups of changed files from importand dirs to the cloud.
This prevents data loss between backups.

Any tool trying to show you what objects are defect after an failure will also fail
on trying to access it the normal way.
The file/dir APIs are not handling such problems in a controlled way
you just get a hanging app or other strange behaviour.

[Thy Grand Voidinesss]

I don't understand why you wasting time in case of such hardware failure.
[...]

But how I am suppose to know which folders in which copies were affected? Other than manually going folder by folder and [after manual and time consuming evaluation] deleting them

I want to delete those where I lost more than ~5% of data. And those only slightly affected I will e.g. rename by adding an X at their name's end so that in distant future they will be the first be deleted for good, making room for newer backups. And undamaged ones I will keep

[NotNull]

I was thinking more around the lines of Everything trying to access all folders - and somehow marking those, which I am unable now to enter when navigating in FreeCommander or Windows Explorer.

What does that mean? Do you see a folder in File Explorer and you can't go into that folder? What happens if you try? Any message?
Or is there a folder missing that you know was there in the past?


FWIW: My suggestion was about listing missing files/folder; void's suggestion was about a way to restore these missing files.

[Thy Grand Voidinesss]

I was thinking more around the lines of Everything trying to access all folders - and somehow marking those, which I am unable now to enter when navigating in FreeCommander or Windows Explorer.

What does that mean? Do you see a folder in File Explorer and you can't go into that folder? What happens if you try? Any message?

An OS pop-up window informing me of inability to access such sub-folder upon trying to navigate to it; on which executing two methods of Take Ownership did not change its availability

But just know, wanting to provide you with verbatim info, I got a window saying

You don't have permission to access this folder

but this time it was a pop-up with a button for granting me such access- after pushing of which I really did gain it. There was no such button on my previous attempts at randomly pin pointed sub-folders - and now I do not seem to be able to locate other examples of affected / blocked ones

Or is there a folder missing that you know was there in the past?
[...]

Also

That is why I had said

[...]
[Because what I lost / damaged are most likely some backups of various folders from the last 3 years from archive drive A
[...]

as those outright / complete absences are easy to spot for me

[therube]

sizefilelist0:

Thanks for reminding me of that. It's useful .

[Thy Grand Voidinesss]

I was thinking more around the lines of Everything trying to access all folders - and somehow marking those, which I am unable now to enter when navigating in FreeCommander or Windows Explorer.

What does that mean?
[...]

In terms for what kind of neat indications I am looking for:

for comparison after loading all of my audio files to Mp3tag , its Column with time / length data for few MP3 files was showing nothing - which I spotted easily after sorting list of files by this Column with tag field

%_LENGTH%

. [And thus is how I detected a damage that must had occur ~10 years ago, because such ill files were copied over and over to every next backups]


So does anyone have idea for something similarly quick / easy in Everything - but in regards to all file formats and also folders

[therube]

Well...

If you hashed all your files, including the FILE0000.CHK files, then any UNIQUE FILE0000.CHK files simply are not backed up or are damaged versions compared to your backups. (Everything can do UNIQUE's .)

If you know more about the .CHK files, or more ideas about the .CHK files, like particularly directories they're likely to have come from, or you can limit your hash gathering to particular directory trees...

Everything can display Length & other properties.
You can use a UNIX-like file utility (with "magic" numbers) & run that on your .CHK files to (attempt to) identify the file types (& then rename them as such). .CHK.txt .CHK.mp3 .CHK.xls ...

Code: Select all

C:\out\>file -m C:\BIN\UNIX\magic *.chk

FILE0000.CHK; ISO-8859 English text, with very long lines, with CRLF line terminators
FILE0001.CHK; data
FILE0002.CHK; data
FILE0003.CHK; data
FILE0004.CHK; data
FILE0005.CHK; data
FILE0006.CHK; data
FILE0007.CHK; data
FILE0008.CHK; data
FILE0009.CHK; data
FILE0010.CHK; data
FILE0011.CHK; data
FILE0012.CHK; data
FILE0013.CHK; ASCII text, with CRLF line terminators
FILE0014.CHK; Zip archive data, at least v2.0 to extract
FILE0015.CHK; Zip archive data, at least v2.0 to extract
FILE0016.CHK; DOS executable (device driver)
FILE0017.CHK; DOS executable (device driver)
FILE0020.CHK; DOS executable (device driver)
FILE0034.CHK; DOS executable (device driver)
FILE0035.CHK; DOS executable (device driver)
FILE0036.CHK; data
FILE0039.CHK; data
FILE0046.CHK; data
FILE0090.CHK; data
FILE0152.CHK; data
FILE0516.CHK; data
FILE0789.CHK; data
FILE2710.CHK; DOS executable (device driver)
FILE2723.CHK; DOS executable (device driver)

C:\out\KKK\K-ORSAIR-0202\Y-OK\FOUND.000>

Along the same lines (& untested), CHK File Recovery: 9 Ways to Recover CHK Files.
(I do use the command line version of TRIDNet, but not for these purposes.)

[therube]

And thus is how I detected a damage that must had occur ~10 years ago

Not necessarily "damage", per se - could just be unable to read the Length.
(As it is Everything relies on Windows facilities, most often, so if Windows can't read a particular files' Properties, Everything also will not see it. Even an incorrect file extension, on an otherwise correct file, can cause "missing information".)

[Thy Grand Voidinesss]

So as it seems that there is no relatively quick and easy way to know where that repetitive content in found.000 came from I will just delete it

Also: looking trough its content it seems that only backups of my music were rescued, while many other types of my backups are lost. And looking thorough remaining folders / backups it seems than was is missing is all backups from the last two years, while a little older ones and the newest ones were inaccessible [but eventually I was granted access to them].

Now I wonder what could cause such narrowed down damaged and then rescue

[horst.epp]

So as it seems that there is no relatively quick and easy way to know where that repetitive content in found.000 came from I will just delete it

Also: looking trough its content it seems that only backups of my music were rescued, while many other types of my backups are lost. And looking thorough remaining folders / backups it seems than was is missing is all backups from the last two years, while a little older ones and the newest ones were inaccessible [but eventually I was granted access to them].

Now I wonder what could cause such narrowed down damaged and then rescue

To detect such problems early enough
I suggest to create checksum files for your importand trees
and verify the created backups with this files.

[Thy Grand Voidinesss]

[...]
create checksum files
[...]

Thank you for the advice

But with the amount of backups I use and the overall number of items counted in millions it would be just a waste of time - as I would also need to store at least two versions of the same timestamped backup on two separate drive. And would have to deal with not connecting them at the same time, thus being forced to use some intermediary when wanting to replenish a damaged one. And what if a mismatch would be shown on the account of damaged but totally unimportant e.g. desktop.ini file of some sub-folder - hash value would not inform me that it was the cause of the mismatch unless I would hash millions on files on a one-by-one basis

[What I crave now is a minimum 12 TB SATA SSD - but they do not exist, while 8 TB is not enough and 16 TB cost way too much]

[ChrisGreaves]

Now I wonder what could cause such narrowed down damaged and then rescue

I would bet money on the type of damage and hence the program code (i.e. application) that caused it.
I know that that doesn't solve your problem, but "damage" isolated by type and time is usually "damage caused" by type (of action) and time.
Just something to think about.
Cheers, Chris

[ChrisGreaves]

[What I crave now is a minimum 12 TB SATA SSD - but they do not exist, while 8 TB is not enough and 16 TB cost way too much]

This is just user-economics, and is understandable. My New HP15 laptop is constrained by what features I want ("A bigger SSD than anyone else in the world has") and the credit limit on my credit-card.
Just out of interest, what prices are you looking at for 8*, 12, and 16 TB drives. Roughly?
Cheers, Chris

[Thy Grand Voidinesss]

[This got off-topic]

[...]
Just out of interest, what prices are you looking at for 8*, 12, and 16 TB drives. Roughly?

The longer I think and educate myself about it, the harder is my choice

I am willing to spend ~555 €, as I know I will use this device for the next ten years, thus making its daily cost negligible. But I refuse to buy a [yet another big or huge on-the-fly backup or offline archive] drive by spending as much as I would have to in order to buy an ultrawide OLED; which is [with proper parameters] something I crave for much more [still waiting for advancements in display technology].

In other words - I seek an optimal solution: viewtopic.php?t=13273

[Thy Grand Voidinesss]

Now I wonder what could cause such narrowed down damaged and then rescue

I would bet money on the type of damage and hence the program code (i.e. application) that caused it.
I know that that doesn't solve your problem, but "damage" isolated by type and time is usually "damage caused" by type (of action) and time.
[...]

Which would point out me to either Everything or FreeCommmander or just the Windows 10 - all of which I use daily

Or: it was a case of experiencing of data fade - just when as I finally had started using DiskFresh; and after rewriting of first offline archive drive took on the other storage device and prepared it by removing excess of some data-stamped backups

Or any combination of the above

[Thy Grand Voidinesss]

So as it seems that there is no relatively quick and easy way to know where that repetitive content in found.000 came from I will just delete it
[...]

I did it- and no without issues

Because in the end I had to use FastCopy's feature of wiping folders - and for whatever ill reason it was doing it very slowly