Do an Everything search on remote FTP?

awesnap · Post by **awesnap** » Mon Dec 19, 2022 4:55 pm

Hello there. First of all, thank you for making this tool, its quite useful!

However, i'm hoping you can help me. I have a rather large (15TB) ftp server that I use for work, and searching on Filezilla is..... painfully slow.
Wouldn't it be nice to crawl the FTP server, index its contents, and then use a program like EVerything to search for files, folders, etc? I thought so too! However, I cannot seem to figure out if I can accomplish this with Everything.

WHen I attempt to add the FTP credentials in Tools>Options> ETP/FTP Server, i'm met with an error: Unable to start ETP Server: bind failed 10049

I also attempted to use a program such as WebDrive to map the FTP server to a drive letter (W:) however, this also gives me an error.

Can you tell me, is there a way that I can use Everything for my needs? I"m REALLY hoping so, as searches may take so long, that I have to run them overnight for specific files!

Thanks in advance!

horst.epp · Post by **horst.epp** » Mon Dec 19, 2022 5:26 pm

You can map a drive letter to your FTP server
and use Folder indexing in Everything.

awesnap · Post by **awesnap** » Mon Dec 19, 2022 8:28 pm

Thanks for the quick reply, however in WIndows 10 you can't map a drive letter to an FTP server.

That's why I downloaded WebDrive in order to accomplish this. However, when I do access the mapped FTP letter'd drive (via WebDrive) I get an error: Unable to start ETP Server: bind failed 10049

I am able to "Add Network Location" however I am unable to add this as a folder to be scanned in Everything.

Post by **void** » Mon Dec 19, 2022 10:18 pm

Thank you for your feedback awesnap,

Please don't use Tools>Options> ETP/FTP Server
This is for hosting an FTP server, not connecting to one.

To index an FTP server in Everything 1.5:

In Everything 1.5, from the Tools menu, click Options.
Click the Folders tab on the left.
Click Add custom....
Type in your ftp address, for example: ftp://server/share
Click OK.
Click OK.

Please note: Indexing an FTP server will take a very very long time.

awesnap · Post by **awesnap** » Mon Dec 19, 2022 10:37 pm

Thank you void for your response, a few more questions for you.

1 - is there somewhere that i'd have to put my login credentials?
2 - if I scan the ftp, can I then take that database with me to another machine to speed up the process of searching on that ftp server?

Post by **void** » Tue Dec 20, 2022 2:13 am

1 - is there somewhere that i'd have to put my login credentials?

Please try the following custom folder:

ftp://username:password@server:port/share

if I scan the ftp, can I then take that database with me to another machine to speed up the process of searching on that ftp server?

Yes, with the Everything Server:

Install Everything on a PC that is always on.
Setup an Everything Server.
Have the Everything Server index your FTP site.
Have the Everything Server rescan your FTP site daily (this would be the default option)
Add the Everything Server to your multiple Everything clients under Tools -> Options -> Network indexes.

awesnap · Post by **awesnap** » Tue Dec 20, 2022 3:23 am

I have a synology server thats always on, is it possible to install it on that?

Post by **void** » Tue Dec 20, 2022 3:24 am

No, sorry.

Everything is currently Windows only.

awesnap · Post by **awesnap** » Tue Dec 20, 2022 3:40 am

Ok, I had added the FTP server and it is connecting, however it is not recursivly adding any folders, or files. I am looking at the debug window, but I can't understand what its telling me. It seems to be stuck on HCBT_SEWTFOCUS

awesnap · Post by **awesnap** » Tue Dec 20, 2022 5:33 pm

Alright, I spoke too soon, it seems to be recursing through the ftp, at least in the dialog box.

Question, with version 1.5a, I had to shut it down and reopen it a couple of times, however the settings that I made in the Options menu go back to defualt every time I restart the program. Is there a reason its doing this, or is there a way to save my settings so I don't have to keep doing my Excludes, and adding of the custom FTP settings?

Thanks in advance!

horst.epp · Post by **horst.epp** » Tue Dec 20, 2022 6:18 pm

awesnap wrote: ↑Tue Dec 20, 2022 5:33 pm Alright, I spoke too soon, it seems to be recursing through the ftp, at least in the dialog box.

Question, with version 1.5a, I had to shut it down and reopen it a couple of times, however the settings that I made in the Options menu go back to defualt every time I restart the program. Is there a reason its doing this, or is there a way to save my settings so I don't have to keep doing my Excludes, and adding of the custom FTP settings?

Thanks in advance!

I guess you installed Everything under Program files
which doesn't allow to update your settings.

Set in the GUI under Options / General
Store settings and data in %APPDATA%\Everything

Or use the portable version which I prefer.

awesnap · Post by **awesnap** » Tue Dec 20, 2022 10:45 pm

Ok, i'm giving the portable version a try. Where does the portable version keep the database?

Oh, and thank you so much for going through this with me, this has been super, super helpful support

Post by **void** » Tue Dec 20, 2022 10:55 pm

The portable version stores settings and data (the Everything.db) in the same location as your Everything.exe

Please make sure Everything will have write access to this folder.
-or-
Enable Tools -> Options -> General -> Store settings and data in %APPDATA%\Everything

awesnap · Post by **awesnap** » Tue Dec 20, 2022 11:59 pm

Ok, quick question about the database.

I can't seem to find it, even when I was using version 1.4. Is the folder that the option %APPDATA%/Everything referring to this?
C:\Users\USERNAME\AppData\Local?

Also, how can I make sure that everything.exe has permissions to write to its own directory?

I had Everything (portable 1.5) scan one folder on the FTP, and it just completed it. I don't want to shut the program just yet until I make sure it has saved its progress (the databse), and before I have it scan the FTP further. So i'd justr want to know the answers to these before I proceed.

Thanks in advance!

Post by **void** » Wed Dec 21, 2022 12:07 am

Everything stores its database in memory.

The database is only written to disk when you exit Everything.

Exit Everything to update the database on disk.

awesnap · Post by **awesnap** » Wed Dec 21, 2022 12:50 am

Ok, I just exited everything, however there is no Everything.db, or any file that I can tell in the everything folder with the portable version. In addition, the dialog box has some red text saying:
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Search History-1.5a.csv.tmp
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
failed to create config file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Plugins-1.5a.ini.tmp
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
failed to create config file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
processed 0 usn records in 0.000000 seconds
REFS 0000000000000000
db_save_local 5512 folders, 178657 files
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Everything-1.5a.db.tmp
failed to save C:\Program Files\Everything 1.5a\Everything-1.5a.db.tmp 5
saved db: 0.000338 seconds
ready
DB_WAIT: _db_ready_proc waiting for _db_save_thread_proc...
DB_WAIT: _db_ready_proc waited 0.000064 seconds
_db_query_results_changed_event_proc 0000000000000000
destroy ui
fileinfo clear all

awesnap · Post by **awesnap** » Wed Dec 21, 2022 12:52 am

Correction, there is now a Everything-1.5a.db file, however the size is only 1kb
Reopening the program does however have my settings saved from the Tools menu

Post by **void** » Wed Dec 21, 2022 12:53 am

Everything will not have write access to C:\Program Files\Everything

Please enable "Store settings and data in %APPDATA%\Everything" from Tools -> Options -> General.

tuska · Post by **tuska** » Wed Dec 21, 2022 1:18 am

... Here is an example:

: 2022-12-21_APPDATA-Everything__Save database to disk without having to close Everything.png (66.63 KiB) Viewed 18897 times

Based on this topic, I can save my database to disk via a shortcut (.lnk) with parameter -update without having to close Everything, e.g.
[adjust the path if necessary ...]

Code: Select all

C:\Everything\Everything64.exe -update

awesnap · Post by **awesnap** » Wed Dec 21, 2022 2:20 am

I'm not sure why, but now Everything is not scanning the FTP at all.

The only change that I made was in the General tab, to store settings and data in %APPDATA%\Everything, but now even changing that option to be off, its just not scanning.

The noly thing the Debug console says is this:

update ftp://USER:PASS@SERVER:21/SHARE now
check for folder updates 0000000000000000
update folder start ftp://USER:PASS@SERVER:21/SHARE
scanning ftp://USER:PASS@SERVER:21/SHARE...
scan completed in 2.888309 seconds
rescan update db
rescan update db in 0.000317 seconds
next update at 21/12 3:00
leave folder update
check for folder updates 0000000000000000

Post by **void** » Wed Dec 21, 2022 2:25 am

Please make sure "Fast rescan" is unchecked under Tools -> Options -> Folders -> (your FTP folder)

awesnap · Post by **awesnap** » Wed Dec 21, 2022 2:51 am

It is not checked, the only thing that is checked is "Index Subfolders"

awesnap · Post by **awesnap** » Wed Dec 21, 2022 2:57 am

Ahhh, ok I got it. It was my mistake, I had the wrong folder it was pointing to, thats why it was not scanning any farther. Ok, I will let this run, and see what happens.

Post by **void** » Wed Dec 21, 2022 2:58 am

Everything is likely not seeing the root ftp folder.

Please try forcing a rebuild from Tools -> Options -> Indexes.

awesnap · Post by **awesnap** » Wed Dec 21, 2022 3:32 am

Ok, so its been running for a bit (still running) and I typed in C:\program files\everything 1.5a>Everything64.exe -update
However, when I navigate to C:\Users\USER\AppData\Local\Everything
The database is there, and its been modified at 10:33pm. but the size is still only 1kb.

Post by **void** » Wed Dec 21, 2022 3:34 am

When "Store settings and data in %APPDATA%\Everything" is enabled, the Everything.db is store in:

%LOCALAPPDATA%\Everything

awesnap · Post by **awesnap** » Wed Dec 21, 2022 1:43 pm

Ok, great news. I woke up today to a database with the size 10mb, in appdata/local/everything.
And the results from the scan were in Everything when I was looking at it. I then added each folder that I wanted to scan with Everything (Another 8 directories on the FTP) and hit rescan now.

However, when I do this, the results dissapear in Everything, and on the bottom status bar, it just says "Scanning ftp://..............."

Is there a way to still see what results it has up to this point, while its scanning?

awesnap · Post by **awesnap** » Wed Dec 21, 2022 3:41 pm

Ok, the scan for the rest of the server has (for some reason) stopped. Is there some sort of maximum size of the database that is ticked somewhere?

I hit rescan, and its saying the scan is complete, but its not. It has scanned about 35-40% of the server, but i'd like it to scan the entire thing. Thanks in advance!

Post by **void** » Thu Dec 22, 2022 7:38 am

Everything 1.5.0.1332a fixes an issue with rescanning virtual folders.

Please try rescanning your FTP folder with this version.

Does the issue persist?

awesnap · Post by **awesnap** » Thu Dec 22, 2022 5:34 pm

Yes, unfortunately. I ran the new version you'd linked to in portable mode, and rescanned.
It seemed to stop at the same point as the last one, its currently showing 462,610 objects, and around 4.16 TB of data, which is almost exactly where it stopped last time. The total size is more than that, i'd say it scanned about 35% of the server.

Is there some sort of maximum number it can scan? Or a time out? Because it stopped right around the same point during the scan. And, when I attmept to "rescan now" , the debug is saying the scan is completed in 0.022 seconds.

As a test, i'm able to connect to the FTP via both filezilla, and my mapped frive via webdrive, and look around the folders/files, so I still have access.

Post by **void** » Fri Dec 23, 2022 3:38 am

Sounds like Everything is getting stuck on a particular path.

Everything has a limit of 259 characters for FTP paths.

Debug logs may help find the issue:

Restart Everything (File -> Exit -This will clear virtual folder cache)
In Everything, from the Tools menu, under the Debug submenu, check Verbose.
From the Tools menu, under the Debug submenu, check Start Debug Logging.
Rescan your virtual folder:
From the Tools menu, Click Options.
Click the Folders tab.
Select your ftp folder.
Click Rescan Now.
---wait for rescanning to complete (progress is shown in the top right and in the status bar)
In Everything, from the Tools menu, under the Debug submenu, check Stop Debug Logging.
This will open your %TEMP%\Everything Debug Log.txt in Notepad.
Please examine this debug log for the last scanned for folder.
What was the last scanned folder?
-If unsure, could you please send your Everything Debug Log.txt to support@voidtools.com

Privacy

There is currently an issue with Everything using stale cache when re-scanning your virtual folders.
I am working on a fix..

awesnap · Post by **awesnap** » Fri Dec 23, 2022 5:53 pm

Ok i've done it. Its done scanning, however the last file scanned is shorter than the other files previously scanned (I can send you the debug log to examine). Using a charachter counter, its telling me there is 129 characters in the last file scanned. It did however scan the 2nd folder that i'd asked it to Rescan, which it did not do yesterday. So far its about 45-50% scanned the entire FTP, so thats an improvement, but still not the entire thing.

One other thing to note, I double checked my RAM, and I have 15.9 out of 16 gb used at the moment, not sure if that's anything to be concerned about? (Currently running MANY chrome tabs opened at the moment, but hasn't been an issue in the past, or for other programs, but figured i'd mention it anyway.)

After the last scanned file in the Debug logs, the next lines in the Debug are as follows:

2022-12-23 12:19:29.955: event post: 00007ff7ff522190 00000000011405a0
2022-12-23 12:19:29.955: processed folder fd update events in 0.569638 seconds
2022-12-23 12:19:29.955: MSG: 0000000000b61494 1401 0000000000000000 0000000000000000
2022-12-23 12:19:29.955: TRAY 00001401 0000000000000000 0000000000000000
2022-12-23 12:19:29.955: update EVENTs
2022-12-23 12:19:29.956: EVENT: 00007ff7ff522190 00000000011405a0
2022-12-23 12:19:29.956: DB_WAIT: _db_monitor_finished_process_fd_update_events_event_proc waiting for _db_monitor_folder_process_fd_update_events_thread_proc...
2022-12-23 12:19:29.956: DB_WAIT: _db_monitor_finished_process_fd_update_events_event_proc waited 0.000041 seconds
2022-12-23 12:19:29.956: _db_query_results_changed_event_proc 0000000001148070
2022-12-23 12:19:29.956: _db_monitor_process_update_event_available_event_proc

One other thing i've noted, the database size is actually SMALLER than the old one.
The first one is 115mb, and after this recent scan its only 41bm. Now showing 889,240 objects, and 7.94 TB

awesnap · Post by **awesnap** » Fri Dec 23, 2022 11:28 pm

One more issue i've come up with.

I haven't actually used Everything to download any of the files on the ftp, but just now I attempted to click and drag one of the files to a local folder, and was met with this error:

The item could not be transferred. Make sure you have permission to store files here and that the file name is valid.
Details:
No more internet handles can be allocated

However, if I navigate via webdrive to that same folder and click and drag, it works with no issues.

Post by **void** » Fri Jan 06, 2023 1:53 am

Thanks for the debug logs.

Everything uses the Windows Shell to list FTP folders and open/copy ftp files.

The Windows Shell is rather limited and broken as an FTP client.
It's fine for a few folders.
There seems to be handle leaks in the Windows Shell and there is only so many folders you can browse before you hit some internal limit.

Everything 1.5.0.1333a will now refresh ftp cache when scanning an ftp folders.

This might help with handle leaks? and should also help with keeping your indexes up-to-date by forcing a refresh.
Previously the Windows Shell would cache ftp folders and never refresh.

awesnap · Post by **awesnap** » Sat Jan 07, 2023 5:48 pm

Thank you for the update. I loaded the portable version, re-setup the custom folders with the ftp location to index, and hit rescan.
The status bar on the bottom says its scanning the folder, however when I pull up the debug, it seems to just sit there, it doesn't show the folders being scanned. I'll let it sit for a few hours to see if there is any changes, but it doesn't seem to want to go anywhere.

I'll post my findings here once there's an update.

awesnap · Post by **awesnap** » Mon Jan 09, 2023 12:26 am

Ok, so Everything has stoppeddthe scanning. It has again, stopped at 4.16TB worth of files, and 462,603 objects. This was using the 1.5.0.1333a version, with the portable.exe. Not sure if this helps, but it seems that its stopping around the same point, even from the previous version.

It also didn't scan the other folder (I had put 2 of the large folders in the FTP to scan) but it stopped midway through one of the larger folders.

Post by **void** » Tue Jan 10, 2023 9:07 am

Thank you for the information awesnap,

What's the last reported folder scanned?

In Everything 1333a, from the Tools menu, under the Debug submenu, check Verbose.
From the Tools menu, under the Debug submenu, check Start Debug Logging.
Rescan your virtual folder:
From the Tools menu, Click Options.
Click the Folders tab.
Select your ftp folder.
Click Rescan Now.
---wait for rescanning to complete (progress is shown in the top right and in the status bar)
In Everything, from the Tools menu, under the Debug submenu, check Stop Debug Logging.
This will open your %TEMP%\Everything Debug Log.txt in Notepad.
The last folder rescanned will be shown near the bottom as:
Code: Select all
```
scan ftp://server/share/...
scan completed in x seconds
```

Please try repeating the same rescan, is Everything dying on the same path?

awesnap · Post by **awesnap** » Fri May 24, 2024 6:43 pm

Hello there. I'm back. I know my last comment on the forum was over a year ago, but I would like to attempt to use Everything for my specific purpose once again.

I had downloaded the most recent portable version (1.5.0.1378a) and I had added a custom folder in Tools>Options>Indexes>Folders
The folder has an FTP style tag, aka :
ftp://username:password@ftp.ftpserver.c ... /subfolder
The location that I had decided to start Everything on has MANY sub folders, and an unusually large amount of folders/files. Nevertheless, I decided to point Everything to this folder and attempted this once more.

On the status line on the bottom of Everything, I see "Scanning (the ftp folder location)
However, I see nothing in the Everything window, its just blank.

I had then went into the debug console to see what I could see, and I just keep seeing the same line, but with different locations, like this:

SHCNE CREATE This PC\Windows (C:)\Users\(my user)\AppData\Local\Microsoft\Windows\History\Today\private.ftpserver (private.ftpserver.me)\subfolder 2208
SHCNE CREATE This PC\Windows (C:)\Users\(my user)\AppData\Local\Microsoft\Windows\History\View By Order Visited Today\subfolder 2208
SHCNE CREATE This PC\Windows (C:)\Users\(my user)\AppData\Local\Microsoft\Windows\History\Today\private.ftpserver (private.ftpserver.me)\another subfolder 2208

But no actual files, just folders and sub folders. Still though, nothing coming up in the Everything window.

Also, I had attempted to scan my local drive just for testing with this version, and I definitely saw and searched for files successfully, however I cannot locate the DB file. I am using the portable version, but I do not see any file i nthe same folder as where i'm running Everything in.

One last question, I keep seeing this same line over and over in the debug, which makes NO sense at all:

SHCNE UPDATEITEM S:\SynologyServer\FolderInsideSynologyServer\Thumbs.db

and

SHCNE UPDATEDIR C:\Users\h8the\Desktop

The S:/ is a synology server that i have mapped to a network drive
Not sure why these are coming up very very often. In fact, if the program is idle, the S:/ line appears every second.

awesnap · Post by **awesnap** » Fri May 24, 2024 7:33 pm

Quick update....

Letting Everything do its thing for a while, the files did start to show up in the Everything search window!

However, since this folder won't change from here on out, there was no need to have it rescanned. So, I went into tools>options>indexes>folders, and unticked "Attempt to monitor changes" and "Never Rescan" selected, as well I had added the rest of the folders I wanted to have scanned, and POOF

All of the files that were searchable are now gone, lololol. No idea where they went, or where the DB is to check its size, but... It seems as if all of the crawling it just did was for nothing, and the folders/files went away.

awesnap · Post by **awesnap** » Sat May 25, 2024 12:41 am

Ok, another quick update.

Everything has just stopped scanning for some reason. Its showing 260,146 items, and 2.14 TB. (For my records also)
The debug console is just filled with the same lines over and over again, please see the attached image.

awesnap · Post by **awesnap** » Sat May 25, 2024 6:44 pm

Another update.

Ok, the message above has stopped, and it seems that the folders are being scanned, and somehow temporarily stored in Everything, because I can search them. However, the database file doesn't seem to be updated at all. Now, I have the option for "Store Settings and data in %APPDATA%\Everything, however when I browse both Local and Roaming, there is no new database. Instead, the database file in the same directory as the EXE is updated, but it seems that it was saved at 4am (the default hour for the Daily DB save option). I had since turned the option to Interval, and set the interval to 3600000 (1 hour) We will see if that works.

However, its very funny. The option for "save at exit" for the db is turned on, but when I exit Everything, nothing gets saved.
It takes about an hour, maybe a little less, for one of the folders to be scanned by everything (there's about 11 folders to scan) so after its done scanning and NOT saving to the DB, I have to just keep on running the scan, to see if its saving anywhere. Once its scanned and its showing up in Everything, things seem to be working as intended, but like I said, no saving, and no recalling of data once opening.

Post by **void** » Mon Jun 10, 2024 4:52 am

On the status line on the bottom of Everything, I see "Scanning (the ftp folder location)

Hover over this status bar item to see the current folder being scanned.

However, I see nothing in the Everything window, its just blank.

This is expected for the initial scan.

I had then went into the debug console to see what I could see, and I just keep seeing the same line, but with different locations, like this:

SHCNE CREATE This PC\Windows (C:)\Users\(my user)\AppData\Local\Microsoft\Windows\History\Today\private.ftpserver

These are local file changes.

The progress for indexing your ftp site is only shown in Verbose debug mode (Tools -> Debug -> Verbose)

SHCNE UPDATEITEM S:\SynologyServer\FolderInsideSynologyServer\Thumbs.db

Something is spamming your system with changes to this file.
Check the Windows Resource Viewer to find the process writing to this file.

SHCNE UPDATEDIR C:\Users\h8the\Desktop

Same thing again, a process is constantly writing to this folder.
It can be ignored.

However, since this folder won't change from here on out, there was no need to have it rescanned. So, I went into tools>options>indexes>folders, and unticked "Attempt to monitor changes" and "Never Rescan" selected, as well I had added the rest of the folders I wanted to have scanned, and POOF

Check Tools -> Debug -> Statistics -> Build -> Last Rebuild reason.
This will show why Everything performed a reindex. (just changing the monitoring shouldn't have performed a reindex)

However, the database file doesn't seem to be updated at all.

Everything will only update the database on disk when you exit Everything.
For Everything 1.5, Everything will periodically save to disk daily.

However, its very funny. The option for "save at exit" for the db is turned on, but when I exit Everything, nothing gets saved.

This is most likely a write permission issue.

Looks like you have changed your database location.
Please make sure Everything has write permission to this folder.

Please try resetting your database location back to empty to use the default location:
%LOCALAPPDATA%\Everything

jard9911 · Post by **jard9911** » Sun Jul 28, 2024 11:08 pm

I would like to do the same thing for my Website files, but after reading the dialog, I think I will stick to my old procedure of periodically backing up all my Website files to my Windows computer and then let ET tell me what I have!