Subversion repository: https://cbftp.eu/svn/cbftp

Last Changed Rev: 956
Last Changed Date: 2018-11-15 23:55:04 +0100 (Thu, 15 Nov 2018)
Latest tar.gz: cbftp-r956.tar.gz

FTP bouncer: cbbncd-0.9.tar.gz

IRC: #cbftp on EFNet

Latest changes:
r956 2018-11-15 - dont count timouted sites as done
r955 2018-11-15 - improved race switching behavior during multiple ongoing
                  spread jobs within a site where a job does not have any dirs
                  created yet
r954 2018-11-15 - improved timeout tracking further
                  fixed a minor scoreboard refresh bug
r953 2018-11-15 - a new stab at timeout balancing
r952 2018-11-13 - improved spread job timeout behavior by not increasing timeout
                  counter while any involved site is busy on all connections
r951 2018-11-12 - ignoring filelist metadata changes (owner, group,
                  lastmodified) in the complete detector and timeout tracker to
                  avoid endless races with buggy sites
r950 2018-11-12 - fixed off-by-one request count in browse mode
r949 2018-11-12 - fixed a crash when deleting the last site in the site list
r948 2018-11-10 - fixed a problem with endlessly running transfer jobs when the
                  source is deleted
r947 2018-11-08 - fixed a few potential errors due to new list splitting
                  behavior
r946 2018-11-07 - fixed a bug with sections/skiplist not being saved properly
                  when configuring a new site to be added
r945 2018-11-07 - fixed a crash when nuking without reason due to unexpected
                  string splitting behavior
r944 2018-11-07 - improved section handling
                  added a section select from list when adding new section on a
                  site
                  added support for section-based skiplists
                  press s from the mainscreen
r943 2018-11-03 - fixed the broken site upload checkbox in the mainscreen
r942 2018-11-03 - improved browsescreen request handling
                  using asynchronous filesystem functions in browsescreenlocal
r941 2018-11-02 - added asynchronous functions for some local filesystem
                  operations
r940 2018-09-28 - improved handling of automatic reconnect
     _   ___ _
 ___| |_|  _| |_ ___
|  _| . |  _|  _| . |
|___|___|_| |_| |  _|
                |_|

Cbftp is an advanced multi-purpose FTP/FXP client that focuses on efficient
large-scale data spreading, while also supporting most regular FTP/FXP use
cases in a modern way. It runs in a terminal and provides a semi-graphical
user interface through ncurses.

MAJOR FEATURES

- Spread jobs for efficient many-to-many FTP server data spreading through FXP
- Transfer jobs for regular file transfer needs: FXP, Upload, Download
- Efficient slot handling: slots are used and reused for any purpose at any
  time. A large job does not "lock" slots, and neither does the UI
- Concurrency: Multiple jobs can run at the same time.
- Encrypted data file with AES-256
- SSL support (AUTH TLS) and SSLFXP support
- Remote commands for performing various tasks through UDP calls
- Ncurses UI, runs in a terminal which makes cbftp highly flexible
- Advanced skiplisting
- Internal file viewer with support for multiple encodings
- File viewing through external applications when running in a local terminal
- SOCKS5 proxy support

INSTALLATION

Build dependencies: make g++ libssl-dev libncursesw5-dev

Procedure: run 'make' in the cbftp root. This produces a cbftp binary in the
'bin' directory, which can then be placed anywhere in the system. Cbftp does
not depend on any other files in the build tree, and it can be removed
afterwards if desired.

If you wish to use your cpu more efficiently while compiling, you can specify
i.e. -j8 to make, to make it build with 8 separate threads.

When compilation has finished, you can start cbftp by simply running the
newly created cbftp binary: bin/cbftp.

FTP site details and other settings are stored in the cbftp data file, which
will be created the first time cbftp is started. It is located at
~/.cbftp/data, and is encrypted with AES-256. You will be prompted for a
passphrase upon starting cbftp.

GETTING STARTED

After starting cbftp and entering your passphrase, you will end up on the
main ui screen of the client. This is where your FTP sites will be listed,
along with the latest few spread jobs and transfer jobs if there are any.
In the bottom of the screen, there will be various key binding information
scrolling by. This area is known as "the legend bar" and is used to list
available key bindings at any given time.
The first thing you will want to do here is probably to add a site, so
press A.

THE UI

The cbftp user interface is based on a bunch of various views - main screen,
edit site screen, browse screen, etc. The available views can be considered
as a stack - after you've entered a new view, leaving it will normally take
you back to where you came from.

BROWSING

After adding a site, you can browse it by selecting it and pressing b
from the main screen. This will take you to a different view where the
contents of the root (or base path) of the site will be shown. You can browse
around through directories by using the arrow keys.
Various features are available here, see the legend bar at the bottom.
The browse screen is also considered a "main window" in cbftp; you can toggle
back and forth between the browse screen and the main screen by pressing esc.

STARTING A TRANSFER JOB

If you would like to transfer something, there are some ways to start a
transfer job. The simplest way is to select an item that you wish to download
and then press D. A transfer job for downloading that item will be started
in the background, and will also be visible in the bottom legend bar while the
browse screen is open. By default, download jobs will be downloaded to the
download path specified in the global options screen, press G from the main
screen to get there.
For other types of transfer jobs, you'll need to open the split view.
Press TAB to open the split view. In the newly opened view, you'll be
presented with the option to either browse your download directory, or any of
your added sites.
After opening the split view, you can press TAB at any time to switch side.
If you wish to start an upload job, browse your download directory, select
an item that you wish to upload, and then press t. A transfer job will be
started in the background that uploads your specified item to the directory
on the site that was opened on the other side of the split view. It will also
be visible in the bottom legend bar while the browse screen is open.
The same thing goes for FXP transfer jobs, but you will obviously need to open
a site instead of the local view. Choose item and press t to start a job
in the background. It will also be visible in the bottom legend bar while the
browse screen is open.
Your newly created transfer job will be visible in the main screen of cbftp.
Transfer jobs use a single slot on each site by default; this can be modified
in the detailed view of each transfer job.

STARTING A SPREAD JOB

A spread job is a larger form of transfer job that spreads an item among
a selection of multiple sites through FXP. It is an action that does not
exist natively in other clients (at the time of writing), and it is the
original purpose of cbftp. Spread jobs do not use 'chains' like other
clients do - see the "the transfer engine" section further down for more
information.

Spread jobs rely on sections being properly defined on all sites that should
be involved - a site can only be added to a spread job if it has a matching
section defined with a suitable path.
To start a spread job, browse a site and select the item that you wish to
spread, and then press 'r'. If the current working directory is bound to a
section, the "new spread job" screen will be opened where you can specify
which sites to include in the job, and possibly which section to use if there
are several bound to the same directory. Sites that have the same section
defined as the one where the selected item was located will be available for
selection.
After selecting sites, press 's' to start the spread job.

SLOT HANDLING

In many traditional FTP clients, a login slot is "locked" to the UI window
that it occurs in. To use multiple slots or multiple sites, separate tabs
or windows must be opened. This is possibly where cbftp differs the most from
other clients - cbftp handles slots and pairs in a very different way.

First and foremost, the UI in cbftp s merely a complement to the backend.
Cbftp can actually be compiled and run just fine completely without UI.
Secondly, cbftp considers each site as an entity, rather than each login
slot - a specific slot can be used and reused for any purpose at any time.
Slots are locked only for the duration of a single file transfer. When the
single transfer finishes, the slot is free to be used for any other task -
most likely the next file in the same job, but it can also be a task from
another job, a raw command, a file list request, a local download for
viewing - whatever cb considers most important at the time.
There is simply no "queue" built up around a single slot, instead cbftp
has a global queue of things to do and jobs to work on, and evaluates what
is most important to do next whenever a slot becomes available.

This also results in that a slot is never bound to be used specifically for
uploading or downloading - a slot can be used for downloading a file,
and then right after it might be used for uploading another file.
The user has partial control over this by specifying how many slots that can
be used for download and upload on each site.

The UI is not built around transfer tabs, since everything happens in the
backend.

THE TRANSFER ENGINE

The transfer engine is the heart of cbftp, and it decides where, when and
how to perform file transfers. It summarizes information about the state of
all current spread jobs and transfer jobs from all sites, and then calculates
which transfers that are most favorable to perform by assigning scores to
potential transfers based on various criteria - site speed vs other sites
where the file is not yet available, file size, race progress on the site
versus other sites, percentage of files uploaded by you, target site priority,
etc. The potential transfers are summarized in a scoreboard where the ones
with the highest scores will be performed first, until there are no more
transfer slots free on the sites. Whenever a file list is refreshed on some
site or a slot becomes available when a transfer finishes, the procedure runs
again.

This action pattern results in that cbftp can and will pair connections
against varying sites frequently. Between every single file transfer,
all conditions are reevaluated and if it becomes more favorable to pair
sites differently, that will happen. This is the main reason why large-scale
data spreading in cbftp is so simple to deal with - there are no chains, no
tab setup, no preparation necessary at all before starting each spread job.
Just specify what to transfer onto which sites, and cbftp will handle the
rest.

The user has partial control over the transfer patterns by limiting which
sites that can transfer to where by using the allow/block lists available for
each site, and also by specifying site priorities.

SKIPLISTING

Cbftp supports advanced skiplisting of what to deny or allow, both on job
and file basis. Skiplisting can be specified globally, per section, and/or per
site.

The global skiplist can be accessed by pressing G from the main screen and
then selecting "configure skiplist". The section-specific skiplist is available
when editing sections globally. The site-specific skiplist is available when
editing a site.

The skiplist works by matching items against the list from the top to the
bottom. If a rule that matches the item is found, the action of that rule
is applied to the item, and the remainder of the skiplist is ignored.

Possible actions are Allow, Deny and Unique. Unique means that only the first
file found in a directory that matches the rule will be allowed - others
will be skipped/denied.

An item is normally a file name or a directory name. There are buttons
available on each entry in the skiplist if it should match files and/or
directories.

The scope setting specifies where the skiplist should apply.
"In race" entries will only be used for matching inside a spread job
directory. It will not be able to match on the name of the spread job
directory itself, and the path in the entry should start inside the
spread job directory.
"Allround" entries will match on entire paths, and can be used to skip
entire jobs on a specific site, or even globally. Allround rules also apply
on regular transfer jobs, which 'in race' rules do not.

To test your skiplist rules, you can use the TEST PATTERN function at the
top.

The site-specific skiplist is applied first for matching on a specific site.
If no match is found it falls through to the section skiplist (for spread
jobs), which in turn falls through to the global skiplist.

Note that regex is currently not supported. * (match any number of any
character) and ? (match any single character) are the currently supported
wildcards. The skiplists are not case sensitive.

Some skiplisting examples:

Skip all files ending with .jpg in the main dir of spread jobs:
*.jpg  [X]  [ ]  Deny  In race

Skip all files ending with .jpg in all subdirs of spread jobs:
*/*.jpg  [X]  [ ]  Deny  In race

Allow only "Sample" and "Proof" as subdirs in spread jobs:
sample  [ ]  [X]  Allow  In race
proof   [ ]  [X]  Allow  In race
*       [ ]  [X]  Deny   In race

Skip all spread jobs with .INTERNAL. in the name:
*.INTERNAL.*  [ ]  [X]  Deny  Allround

Only allow one nfo and sfv file within each directory:
*.sfv  [X]  [ ]  Unique  In race
*.nfo  [X]  [ ]  Unique  In race

Skip files with spaces in the name everywhere:
* *  [X]  [X]  Deny  Allround

And again, note that _the first match applies_! If the skiplist does not
behave as you expect it to do, then you will need to think through if there
might be other rules that are matching your item too early. Use the pattern
test feature.

REMOTE COMMANDS

Cbftp supports starting various tasks via UDP commands. The UDP listener
can be configured in the global options screen (press G from the main screen).
You will need to set a suitable password that the UDP client must provide
for cbftp to accept the UDP commands.

The available command formats are:

<password> download <srcsite> <srcpath> [srcfile]
This command will result in cbftp starting a transfer job for downloading
the specified item to your default download directory. The srcfile field is
optional, if omitted cbftp will use the base name of the srcpath as file name.
The srcpath field can also be a section name, in which case srcfile must also
be specified.

<password> upload <srcpath> [srcfile] <dstsite> <dstpath>
This command will result in cbftp starting a transfer job for uploading
the specified item to the specified site and path. The srcfile field is
optional, if omitted cbftp will use the base name of the srcpath as file name.
The dstpath field can also be a section name.

<password> fxp <srcsite> <srcpath> <srcfile> <dstsite> <dstpath> [dstfile]
This command will result in cbftp starting a transfer job for sending
the specified item from one site to another via FXP. The dstfile field
is optional and will be the same as srcfile if omitted.
The srcpath/dstpath fields can also be section names, and if dstpath is a
section name then dstfile must also be specified.

<password> race <section> <file> <sitelist>
This command starts a spread job with the 'race' profile for the specified
item on the specified sites. The site list is a list of site names,
comma-separated without spaces. (Example: site1,site2,site3). Wildcard (*)
also works, which means all sites added in cbftp will be selected.

<password> distribute <section> <file> <sitelist>
This command starts a spread job with the 'distribute' profile for the
specified item on the specified sites.

<password> prepare <section> <file> <sitelist>
This command prepares a spread job with the 'race' profile, which means
cbftp will log onto the specified sites and await user input. Prepared
spread jobs are started from a list on the main screen and can be started
by selecting the prepared job and pressing enter, or pressing p from anywhere
in the cbftp UI to start the latest prepared job.

<password> raw <sitelist> <command>
This command makes cbftp issue the given raw command on the listed sites.

<password> rawwithpath <sitelist> <path> <command>
This command makes cbftp change directory to the given path and then issue
the given raw command on the listed sites. The path field can also be a
section name.

<password> idle <sitelist> [time]
This command makes cbftp login and idle on the specified sites. Specifying
how long the sites should stay logged in is optional, and defaults to the
"max idle time" specified per site.
The sitelist can also be a section name.

<password> abort <job>
This command makes cbftp abort the specified spread job.

<password> delete <job>
This command makes cbftp abort the specified spread job, and then delete
all related files on all involved sites.

<password> reset <job>
This command makes cbftp soft reset (no re-mkdir) the specified spread job.

<password> hardreset <job>
This command makes cbftp hard reset the specified spread job.

So let's say you've configured your remote command port to 55477, your
remote command password to "bestpassword" and enabled the raw
command listener. You now feel a sudden urge to delete yourself on SITE1.
You can then send an UDP command to cbftp directly via
the command line as follows:
echo -n "bestpassword raw SITE1 site deluser me" > /dev/udp/127.0.0.1/55477

CONNECTION DETAILS

To see details about what cbftp is doing on each connection to a site,
select the site from the main screen and press enter. Here you can cycle
between the connections (if there are multiple) by using the left/right
arrow keys. You can also force connect/disconnect specific connections
from this view.

RAW COMMANDS

To send raw commands to a site, select a site on the main screen and press w.
You will be presented with a new window where raw commands and their results
are shown. Just type and press enter for the command to be sent to the site.
By default, raw commands will be executed from the base path of the site.
You can also go to the raw command view when browsing a site by pressing w.
Raw commands will then be issued in the directory that you were browsing.
The currently selected file name can be pasted by pressing Insert.
If you want to send raw commands on a specific connection, go to the specific
connection (see "connection details" above) and press w there.

ADD A SITE / EDIT A SITE

When selecting to add or edit a site, you will be presented with a bunch
of fields where you can enter settings and parameters for your site.
When you are done editing your site, press 'd' to save your changes.

Field summary:

- Name - The name that cbftp knows this site by.
- Address: the hostname or IP address and port of your site. This field
  supports multiple values if a site has multiple entry addresses available,
  and they can be entered on the same line by separating them with spaces.
- Username: The username that you use to login onto the site. For sites where
  you do not have a username, 'anonymous' should be entered.
- Password: The password for your user on the site. If you do not have a user,
  'anonymous' should be entered.
- Login slots: The number of simultaneous slots that the site allows you to
  log in with.
- Upload slots: The number of simultaneous uploads that the site allows you
  to perform. Enter 0 for unlimited, or same as the number of logins.
- Download slots: The number of simultaneous downloads that the site allows
  you to perform. Enter 0 for unlimited, or same as the number of logins.
- AUTH SSL: Whether the site should be connected to securely with SSL. Note
  that not all FTP servers support this - it depends on the server whether it
  works or not.
- SSL Transfers: The security behavior for transfers to/from this site.
- List command: Which command the site should use to list directories.
  STAT -l is normally faster and better if it is available, but not all FTP
  servers support it.
- Base path: The default path that should be listed first when browsing the
  site, or changed into by default when performing raw commands.
- Max idle time: The number of seconds that the site will stay connected
  before logging out if there's nothing to do.
- SSCN supported: SSCN is a special command used for SSL FXP transfers.
  Not all servers support this, but it should be enabled if it is supported.
- CPSV supported: CPSV is a special command used for SSL FXP transfers.
  Not all servers support this, but it should be enabled if it is supported.
- Needs PRET: PRET is a special command needed for transfers on distributed
  FTP servers such as DrFTPD. Should be enabled on such sites, and disabled
  on all others.
- Force binary mode: Force the site to use binary mode for transfers. This
  is normally the default, but on some FTP servers it is not, in which case
  this option should be enabled.
- Broken PASV: Enable this option if the site reports a bad passive IP or
  does not allow connections on the host/ports it provides.
- Proxy: select which proxy (if any) to use for this site.
- Use XDUPE: This is a special command that reduces control overhead during
  multi-file transfers, but few servers support it.
- Configure skiplist: Set up a skiplist specific to this site. In most cases,
  using the global skiplist is preferred instead.
- Disabled: Disables a site from participating in spread jobs.
- Allow upload: Whether to allow uploading to the site during spread jobs.
- Allow download: Whether to allow downloading from the site during spread
  jobs. Can also be set to "Affils only" to only allow downloading of affil
  releases.
- Priority: How important the site is considered to be during spread jobs.
  The priority is factored into the transfer scoring - you can read more about
  this in the "the transfer engine" section.
- Transfer source policy: Sets whether to have a block list or an allow list
  of sites to transfer with. Setting the policy to "allow" means the
  list below will be a block list, and vice versa.
- Transfer target policy: Same as above but when the site is acting as a
  destination for a transfer rather than a source.
- Block transfers from: The list mentioned above.
- Block transfers to: The list mentioned above.
- Affils: Which groups that pre on the site. Set this list properly to avoid
  uploading into affil releases.
- Configure sections: Set sections/bookmarks for the site. Spread jobs use
  sections to match dirs together between sites, i.e. creating a section with
  the same name on different sites and then using that section in a spread job
  will result in the job operating in the specified section directory for
  each site.

GLOBAL OPTIONS

Most global cbftp settings can be accessed by pressing G from the main screen.
When you are done editing, press 'd' to save changes.

Field summary:

- Default network interface - The network interface that cbftp will bind to
  by default. Useful if you have multiple usable network interfaces, otherwise
  you can ignore this setting.
- Active mode port range - The ports that cbftp will use for active mode
  connections. If you are behind a NAT gateway, you will need to forward
  those ports to your local machine in the gateway. Note that cbftp does not
  use active mode by default - only when a site has 'Broken PASV' enabled.
- Use active mode address - see below
- Active mode address - The address to report for active mode connections.
  If you are behind a NAT gateway, you will need to set this to your external
  IP address (check whatismyip.com). Note that cbftp does not use active mode
  by default - only when a site has 'Broken PASV' enabled.
- Enable remote commands - Whether or not to listen on UDP for remote
  commands.
- Remote command UDP port - the UDP port to listen for remote commands on.
- Remote command password - the password that should be provided in remote
  commands for cbftp to accept them.
- Remote command bell - Trigger the terminal bell when remote commands that
  require user action arrive.
- Prepared race expiration time - The time that a prepared race will remain
  available on the main screen before it disappears.
- Legend bar - the mode of operation for the legend bar.
- Default site *** - default values when creating a new site.
- Download path - the default download path that cbftp should use for
  download jobs.
- Configure skiplist - enters a new screen that lets you configure the global
  skiplist.
- Configure proxy settings - add or remove proxies that can be used by sites.
- Configure file viewing - specify which file types that should be opened
  with what applications. Only applicable when running in a local terminal.
- Change encryption key - Set a new encryption key for the data file.

TRANSFERS

The transfers screen is available by pressing 't' from the main screen.
this screen presents a summary of the transfers that cbftp is performing,
and has performed previously. Select a transfer and press enter for detailed
information about that specific transfer.

GLOBAL KEY BINDINGS

There are a few hotkeys that work from (almost) anywhere in the cbftp UI:
\ - Toggle fullscreen mode (i.e. hide info bar + legend bar).
p - Start the latest prepared race.
K - Kill all external viewers (when viewing files in external applications).
N - Toggle the next prepared race auto starter. While this function is
    enabled, the next 'prepare' remote command that arrives will be started
    immediately. The toggle times out after 10 minutes.
- - Highlight the entire table line. Usable in lists, tables etc where it
    might be hard to tell which items that are on the same line

OTHER UI WINDOWS

There are various other views in the cbftp UI that are not mentioned here in
this readme, but most are quite self-explanatory with the help of the key
binding information found in the legend bar. You can probably figure it out.

FAQ

Q: What key should I press to do xyz?
A: Check the legend bar at the bottom - most key bindings available in the
   current view are presented there.

Q: My modifications are not saved when I edit a setting/site/whatever!
A: You usually need to press 'd' (as in Done) to save settings when editing.
   Pressing c or escape normally means cancel without saving changes.

Q: Can I change key bindings?
A: Only by modifying the source code, as of right now. But I encourage you
   to try that :)

Q: Some fields do not seem to be visible in the UI, or are disappearing
   sometimes. What's going on?
A: Cbftp adjusts the view dynamically depending on how much space is needed
   to show the fields, and how much space is available. The fields shown
   are chosen based on an internal priority specification. Make your terminal
   larger!

Q: Cbftp gets SSL error when connecting to some of my sites, what should I do?
A: Your system is probably using an old version of OpenSSL. Either upgrade
   to a newer system version, or grab a copy of the latest OpenSSL version
   from openssl.org, compile it (./config && make) and then let cbftp know
   that it should use that by modifying the top line of Makefile.inc in the
   cbftp root dir, and then rebuild cbftp.

Q: Is there some raw connection data output available anywhere?
A: Yes, see the "connection details" section above.

Q: How can I see which chains cbftp is using?
A: Cbftp doesn't really use chains in the traditional sense. See the "the
   transfer engine" section further up in this file.
   You can see current transfers and their source/destination by pressing t
   from the main screen.

Q: What's the difference between 'race' and 'distribute' when starting a
   spread job?
A: The profile affects the algorithm that assigns scores to potential
   transfers. The 'race' profile focuses on uploading more files than other
   users everywhere, while the 'distribute' profile focuses on finishing the
   job on all sites as quickly as possible.

Q: What is the block of seemingly random characters above the race status
   table supposed to be?
A: Read each column from top to bottom. Cbftp attempts to describe each file
   in the job by finding a sequence of 3 characters in the file name that are
   unique to that file. In many cases it will be the file suffix, or maybe
   some kind of numbering.

Q: How do I disconnect from a site?
A: Disconnecting is an old habit that comes from traditional clients that
   "lock" slots to the user interface. Cbftp does not do this, and there's
   really no gain in disconnecting manually. Cbftp will disconnect by itself
   after a while. If you really want to disconnect manually, press enter
   on the site from the main screen, use the arrow keys to navigate to the
   right connection, and then press d.

Q: How do I exit cbftp? Do I need to save the data file somehow?
A: press ctrl-c. The data file is written automatically once in a while
   when cbftp is running, and upon exit.

Q: Can I edit the data file manually?
A: Yes, there are tools provided for that: bin/datafilecat and
   bin/datafilewrite. You can also read the file directly through OpenSSL
   commands:
   openssl enc -d -aes-256-cbc -md sha256 -in ~/.cbftp/data

Q: Can I run cbftp on Windows?
A: Yes, it should work through cygwin, but it hasn't been tested lately and
   the polling mechanism available there is not as efficient.

Q: Can I share this software with others?
A: Sure, go ahead.

Q: Will feature X be added soon?
A: I'm open to all kinds of suggestions, but I have very little spare time
   and development is therefore rather slow.

Q: How do I upgrade to a newer version?
A: Just compile and run the new version. The data file will be adjusted to
   the new format if necessary. But CAREFUL! Do not start an older version
   again after this, as this might result in some information being lost
   from the data file. Make a backup of the data file (~/.cbftp/data) if you
   are uncertain or want to try things out.

Q: Can I make modifications to cbftp?
A: That's why the source code is provided! If you are adding things that
   would be useful for others, make sure to pass your changes back upstream,
   and they might end up in the upstream source tree eventually.

Q: Where can I donate to show my support for this awesome software?
A: No need, I mostly do this for my own amusement.