## Welcome to Ebulk tool documentation page

# DESCRIPTION

Ebulk tool makes easy to exchange or archive very large data sets. It performs data set ingestion or download from different storage inputs, to Wendelin-IA platform (based on stack Wendelin - NEO - ERP5). It also allows to perform local changes in data sets and to upload the added and modified files. One key feature of Ebulk is to be able to resume and recover from errors happening with interrupted transfers.

# REQUIREMENTS

Java 8: Ebulk relies on Embulk-v0.9.7 bulk data loader Java application (please see Embulk-doc), so Java 8 is required in order to install Ebulk tool.

# SYNOPSIS

ebulk [-h|--help] [-r|--readme] [-e|--examples] <command> [<args>]

[-d|--directory <path>] [-c|--chunk <size>]

[-s|--storage <storage>] [-cs|--custom-storage]

[-a|--advanced] [-dc|--discard-changes]

# COMMANDS

-h

--help

Prints the synopsis and the list of commands and options, with a brief explanation of them.

-r

--readme

Access the README file for a detailed explanation of Ebulk installation and usage.

-e

--example

Prints some basic examples about Ebulk usage.

pull [<dataset>]

Download operation: downloads the content of the specified remote data set from the Wendelin-IA site into the target output. By default, the output is a directory named as the data set.

<dataset> argument: unique reference of the remote data set. It is optional because if no data set is specified, the current directory will be used as data set reference and directory.

Data set reference must be one of the available datasets on the Wendelin-IA site.

Data set argument can be a path to a directory, then the directory name will be used as data set reference: e.g. ‘ebulk pull my_directory/sample/’ --> data set reference will be sample. That directory will be linked to the data set reference, so any future operation on it will refer to that data set, no matter if the directory is moved or renamed.

If pull operation is run on a previously downloaded data set, the tool will offer the options to update it or download it from scratch, warning about any conflict with local changes.

pull options: [-d|--directory <path>] [-c|--chunk <size>] [-dc|--discard-changes]

push [<dataset>]

Ingestion operation: uploads the content of the specified input data set to the Wendelin-IA site. By default, the input data set is the directory named as the data set.

<dataset> argument: unique reference for the data set. It is optional because if no data set is specified, the current directory will be used as data set reference and directory.

Data set argument can be a path to a directory, then the directory name will be used as data set reference: e.g. ‘ebulk push my_directory/sample/’ --> data set reference will be sample. That directory will be linked to the data set reference, so any future operation on it will refer to that data set, no matter if the directory is moved or renamed.

- New data set ingestion: an ingestion with a new data set reference will create a new data set on the site.

- Data set contribution: ingestion of local changes made on a previously downloaded data set. If no local changes were marked as ready for ingestion (see add/remove commands below), then by default the push command will use all the available local changes.

- Partial ingestion: allows to perform ingestions to a data set without downloading it previously, warning about any file conflict. This feature allows to upload portions of a very large dataset in parallel from different locations/computers.

push options: [-d|--directory <path>] [-c|--chunk <size>] [-s|--storage <storage>] [-cs|--custom-storage] [-a|--advanced]

status [<dataset-path>]

Lists the local changes in data set path. If no data set path is specified, the current directory will be used as data set directory. Lists any new, modified or deleted file in the local data set, indicating if they were marked for ingestion or not.

add <path>

Marks new or modified files in path as ready for ingestion. The path can be a specific file or a directory. Any file in path that has been added or modified will be set as ready, then a future push operation will use the marked files for the ingestion.

remove <path>

Marks the files in path for removal. The path can be a specific file or a directory. Any file in path (deleted or not) will be removed. Then a future push operation will delete from remote data set the files marked as removed. Note: if an existing file (not deleted) is marked for removal, the push operation will also delete it from local data set.

reset <path>

Resets marked files in path. The path can be a specific file or a directory. Any file previously marked for ingestion (add or remove) will be reset.

# OPTIONS

-d <path>

--directory <path>

Allows to use a custom location as data set directory. That directory will be linked to the data set reference, so any future operation on it will refer to that data set, no matter if the directory is moved or renamed.

-c <size>

--chunk <size>

Operations on large files are split into smaller chunks; by default, the size of the chunks is 50Mb. This command allows to set the size (in Megabytes) of the chunks in case is needed.

-dc

--discard-changes

Discards all the changes made in the local data set by downloading the corresponding original files from the remote data set.

-s <storage>

--storage <storage>

Uses the specified storage as input for data set ingestion. The storage must be one of the storages available in the installed Ebulk tool version. The tool will ask the corresponding inputs needed for that storage (like authentication, urls, etc.) and it will perform the ingestion of its content to the remote data set on the site. e.g. the command 'ebulk push my-dataset --storage ftp' allows to ingest the contents of a remote located dataset via ftp.

-a

--advanced

When using -s|--storage option, it allows to configure more advanced aspects of the specified storage, by editing the corresponding configuration file.

-cs

--custom-storage

Allows to use a custom storage as input that is not available yet in the tool. The storage must be one of the available in embulk site: 'http://www.embulk.org/plugins/#input'. The tool will attempt to automatically install the plugin and it will request the user to edit the corresponding configuration file.