Intern:File Upload

PANGAEA File Upload Workflow

This a short overview about the current state of file uploads and file archiving.

Upload limits

 * Via ticket: max size attachments 100 MB / max 20 files
 * Via uploader: max size 15 GB per file, no limit in total volume (for curators no limit in file size)

Create data submission ticket
An author creates a submission request: https://www.pangaea.de/submit/

As curator, check submissions: https://issues.pangaea.de/projects/PDI

Request “upload link” for large file uploads
As curator click on “Request file upload”.

Example: https://issues.pangaea.de/browse/PDI-24552

Upload files
Author follows generated link and uploads files to, e.g.

https://issues.pangaea.de/upload/?code=5ef052c2338474.86421682

Web page must be open while uploading. A page reload cancels current uploads. So wait!

Author clicks “Confirm file upload” to comment on submission ticket.

Check uploaded files
As curator, check files and file names e.g. for invalid characters.

Upload files are located on Isilon and can be edited there. This is the source.

can be opened and edited for example in file explorer.

After editing, e.g. file rename, you can check the upload overview:

https://issues.pangaea.de/upload/?code=5ef052c2338474.86421682

You an also connect to the server via ssh or filezilla, see below: Get file list for editorial

Copy files
Until yet, files are located only on the Isilon and not archived. So files must be copied from upload (or staging) area to the archive area (HSM).

Open a command line, terminal or Putty

Have a look to your home directory.

Check and decide where files should be archived on HSM. This is the target.

If the directory does not exists, create it and set permissions.

Use the pcp.sh script located in you home directory to copy files from source to target.

The source files for this example are expected in  and will be copied to the target.

Create file list
A file list is created automatically during the copy process.

Optional: have a look to your home directory.

Get file list for editorial
Download and optionally edit the generated file list (requires VPN connection first). There are three ways:

a) Got to the source directory on the Isilon using your file explorer (probably only works for AWI personal).

There is a dataset-PDI-*.txt file which represent the data matrix.

b) Use Filezilla or similar to download the file from pangaea-im2.awi.de

Conncet to Filezilla with following loqin data:

Server: pangaea-im2.awi.de Username: emustermann Port:22

c) connet via ssh to pangaea-im2.awi.de

e.g.

type:
 * ssh emustermann@pangaea-im2.awi.de
 * passwort

type:
 * cd /
 * cd/isi/pangaea/upload
 * open you PDI-folder

Upload the file list to Editorial
Open 4D and goto Import > Data > Open.