Talk:Export

Mass download for compilation and visualization


Example: How to create a map of heavy mineral distribution of the Arctic Ocean

0. Install PanGet (http://www.pangaea.de/Software/PanGet), PanTool (http://www.pangaea.de/Software/PanTool) and Pan2Applic (http://www.pangaea.de/Software/Pan2Applic) first.

1. Retrieve datasets with PANGAEA Search Engine (http://www.pangaea.de). For details read the PangaVista Help (http://wiki.pangaea.de/wiki/PangaVista)! In this example we create two queries:
 * 1.1 hornblende or amphibole or epidote or garnet or pyroxene or clinopyroxene or orthopyroxene or opaque and latitude > 60
 * 1.2 Quartz or Feldspar or kalifeldspar or plagioclase or calcitev or dolomite and latitude > 60

2. Create a download script with PanGet. For details read the PanGet manual (http://wiki.pangaea.de/wiki/PanGet)!

3. Start the download script. For details read the PanGet manual.

4. After downloading all datasets from PANGAEA verify the files with PanTool.
 * 4.1 Select the data folder (Ctrl+O) and choose "Verify PANGAEA dataset ..." from the "Special tools" menu (F11). PanTool writes the file "VerifyResults.txt".
 * 4.2 Open this file with an editor (e.g. Excel) and report lines marked as error to info@pangaea.de. We will fix the errors as soon as possible.
 * 4.3 When the PANGAEA team has fixed the errors start step 3 again.

5. Excursus
 * 5.1 Open the option dialog of the program PanTool (Ctrl+P) and set the filename format to "zz%a_%N.txt".
 * 5.2 Open five random datasets with an editor like Excel and try to understand the general structure of PANGAEA datasets.

6. Extract parameter lines from all data files
 * 6.1 Select the data folder with PanTool (Ctrl+O).
 * 6.2 Choose "Search one string ..." from the Menu "Basic tools" (F5). Search for "PI:". You get a file named "ResultOfSearch.txt".

7. Make a list of all parameters, methods and comments
 * 7.1 Select the file "ResultOfSearch.txt" with PanTool (Ctrl+E) and choose the tool "Search and replace one string ..." (F8). Search for " * " and replace this with "^t" (^t will be replaced by the tabulator).
 * 7.2 Open "zz1_ResultOfSearch.txt" with Excel and produce a list with three columns (Parameter, Method, and Comment). Sort by parameter, method, comment. Add a column after "Comment" and give "x" as value (This will fix an Excel optimize problem).
 * 7.3 Select "zz1_ResultOfSearch.txt" with PanTool and choose the tool "Delete double lines". The new file (zz1_zz1_ResultOfSearch.txt) contains all parameters and methods once.

Open it with Excel and see the problems: alike parameters have different names (e.g. "Hornblende" = "Amphibole"), alike methods are described with different precision (e.g. "X-ray diffraction, silt fraction" vs. "X-ray diffraction"), comments can be expressed differently too (e.g. "<63 micron" vs. "less than 63 µm"). So, datasets should be harmonized before you can create a map!

8. Harmonize parameter names, methods and comments in all datasets with search and replace
 * 8.1 Find the problems in "zz1_zz1_ResultOfSearch.txt" and create a "search and replace database". First column: string to search for, second column: string to replace it with. Name this file "HeavyMinerals.rdb". See the example file:

Column 1                                               Column 2 Hornblende                                             Amphibole Hbl                                                    Amph X-ray diffraction TEXTUR, clay fraction                X-ray diffraction X-ray diffraction, bulk mineralogy                     X-ray diffraction X-ray diffraction, silt fraction                       X-ray diffraction Counting, binocular                                    Heavy mineral analysis Element analyser CHN, LECO (URL: http://www.leco.com/) Element analyser CHN

9. Apply the PanTool tool "Search and replace many string at once ..." (Shift+F8) to the data folder.

10. Create a data file Start Pan2Applic and set options (Ctrl+T). On card "All" set: "set GEOCODE range", "display DOI", "show method", "show comment". On card "Format" choose "ASCII/tab-separated". Create a new folder and drag all "zz1_*" to this folder. Drag the new folder onto the Pan2Applic window and follow the instructions.