Synapse to NDA Search

The following tool is provided to offer an alternative to searching NDA for legacy PsychENCODE data.

Requirements: 

• User must have an account on NDA and permissions to download PsychENCODE data from NDA. Instructions for applying for access can be found here.

• NDA API Usage (opional): Comfortability with basic scripting. Instructions for installing and setting up nda-tools can be found here. 

Search tool Usage:

• There are three options for finding data. By default, all of the files in PEC and CommonMind are listed. Alternatively, you can select from a list of common projects. These are projects as they were listed in Synapse, and when available, also contain the description text from Synapse. Finally, if you know specific Synapse ids for files or projects, you may search directly for those ids. The search results will include all child files found for that project, or for the Synapse id(s) entered. Multiple ids may be entered at once, and the results will be inclusive of all. 

• There are two options for downloading results. One includes a full table listing the file paths on NDA, associated NDA experiments, and s3 addresses. Alternatively, you can choose to download just the s3 addresses. This results in a file that can be used directly with nda-tools to download the selected files. 

Suggested Usage (with nda-tools):

1. Log into NDA and create a download package from the entire PEC collection

   • Navigate your browser to the main PEC collection C5032.

   • Click "ADD TO CART" at the bottom of the page

   • Wait for the filter cart to update (top of the page, will update from "Filter Cart(0)" to "Filter Cart(1)" 

   • Once the filter cart has populated, click on the filter cart and select "CREATE PACKAGE/ADD TO STUDY"

   • On the next page, select "CREATE DATA PACKAGE"

   • Enter your desired package name, and be sure to check the box next to "Include associated data files"

   • It will take some time for the data package to be completed. You can see the status of the package by clicking on  "My Account" on the top right of the page, followed by selecting "Data Packages" from the drop down menu.

   • Take note of the "ID" for your package. You will need to reference this in order to download files.

2. Use the tool below to find files of interest. 

   • Either use one of the search functionalities, or download the entire table and choose files by path or filetype.

   • Once you have your list of selected files, you will need the s3 addresses listed in a text file, one per line. The tool has an option to download just the s3 addresses of the results to feed directly into the next step.

3. Download the files

   • nda-tools offers options to download files either to your filesystem, or to a cloud location via the downloadcmd tool. Note that there may be more options for the tool than are currently listed in the github repo instructions. downloadcmd -h should list all available. 

   • To download only the files listed in your search results file to your filesystem, use the following command:

downloadcmd -dp <package id> -t <s3FileList.txt> -d </destination/directory>