deweycw/corems

Instructions for running CoreMS in a Docker container

Updated: November 26, 2024 by C. Dewey

Initial Setup

Copy https://***/drive/folders/1TBGmwIIHuL3AO3-WWkYAUgxqoiFXMN1E?usp=drive_link, which contains the files needed to build the CoreMS container, to your Desktop (or any other location on your local machine).

Open a terminal window and navigate to the corems folder you just copied to local machine. For example, if you saved corems to your Desktop, run:

	cd ~/Desktop/corems 

Build the container from the docker-compose.yml file. In the terminal window, run:

	docker compose up -d

When this process completes, you are ready to run CoreMS.

Basics of Running the CoreMS Container

• You can think of the CoreMS container as a lightweight virtual machine that houses all of the code and dependencies needed to run CoreMS. Containerizing CoreMS makes it easy to distribute and run the software without having to install or compile the source code.

• The CoreMS container is started by running docker compose up -d (see above). Once running, you can execute commands and start processes within the container.

• The container is set up so that it shares a directory with your host machine. On your local machine, this is the usrdata directory located in the corems folder that you copied to your computer. Any files you place in this folder will be available within the container. Thus, this is where you will save your data files (.RAW) and Python assignment scripts. In the CoreMS container, this shared directory is located at /CoreMS/usrdata.

• You will use the docker CLI to start an interactive process in the container. For example, if you want to use your container to run a Python script called assign.py saved to your usrdata folder, you would run the following command in your terminal window:

    docker exec -it corems-corems-1 python /CoreMS/usrdata/assign.py
  

• In the above command, docker starts the Docker service; exec tells the Docker service that it needs to execute a command within a container; and -it tells Docker to run exec in an interactive shell. After -it, you provide the container name (corems-corems-1 in this case) followed by the command to be executed (python) and any arguments the command takes (here, the location of the Python script is the only argument, /CoreMS/usrdata/assign.py)

• So if you want to start an interactive bash shell within your container, you would run:

    docker exec -it corems-corems-1 bash
  

• And if you wanted to run a bash script called run.sh saved to `usrdata', you would run:

    docker exec -it corems-corems-1 /CoreMS/usrdata/run.sh
  

Data Processing Workflow

The corems folder downloaded from the above link includes example Python and bash scripts for running calibration and assignments on Thermo .RAW files, as well as for creating feature lists. You can use these as templates for creating your own scripts. The CoreMS container uses the latest version of CoreMS. Your assignment scripts will need to be compatible with this version.

Move data files

• First, copy your .RAW data files into the usrdata folder on your host machine.

Calibration

• For calibration, you will need to generate reference files that will be used during the assignment. To do this, you can use cal-gen.py.

  • If the instrument was calibrated before your analyses and you expect the mass error range to fall between -4 and 4 ppm, you do not need to modify this script prior to running.

  • However, if the instrument calibration was off during your measurement and you expect a mass error range significantly shifted from 0 ppm (e.g., between 14 and 18 ppm), you will need set the expected range by changing the mzerror_range parameter accordingly (line 274 in cal-gen.py).

  • You will also need to set the time range and interval window that you would like to analyze (see lines 276-278 in cal-gen.py).

  • Once you have modified these parameters, you can run the calibration generator:

      docker exec -it corems-corems-1 python /CoreMS/usrdata/cal-gen.py
    

  • This will produce a .ref file for each raw file in usrdata as well as a plots showing the the points selected as calibrations masses. The selection process is iterative, and the script generates a .png showing the calibration masses for each iteration.

  • The script will also write a .csv file called error_range.csv that contains the actual error range for each raw file.

  • The script will also generate a log with information about creation of the reference files. The log will be saved to write_calfiles.log. The log records whether the creation of reference file was successful, and if there was an error associated with the reference file creation, it will be saved here.

  • After cal-gen.py has run successfully, you can move on to running the assignment script.

Assigning molecular formulas

• After you have generated reference files for mass calibration, you can proceed with formula assignment.

• Copy or modify assign.py, which is located in usrdata. This script will define the assignment parameters for the formula search.

• There are several important parameters you will need to modify to define your assignment criteria. It is recommended that you modify the values in assign.py according to your needs. These are the essential parameters. You might want to modify other parameters that are not described below.

  • The first of these are the parameters that define the acceptable mass error range (in ppm) that you wish to allow for your assignments. This range will depend on the instrument used to collect the data. For Tribrid Orbitraps, a range of +/-1.5 ppm is acceptable. For 21T, a range of +/-0.3 ppm would be used.

      # for Orbitrap IQX
      MSParameters.molecular_search.min_ppm_error = -1.5
      MSParameters.molecular_search.max_ppm_error = 1.5
    

  • Next, you will need to set the noise threshold.

      MSParameters.mass_spectrum.noise_threshold_log_nsigma = 7
    

  • Finally, you will need to set the DBEs, elements, and valences allowed in the search:

      mass_spectrum.molecular_search_settings.min_dbe = 0
      mass_spectrum.molecular_search_settings.max_dbe = 16
      
      mass_spectrum.molecular_search_settings.usedAtoms = {
      	'C': (1, 40),
      	'H': (4, 80),
      	'O': (1, 16),
      	'N': (0, 8),
      	'S': (0, 2),
      	'Na': (0, 1)
      }
      		
      mass_spectrum.molecular_search_settings.used_atom_valences = {
          'C': 4,'13C': 4,
          'H': 1, 
          'O': 2, '17O': 2, '18O': 2,
          'N': 3, '15N': 3, 
          'S': 2, '34S': 2, 
          'Na': 1, 
          'Fe':3
      }
    

• After you have defined the criteria for the formula search, you will need to define the range and interval you wish to analyze. These parameters are found in the if __name__ == '__main__' function at the bottom of the assign script:

    	if __name__ == '__main__':
    	    data_dir = r'/CoreMS/usrdata/'
    	
    	    interval = 2
    	    time_min = 0
    	    time_max = 30
    	    times = list(range(time_min, time_max, interval))
    	    
    	    f_raw = [data_dir + f for f in os.listdir(data_dir) if f.endswith('.raw')]
    	    
  

• If you are running the Python script (as opposed to the bash script to parallelize assignment), make sure that f_raw is defined as above (it is defined directly below times).

• You can now run the assignment script:

    	docker exec -it corems-corems-1 python /CoreMS/usrdata/assign.py
  

• From here, we can proceed to calculating dispersity, performing QC checks, and assembling the feature list.

Dispersity calculation, QC checks, & feature list assembly

• After all your .RAW files are assigned, you can proceed to calculating dispersity, performing QC checks, and assembling the feature list. Each of these tasks will be performed with the coremstools Python package, which is installed in the CoreMS container.

• The usrdata folder contains an example script for performing these tasks, called feature-list.py.

  • This script assumes that you have a sample list in usrdata with a column called File that contains the name of each .RAW file in your data set (e.g, RMB_***_BATS03_5m.raw). If you do not have such a file, you will need to create one.

  • Comments in feature-list.py describe the functions of each line of code.

  • To run the script, execute the following command:

      docker exec -it corems-corems-1 python /CoreMS/usrdata/feature-list.py		
    

  • Depending on the size of your data set, its execution may take some time. When it is finished, you will have a feature list for your data set.