# Running QDsim in a cluster
QDs can be run in a cluster environment. This is useful when you have want to create a large dataset and you want to run QDs on multiple nodes to speed up the process. In this tutorial, we will show you how to run QDs in a cluster environment using the `mpi4py` library.

1. Clone the QDsim repository
```sh
cd /scratch/${USER}
git clone https://gitlab.com/QMAI/papers/qdsim.git
cd qdsim
git checkout develop # checkout your desired branch (optional)
```
2. Create and activate a virtual environment
```sh
module load miniconda3
conda env create -f environment.yaml
conda activate qdsim
```

3. Install the QDsim package
```sh
cd ..
pip install qdsim
```

4. Create a Python script to run your desired simulations, named `usermodule.py`
```python
import qdsim
from qdsim import QDDevice
from qdsim import QDParallelSimulator

# Create and customize a quantum dot device instance
device = QDDevice()

# .....
# Your code here to customize the device
# .....


# Simulation setups (List of dictionaries)
# Each dictionary represents the setup for one or more simulations on a single device
# The dictionary for each setup must contain all the keys listed below
# Any number of setups are allowed (i.e. List can have any number of elements)
simsetups = [
    {
        'qd_device': device,                # QDDevice object
        'simulation_type': 'Electrons',     # string, 'Electrons' or 'Holes'
        'configurations': [
            # List of Tuples. Each Tuple represents the configuration for a single simulation of the charge stability diagram for the device.
            # Configurations are specified with a Tuple with the following elements:
            #   - sensor_location: tuple representing the x and y coordinates of the sensor; only 1 sensor per config
            #   - scanning_gate_indexes: tuple representing the indexes of the gates to scan; only 2 gates per config
            #   - gates_voltages: tuple containing the voltages to apply to each gate; can be set to None
            #   - fixed_voltage: float representing the voltage to apply to all the gates that are not being scanned; can be set to None
            ((2, 1), (0, 1), None, 1.5),
            ((1, 2), (2, 3), (1.89, 1.56, None, None), None),
        ],
        'n_points_per_axis': 60,            # int, number of points to probe on the x- and y-axis
        'v_range_x': [-5, 20],              # list, minimum and maximum voltage to probe on the x-axis
        'v_range_y': [-5, 20],              # list, minimum and maximum voltage to probe on the y-axis
        'solver': 'SCIP',                   # str, solver to use for the simulation ('SCIP' or 'MOSEK')
        'save_voltage_occupation_data': True,
        'save_sensing_data': True,
        'save_current_data': True,
        'save_charge_stability_diagram_plot': True,
        'charge_stability_diagram_plot_settings': {
            'cmapvalue': 'RdPu',            # str, default from matplotlib colormaps
            'gaussian_noise': False,        # bool, whether to add gaussian noise to the plot
            'white_noise': False,           # bool, whether to add white noise to the plot
            'pink_noise': False,            # bool, whether to add pink noise to the plot
            'plot_potential': False,        # bool, if False plot current data, if True plot potential data

            # Noise parameters
            'gaussian_noise_params': None,  # None or list of 2 elements [avg, std_dev]
            'white_noise_params': None,     # None or list of 2 elements [min, max]
            'pink_noise_params': None,      # None or list of 2 elements [f_max, amplitude_range]

            'plot_format_extension': 'png'  # str, file format for plot (e.g., 'png', 'pdf', 'jpeg')
        }
    }
]

# Run the parallel simulation for the charge stability diagram
QDParallelSimulator.parallel_simulate_charge_stability_diagram(simsetups, '')
    
```

5. Create a bash script `job.sh`,  to run the Python script in the cluster (the example below is taken for Delft Blue cluster).
```sh
#!/bin/bash
#SBATCH --job-name="qdsim1"
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=2            # 1 if in serial mode 
#SBATCH --cpus-per-task=1     
#SBATCH --mem-per-cpu=16G
#SBATCH --partition=compute
#SBATCH --account=research-as-qn

# Load modules:
module load miniconda3

# Set conda env:
# see https://doc.dhpc.tudelft.nl/delftblue/howtos/conda/
unset CONDA_SHLVL
source "$(conda info --base)/etc/profile.d/conda.sh"

# Activate conda, run job, deactivate conda
conda activate qdsim
mpiexec -np 2 python usermodule.py
conda deactivate
seff $SLURM_JOB_ID
```

6. Submit the job to the cluster
```sh
sbatch job.sh
```