Package managers: Pixi

Put your learning to the test with what you’ve covered so far.

A. General environment

ExerciseI - General knowledge
G.1. What role does a workflow manager play in computational research??
Note

For this exercise, you will need to submit two jobs using the Terminal app.

First job:

  • Enter a job name (descriptive of the task, e.g.: pixi installation)
  • Job settings: selecting a 1 CPU standard node with 3GB memory.
  • Add folders to access while in this job
    • IF you have already run some of the conda exercises already, mount pipesOut folder.
    • Otherwise, create a folder names, pipesOut, on your own drive, that you will use to save any output from the exercises.
  • Once the job is running, make sure your working directory for all exercises is /work/pipesOut and that you have not mount the full personal drive (e.g. YourName###1234).

We will now install pixi and save it to your WD.

# Download
curl -fsSL https://pixi.sh/install.sh | bash

# Move pixi 
mkdir -p /work/pixi
mv ~/.pixi/bin /work/pipesOut/pixi

After installation, create an initialization file to update your shell configuration whenever a new terminal session starts. This step is essential in UCloud environments, without it, Pixi will not be available once the running job is closed. Since the Pixi binary has been moved to a custom directory (which you will be mounting every time you start a new job), the initialization file ensures that its location is correctly added to your PATH each time a new session begins.

setup_pixi.sh
#!/usr/bin

echo 'export PATH="$PATH:/work/pipesOut/pixi/bin""'>> /home/ucloud/.bashrc

source ~/.bashrc

Now, Stop application and resubmit a new job:

  • Job settings: selecting a 1 CPU standard node with 3GB memory.
  • Mount pipesOut folder and double-check you have mounted the right directory (e.g. ls). Do not mount the full personal drive (e.g. YourName###1234)
  • Additional Parameters. Initialization: /work/YourName#1234/pipesOut/setup_pixi.sh
  • Make sure your working directory for all exercises is /work/pipesOut
Important

If your job gets disconnected, make sure to navigate back to your WD before counting with the exercises.

Install Pixi

Open a terminal and install Pixi:

curl -fsSL https://pixi.sh/install.sh | sh

After installation, update your shell configuration so the pixi command is available in future terminal sessions:

echo 'PATH="$PATH:$HOME/.pixi/bin"' >> ~/.bashrc
echo 'eval "$(pixi completion --shell bash)"' >> ~/.bashrc
source ~/.bashrc

Follow these instructions:

Linux/Max
curl -fsSL https://pixi.sh/install.sh | sh
Windows
wget -qO- https://pixi.sh/install.sh | sh

Unsure which command to use in the exercises below? Check Pixi documentation for guidance!

ExerciseExercise: Creating and sharing environments with Pixi

In this exercise, you will create a reproducible software environment using Pixi, install packages from multiple channels, and export the environment so it can also be recreated with Conda.

What is the pixi version you have installed:

Initialize a Pixi Project

Create a new directory, called pipesOut, in your workspace DeiC-KU-L65/Workspaces/<username> and make it your WD. Then, create a new Pixi project and configure it to use the conda-forge and bioconda channels:

pixi init --channel conda-forge --channel bioconda

Pixi creates a file called pixi.toml, which contains the information needed to recreate your environment.

List the files in the directory (ls).

You should now see a new file called pixi.toml.

Install software packages

Ensure you are still in the directory containing pixi.toml, then install a collection of commonly used packages:

pixi add python r r-ggplot2 r-dplyr pandas

Inspect the configuration

Open pixi.toml in your preferred text editor and answer the following questions:

  • How many channels are defined?
  • For how many platform has the environment configured?
  • Can you add another platform?
  • How many tasks are specified?
  • How many dependencies are listed?

Now, use the Pixi CLI to add a new platform so that this environment is compatible with macOS.

pixi project platform add
  • Does pixi.toml file now look like this platform = ["linux-64", "osx-64"]?

Additional channels can be added later if required by specific software packages. We’ll cover how to do this in the exercise below.

Add a Bioinformatics Package

Let’s install a package from Bioconda, fastqc. What is the fastqc version? Use a pixi command to find out!

This demonstrates how Pixi can install software from the same package ecosystem commonly used with Conda.

Compare with Conda

Export the environment to a Conda-compatible file:

pixi project export conda-environment environment.yml

Open the generated environment.yml file and compare it with pixi.toml.

Discuss with your neighbour:

  • Which file do you find easier to read?
  • What information is stored in each file?
  • Can conda manage environments that are compatible with multiple platforms?

Recreate the Environment

A collaborator who receives your pixi.toml file can recreate the environment with:

pixi install

A collaborator using Conda could instead recreate the environment from the exported file:

conda create -f environment.yml

Create a new pixi environment

Let’s create an environment for a new project. Copy the pixi.toml file from below into the new project director, which you could named, project1 and make it your new WD.

pixi.toml
[workspace]
channels = ["conda-forge"]
platforms = ["linux-64"]

[dependencies]
python = ">=3.10"

Can you install snakemake in this new environment without any errors?

Add the bioconda channel to the project, then install snakemake. Verify that these changes are reflected in the pixi.toml file and select true if that’s the case:

Share your environment

Export the environment to a Conda-compatible file, then, create a compressed archive containing the files needed to reproduce your environment:

zip environment.zip pixi.toml environment.yml

Inspect the archive:

unzip -l environment.zip

You can now share this archive with collaborators, who can recreate the same software environment using either Pixi or Conda.

# Pixi version
pixi --version
# Pixi number of packages
pixi list --json | jq length
# Add macos platform
pixi project platform add osx-64
# Add package fastqc 
pixi add fastqc
# Pixi command, fastqc version
pixi run fastqc --version!

Conda does not natively manage a single environment definition that targets multiple platforms at once in the same way tools like Pixi do.

mkdir project1; cd project1
# Add the bioconda channel to new env
pixi project channel add bioconda
pixi add snakemake

For this exercise, set the project1 directory as your WD. Let’s keep modifying/adapting the environment you just created to suit our requirements.

One advantage of Pixi is that a single environment can be made reproducible across different operating systems. Check which platforms are currently supported by your project and add support for Linux, macOS, and Windows. Then, verify all platforms have been added.

Open pixi.toml and find the platforms section.

  • Why is a multi-platform environment useful when sharing software with a broader community?
  • What challenges might arise if a package is not available on all platforms?
pixi project platform list
# Add 
pixi project platform add linux-64 osx-64 osx-arm64 win-64
# Verify all platforms have been added: 
pixi project platform list

So far, we have created a single environment containing all required software. In practice, projects often require multiple environments for different purposes.

For example:

  • A default environment containing only the software required to run an pipeline or analysis.
  • A test environment containing additional tools used for testing and validation.
  • A development environment containing debugging and development utilities.
  • A GPU environment containing libraries that require GPU support.
  • A CPU environment that can run on systems without GPUs.

Pixi allows multiple environments to coexist within the same project while sharing dependencies whenever possible. Choose the challenge below that best matches your own research needs. Create a new directory, for example, project2, for the new environment.

Challenge 1: CPU & GPU environments

Many machine learning projects need to support both systems with GPUs and systems without GPUs. To support this, you can define multiple environments within a single Pixi project, where each environment installs only the dependencies it needs.

A possible setup could look like:

default (CPU)
├── python
├── pandas
├── scikit-learn
└── pytorch (CPU)

gpu
├── python
├── pandas
├── scikit-learn
└── pytorch-cuda

With this approach, the same project can run anywhere while automatically selecting the appropriate dependencies. Could you build a pixi environment with that setup? Remember to create a new directory and initialize a pixi project.

Check the hint if you need some help!

  1. Initialize the pixi project
  2. Add common dependencies (shared by all environments)
  3. Define two environments in the pixi.toml file.
    • default (CPU-only)
    • gpu (with GPU support)
  4. Use features to manage the GPU-specific dependencies (pytorch-cuda in our setup)
pixi.toml
[environments]
default = { features = [] }
gpu = { features = ["gpu"] }

[feature.gpu.dependencies]
# empty for now
  1. Install the correct PyTorch version in each environment

Challenge 2: Default & testing environments

Imagine you are developing a bioinformatics pipeline. The default environment includes only the software required to run the pipeline (e.g., Snakemake and core tools), ensuring it remains lightweight and reproducible for users. At the same time, you maintain a separate testing or development environment that includes additional tools for validating and improving the pipeline, such as testing new tools.

The following pixi.toml illustrates this idea:

pixi.toml
[workspace]
channels = ["conda-forge"]
platforms = ["linux-64"]

[dependencies]
python = ">=3.10"

[environments]
default = { features = [] }
dev = { features = ["dev"] }

[feature.dev.dependencies]
# empty for now

Modify the configuration so that snakemake is included in the default environment and pytest is included in the development (dev) environment. Then, use Pixi commands to verify that both tools can be executed successfully. Try executing the different environments (e.g., checking the versions of the different tools) to confirm that each tool is only available in the correct environment.

Get an interactive shell inside Pixi ro run all commands installed in either dev or default environemnt.

pixi add --feature <name> <package>
pixi run --environment

Create a directory, and initialize a pixi project with two channels. Then, add common dependencies shared by all envs:

mkdir -p project2; cd project2
pixi init --channel conda-forge --channel bioconda

pixi add python pandas scikit-learn
# Add CPU version of PyTorch (default environment)
pixi add pytorch

Configure environments and features Make sure you edit the pixi.toml to add the two different envs and declare the features you will be using for the multi-environment settings.

pixi.toml
[environments]
default = { features = [] }
gpu = { features = ["gpu"] }

[feature.gpu.dependencies]

This sets up:

  • default: uses only base dependencies
  • gpu: activates the gpu feature

Add GPU-specific dependencies Then, make things ready for the GPU env:

pixi project channel add pytorch nvidia
pixi add --feature gpu pytorch-cuda

CUDA-enabled PyTorch is only installed for the GPU feature:

Finally, pixi install.

Below are several commands that can be used to install the required software. Some of them are alternative approaches that achieve the same result. If you are checking the solution, focus on using only the commands you need and make sure you understand how they differ.

mkdir -p project3; cd project3
touch pixi.toml 
# edit the pixi.toml by copying the example pixi.toml from the instructions

pixi project channel add bioconda
pixi add snakemake
pixi add --feature dev pytest

# Make sure both environment are installed
pixi install
pixi install --environment dev

# Running one of the envs or testing tools installed in only one env
pixi run --environment dev pytest

# genomedk/laptop (not UCloud)
pixi shell --environment dev 
# OR
pixi run --environment dev bash