How to run ezmsg from the command line?

In order to run an ezmsg pipeline from the command line, you need a python script with the following three things:

1. a dictionary of components (Units or Collections)

2. a tuple of connections between (the input and output streams of) the components

3. a call to ezmsg.core.run() with the components and connections as arguments.

The ezmsg pipeline is a graph defined by these components and connections. The run() function is the entry point to start the pipeline.

When we use the command line to run an ezmsg pipeline, we can choose the address of the backend server (GraphServer) that will manage the pipeline execution. This is useful when we want to run multiple pipelines on the same machine or when we want to run a pipeline on a remote machine.

Run the pipeline using the command line

The ezmsg command line interface exposes extra tools to manage a pipeline that is running in the background on a machine. Run ezmsg -h to see all the available options. Currently, there are five commands available: start, serve, shutdown, mermaid, and graphviz. The first three are used to manage the ezmsg system, while the last two are used for visualising the pipeline.

Step 1. Start the ezmsg system

Paste and run the following in your terminal:

ezmsg serve --address 127.0.0.1:25978

This command will start the ezmsg system and listen for incoming connections on host 127.0.0.1 at port 25978. You can change the address and port as needed - the values here are actually the default that ezmsg will use if you leave out the --address option.

Note

You will notice that with the command serve, ezmsg takes over your terminal in the sense that it runs ezmsg in the foreground, not allowing you to use this terminal for other purposes. In order to continue using the terminal, you will need to open a new terminal (you can have multiple terminals open). If instead you use the command start, ezmsg will be forked to another process allowing you to use the current terminal. This is useful for running ezmsg in the background or as a service:

ezmsg start --address 127.0.0.1:25978

Step 2. Run the pipeline in ezmsg

Now that an ezmsg instance is running one can connect and run a pipeline to it. You will need to change the way you call ez.run() by adding an address to the call.

In your script, change the run() call to include the address from Step 1. For example:

ez.run(components=components, connections=connections, address="127.0.0.1:25978")

Now you can run your pipeline script from a new terminal window.

python3 my_pipeline.py

Step 3. Shutdown ezmsg instance

When you are done with ezmsg, you can shutdown the ezmsg system by running the following command in a new terminal window:

ezmsg shutdown --address 127.0.0.1:25978

This will gracefully shutdown the ezmsg system and release any resources it was using. Make sure to use the same address you used to start and run the ezmsg system.

Note

If you used the serve command to start the ezmsg system, after running the shutdown command, you will see that the terminal where you started the ezmsg system will return to the command prompt.

What happens under the hood?

What actually happens when you run a script with an ezmsg pipeline and a run() function is that ezmsg starts a backend server called a `GraphServer`. This GraphServer manages the operation of any ezmsg pipeline connected to it. The naming comes from the fact that ezmsg pipelines are simply nodes - Units or Collections - connected in a directed acyclic graph (DAG).

The GraphServer is coupled with a `SharedMemoryServer` that allows for efficient data transfer between the different components of the pipeline. This is particularly useful for high-throughput applications where low latency is crucial. It utilises the in-built Python multiprocessing.shared_memory library to handle the data transfer between processes.

The reverse is true too: you can run ezmsg instance (start a GraphServer) without a pipeline, but until you attach an ezmsg pipeline nothing interesting will happen.