Adding Distribution

Continuing from our previous example, we will now look at how to deploy our program to run on multiple processes.

We achieve this by using Hydro Deploy. Hydroflow+ integrates with Hydro Deploy to automatically construct the topology based on the flow graph. We can create a new file examples/first_ten_distributed.rs with the following contents:

examples/first_ten_distributed.rs

use hydro_deploy::Deployment;
use hydroflow_plus_cli_integration::TrybuildHost;

#[tokio::main]
async fn main() {
    let mut deployment = Deployment::new();
    let localhost = deployment.Localhost();

    let flow = hydroflow_plus::FlowBuilder::new();
    let (p1, p2) = hydroflow_plus_template::first_ten_distributed::first_ten_distributed(&flow);

    let _nodes = flow
        .with_default_optimize()
        .with_process(&p1, TrybuildHost::new(localhost.clone()))
        .with_process(&p2, TrybuildHost::new(localhost.clone()))
        .deploy(&mut deployment);

    deployment.run_ctrl_c().await.unwrap();
}

Most importantly, we specify a DeployProcessSpec, which constructs a Hydro Deploy service for each process in the flow graph. In our case, we use the TrybuildHost service type, which compiles and deploys a Hydroflow+ graph.

We can then run our distributed dataflow with:

cargo run --example first_ten_distributed
[service/1] 0
[service/1] 1
[service/1] 2
[service/1] 3
[service/1] 4
[service/1] 5
[service/1] 6
[service/1] 7
[service/1] 8
[service/1] 9