Making generators

Previous post: Functional async.

• Simple generators

  • Iterators
  • Blocking generators
  • Simple async generators
• Maintainable generators

  • Generator state
  • A more structured construction
  • Functional combinators

Simple generators

Iterators

In functional programming, iterators replace loops. Rust provides many helper methods (called adapter methods) for iterators such as map, filter, step_by, … . But to apply this style of programming, you need some base iterators to start with.

The base (or leaf) iterators are the ones that are actually important. They are provided by the core language or a foundational user crate. Usually, it is a bad idea to implement your own iterators.

In case you decide to implement a new iterator anyway, have a look at the definition of an iterator:

pub trait Iterator {
    type Item;
    fn next(&mut self) -> Option<Self::Item>;
}

Blocking generators

Although you could directly implement next for your own data-types, it might be more straightforward to use a generator to create an iterator for you. Generators are functions that output an anonymous type that implements the Iterator trait. The body of a generator has yield X statements that represent the result of an invocation to next() being Some(X).

Remark: There is nothing special or new about generators in Rust. They have existed, for example, in JavaScript for many years.

A good crate in Rust for writing generators is genawaiter.

let generator = gen!({
    yield_!(10);
});
let xs: Vec<_> = generator.into_iter().collect();
assert_eq!(xs, [10]);

Using this crate, the generator variable is actually more than just a generator (something that can be converted into an Iterator). It is also a coroutine. See other posts on this blog to know more about coroutines in Rust.

Remark: The gen! and yield_!-macros will become built-in the core Rust language in the coming months. The gen! simply becomes the gen keyword for code blocks. Inside gen-blocks you can use yield. For now, you need nightly to use this.

Simple async generators

The iterators generated by the previous kind of generators is blocking (or synchronous ). The asynchronous (non-blocking) variant of a blocking iterator is a stream (an asynchronous iterator).

You can just keep using the genawaiter crate and add await-points in the body of your gen! generator definition. You need to, however, enable the futures03 feature.

(From the documentation)

async fn async_one() -> i32 { 1 }
async fn async_two() -> i32 { 2 }

let generator = gen!({
    let one = async_one().await;
    yield_!(one);
    let two = async_two().await;
    yield_!(two);
});

let items: Vec<_> = stream.collect().await;
assert_eq!(items, [1, 2]);

An alternative is the async-stream crate (which has been updated more recently). The generators written with its stream! macro are always asynchronous streams (in contrast to genawaiter which also supports iterators).

Remark: Asynchronous generators will be included in stable rust in the coming months. As of May 2025 you still need to switch to a nightly compiler version and enable unstable features.

Maintainable generators

Generator state

While creating generators and putting yield statements, you will quickly run into very complex code. Code making use of yield may be hard to maintain. You will need a place to store the state of the generator. In the brute-force approach you just add local variables outside the main loop of your generator body.

For example, you could have something like:

let generator = gen!({
    let mut state = Some(0);
    loop {
        do_something();
        state.update();
        yield_!(state.method());
    }
});

When state becomes bigger, it is time to switch to another approach.

A more structured construction

The most straightforward alternative for async generatorss is to use the futures::stream::unfold function. This function stores the state explicitly in its first argument and updates it incrementally with a closure (returning a Future).

use futures::{stream, StreamExt};

let stream = stream::unfold(0, |state| async move {
    if state <= 2 {
        let next_state = state + 1;
        let yielded = state * 2;
        Some((yielded, next_state))
    } else {
        None
    }
});

let result = stream.collect::<Vec<i32>>().await;
assert_eq!(result, vec![0, 2, 4]);

A synchronous version of this (for normal Iterators) can be found in the crate itertools: itertools::unfold.

An example from the crate docs:

let mut fibonacci = unfold((1u32, 1u32), |(x1, x2)| {
    // Attempt to get the next Fibonacci number
    let next = x1.saturating_add(*x2);

    // Shift left: ret <- x1 <- x2 <- next
    let ret = *x1;
    *x1 = *x2;
    *x2 = next;

    // If addition has saturated at the maximum, we are finished
    if ret == *x1 && ret > 1 {
        None
    } else {
        Some(ret)
    }
});

itertools::assert_equal(fibonacci.by_ref().take(8),
                        vec![1, 1, 2, 3, 5, 8, 13, 21]);

Functional combinators

One problem with unfold is that less well-suited for scenarios in which you need to combine or split several streams/iterators, for example while making your own functional combinators. The constructor unfold seems to be most appropriate when you need to create iterators or streams from scratch.

Writing your own combinators gives you the tools to recombine streams in a more functional or declarative way. Have a look at the combinators defined in futures::stream to see what a common combinator implementation looks like.

Remark: See other posts about combinators for information on how to build your own declarative combinators such as custom variants on merge or flatten.