The complexity of writing responses

It's been a while since I posted anything to my blog. I thought I would write this article though because it has been an issue that has had a lot of discussion among key members of the swift on server community.

We recently released version 2.0 of Hummingbird the swift server framework. During its development the thing that has probably taken the most of my head space is how to return an HTTP response with a streamed payload within the structured concurrency paradigm.

To provide some context, each request to the server is processed by a middleware stack and then an endpoint that returns a response which will be processed by the middleware stack in the reverse order. One of the key features is that each middleware must be able to process the response payload as it traverses back down the stack.

The example route I'm going to use to demonstrate the issues will return a response where the payload consists of 10 chunks of data, each 1K of random bytes, with a one second pause between sending each chunk.

AsyncSequence

The first tool in the box that most people will reach for in this situation is an AsyncSequence of byte buffers. It seems obvious they allow us to provide a stream of data over time. In middleware you can process the contents of the response payload by mapping the results of one AsyncSequence onto another.

But in this situation it is not the right tool. You need to return from your endpoint handler back to the server level of your application to read your AsyncSequence and write the results out to the underlying transport, but you also need a task to feed the AsyncSequence with buffers, but if you have already returned from your endpoint that task cannot be generated in a structured way. Below is an example demonstrating the issue.

func stream(request: Request) async throws -> Response {
    let (stream, cont) = AsyncStream.makeStream(of: [UInt8].self)
    // the only way I can feed the AsyncStream is with an unstructured
    // task.
    Task {
        for _ in 0..<10 {
            try await Task.sleep(for: .seconds(1))
            let buffer = (0..<size).map { _ in UInt8.random(in: 0...255) }
            cont.yield(buffer)
        }
        cont.finish()
    }
    return Response(status: .ok, body: stream)
}

Response body writer with closure

The second solution is to use a closure that accepts a writer type for the response body. The endpoint returns the closure and at the server level this closure is called with a writer type that writes data to the underlying transport.

func stream(request: Request) async throws -> Response {
    let responseBody = ResponseBody { writer in 
        for _ in 0..<10 {
            try await Task.sleep(for: .seconds(1))
            let buffer = (0..<size).map { _ in UInt8.random(in: 0...255) }
            try await writer.write(buffer)
        }
    }
    return Response(status: .ok, body: responseBody)
}

Using this method it is easy to add shortcuts to ResponseBody to make it easy to return single buffers, sequences of buffers etc

extension ResponseBody {
    public init(buffer: ByteBuffer) {
        self.init { writer in
            try await writer.write(buffer)
        }
    }

    public func init(contentsOf buffers: some Sequence<ByteBuffer>) {
        self.init { writer in
            for buffer in buffers {
                try await writer.write(buffer)
            }
        }
    }
}

By creating a new writer type that processes your response payload and passes it onto the original writer provided you can process the response body in a middleware as well. For this to work though you need a protocol defining the requirements for your writer type and then accept an existential of this protocol in your response closure so any writer can be used.

let newResponseBody = ResponseBody { writer in
    let newWriter = TransformingWriter(writer)
    try await oldResponseBody.write(newWriter)
}

This isn't strictly true structured concurrency though. The closure you construct in your endpoint function isn't being run while you are in that function. The idea behind structured concurrency is that all the code encapsulated inside a single function has finished by the time you leave that function.

Passing in a response writer

The third solution allows you to write endpoint functions in a completely structured way. We do this by passing the writer into the endpoint function and have the function write the response itself.

func stream(request: Request, responseWriter: ResponseWriter) async throws {
    try await responseWriter.writeHead(status: .ok)
    for _ in 0..<10 {
        try await Task.sleep(for: .seconds(1))
        let buffer = (0..<size).map { _ in UInt8.random(in: 0...255) }
        try await responseWriter.writeBody(buffer)
    }
    responseWriter.finish()
}

By the time we leave this function we know for sure all response writing has finished. It passes all the requirements of structured concurrency. It is the most flexible method and allows us to do more complex things like sending informational status (1xx) responses separately from the main response.

But it also passes a lot of the complexity of writing the response onto the end user. Ergonomically it is more awkward to write than just returning a Response. Error handling is complex, you have to keep track of what parts of the response you have written. You have to differentiate between errors generated that you want to provide responses for eg authentication errors and errors that occurred when writing the response to the underlying transport. Some of this can be caught at compile time with clever usage of non-copyable types, but even so this doesn't reduce the complexity pushed onto your endpoint function.

I don't know of a solution for implementing middleware processing of response payloads using this method though. I haven't looked into it too hard but I imagine it would involve passing in a new response writer type that wraps the original response write.

Hummingbird

So what did I end up going for in Hummingbird? I ended up going for a combination of the third and second methods. I don't think most end users want the added complexity, so at the router level where most users will experience Hummingbird we use a version of method 2. We can hide the complexity of the response writing as it is encapsulated in one type but still allows users to do more complex things.

It is possible to implement method 2 on top of a server using method 3. If you return a response which consists of the response head and body writing closure, you can use a response writer to write the returned head, then pass a response body writer that writes to the response writer into the response body closure. Because of this at the server level we have implemented method 3.

If someone wants to write a server that uses response writers they can still use the Hummingbird server, they'll just need to write their own Router.

You find the Hummingbird implementation here.

Other packages

What are other packages in the Swift on server eco-system doing? There are two other packages I know that are tackling this issue at the moment. Version 2.0 of gRPC currently in development is using a version of method 2. The proposal for v2 of swift-aws-lambda-runtime is considering something similar to method 3. So there is no correct way to do this, it just depends on your preferences.