Messing with Swift AWS Lambda Runtime (Part 2)

Last week I posted an article about my initial experiences with the Swift AWS Lambda runtime. In that article I promised a second article discussing a second Lambda I was developing/converting from Node.js. Well here it is.

SES Email Forwarder

The Simple Email Service (SES) is an AWS service that can be used to send and receive emails. It doesn't have storage capabilities but one thing it is useful for is forwarding emails sent to one domain onto another. SES doesn't do this out of the box though. To do this a Lambda is required to process the email and then send it on to the forwarding email address. The basic flow is setup SES to accept emails from a domain. When an email comes in it is saved to an S3 bucket and a Lambda is invoked which takes the email from the S3 Bucket, edits some key headers and forwards it on. Up until this point I had been using the Node.js Lambda https://github.com/arithmetric/aws-lambda-ses-forwarder to do this for me. Since my SNS to Slack Lambda had been quite successful I felt I could take on something a little more complicated and produce a Swift version of the SES forwarder.

AWS SDK Swift

I'm the main developer on AWS SDK Swift so I am quite keen to provide examples of the new Lambda runtime using it. For the Lambda runtime to really shine it needs access to other AWS services and that is what AWS SDK Swift provides. The email forwarder seemed like the perfect example. It needs S3 to get objects from an S3 bucket and then needs SES to send the email onto the forwarded addresses.

Oh no!

I hit a barrier almost immediately. If I use v4.x or earlier of the SDK it will not compile with the Lambda runtime due to a symbol clash. If I import the target AWSLambdaEvents it has a namespace enum called S3. The S3 library in AWS SDK Swift is called S3 and the main class in that library is also called S3. So when I create an S3 object it doesn't know which S3 I'm talking about. Normally you can resolve this by prefixing the framework name. But S3.S3 confuses the compiler as well, returning a 'S3' is ambiguous for type lookup in this context error. This is mainly the fault of AWS SDK Swift and is resolved in the alpha version of AWS SDK Swift v5.0. The framework names have been prefixed with the letters "AWS". The compiler isn't confused by AWSS3.S3.

So if you want to use the AWS SDK Swift with the Lambda runtime you are best using the 5.0 alpha version. It's got so much cool new stuff in it maybe you should anyway. Because we are using the alpha version, some of the API's may change before the 5.0 release and the code in this article might go out of date.

Setup

Before we start, the code for this Lambda can be found here.

I'm going to assume you have read my previous article. We will setup a similar structured project to the SNS to Slack Lambda from that article except the Package.swift will need to include the AWS SDK Swift dependencies.

...
    .package(url: "https://github.com/swift-aws/aws-sdk-swift", .upToNextMinor(from: "5.0.0-alpha.4"))
],
targets: [
    .target(name: "SESForwarder", dependencies: [
        .product(name: "AWSLambdaRuntime", package: "swift-aws-lambda-runtime"),
        .product(name: "AWSLambdaEvents", package: "swift-aws-lambda-runtime"),
        .product(name: "AsyncHTTPClient", package: "async-http-client"),
        .product(name: "AWSS3", package: "aws-sdk-swift"),
        .product(name: "AWSSES", package: "aws-sdk-swift")
...

This is the SESForwarderHandler class conforming to EventLoopLambdaHandler. It holds the http client plus the S3 client and the SES client.

struct SESForwarderHandler: EventLoopLambdaHandler {
    typealias In = AWSLambdaEvents.SES.Event
    typealias Out = Void

    let httpClient: HTTPClient
    let s3: AWSS3.S3
    let ses: AWSSES.SES

    init(eventLoop: EventLoop) {
        self.httpClient = HTTPClient(eventLoopGroupProvider: .shared(eventLoop))
        self.s3 = .init(region: Region(rawValue: Lambda.env("AWS_DEFAULT_REGION") ?? "us-east-1"), httpClientProvider: .shared(self.httpClient))
        self.ses = .init(httpClientProvider: .shared(self.httpClient))
    }
    ...

Unfortunately the S3 client doesn't pick up the default region properly so you have to provide it (AWS SDK Swift bug to be fixed). The SES client will pick it up correctly. The input type In is typealiased to SES.Event which can be found in the library AWSLambdaEvents. This is the event sent from SES to the Lambda.

Handle function

The basic handle function is as follows. It is a Swift NIO chain of functions, each being called after the other. I will expand on the individual functions after. The SES.Event structure has support for multiple messages but for this example we'll assume only one message has been sent. In practice I have never seen more than one message sent per event but that isn't to say that is the case always.

func handle(context: Lambda.Context, event: In) -> EventLoopFuture<Void> {
    let message = event.records[0].ses
    let recipients = getRecipients(message: message)
    guard recipients.count > 0 else { return context.eventLoop.makeSucceededFuture(())}

    context.logger.info("Fetch email with message id \(message.mail.messageId)")
    return fetchEmailContents(messageId: message.mail.messageId)
        .flatMapThrowing { email in
            return try self.processEmail(email: email)
    }
    .flatMap { email -> EventLoopFuture<Void> in
        context.logger.info("Send email to \(recipients)")
        return self.sendEmail(data: email, from: Configuration.fromAddress, recipients: recipients)
    }
    .map { _ in }
}

Get Recipients

We call getRecipients which converts the recipients in the SES.Message into the email addresses we want to forward this email to. Currently this uses a map from email address to array of email addresses defined in the configuration.swift file. If there aren't any forwarding addresses we are done and we quit.

Fetch email

Now we know who we are going to be sending our email to we need the original email. SES has already saved the email to S3 using the message id as the key. We fetch the email contents from the S3 bucket. The function calls s3.getObject and transforms its response into the body member of the response which contains the raw email.

func fetchEmailContents(messageId: String) -> EventLoopFuture<Data> {
    return s3.getObject(.init(bucket: Configuration.s3Bucket, key: Configuration.s3KeyPrefix + messageId))
        .flatMapThrowing { response in
            guard let body = response.body?.asData() else { throw Error.messageFileIsEmpty(messageId) }
            return body
    }
}

Process email

We cannot just send on this raw email. SES would reject it. We need to edit the headers to allow for it to be sent. The raw email contents is sent to the processEmail function. This ends up being a mess of string parsing, which isn't pretty code, so I'll leave it out the article. You can go find it here. I break the email into header and body as I am only processing the header. The header is processed line by line. We need to replace the from header. SES cannot send emails from addresses it hasn't verified. All addresses belonging to the domain you are receiving emails into are considered verified. If your domain is "mydomain.com" it is best to change the from header from John Smith <johnsmith@test.com> into John Smith <donotreply@mydomain.com>. The email still looks like it is from John Smith. To ensure you can reply to John Smith add a reply-to header John Smith <johnsmith@test.com>. We then remove a number of headers (return-path, message-id, sender and any header containing dkim-signature) as they would be invalid in the forwarded email. Combine the newly processed header with the body and return this as our processed email.

Send it

Now we have our processed raw email we can send it. This is a simple call to SES. We use the list of recipients we got from getRecipients as the destination addresses and the donotreply@mydomain.com as the from address.

func sendEmail(data: Data, from: String, recipients: [String]) -> EventLoopFuture<Void> {
    let request = AWSSES.SES.SendRawEmailRequest(destinations: recipients, rawMessage: .init(data: data), source: from)
    return ses.sendRawEmail(request).map { _ in }
}

Building and deploying

As with the SNS to Slack Lambda detailed in the previous article I have setup a series of shell scripts to build and deploy the Lambda. This time there are a few more steps. There is a configuration.swift file to edit and because the Lambda interacts with other AWS services a policy document which also needs some details. This is dealt with in more detail in the README for the project.

Also once the Lambda is up and running, if you haven't already, you will need to setup SES to receive emails for your domain and create the rules sets to save to S3 and invoke your Lambda. Again there is more detail on this in the project README.

Error checking

Given forwarding email is quite an important task. Getting it wrong can mean missing that all important email. If the email forwarder fails to forward an email due to some unforeseen circumstance, I want to know. It is possible to link reported errors in a Lambda to an SNS Topic. In the AWS dashboard for the SES forwarder lambda if you press "Add Destination". A Destination configuration dialog appears. If you link the "On failure" condition to an SNS Topic. You can use the SNS to Slack Lambda from the previous article to have email forwarding errors posted in Slack.

Conclusion

I think when people think of serverless computing they think of it in terms of server based computing and how they compare. Lambdas can be more than that though. In the past two articles I have demonstrated how you can glue different services together with a little bit of business logic inside a Lambda function and make the two work together. Many AWS services have links to run Lambdas based off events, which means you can respond to any of these events in your own way.