Introduction

AWS Lambdas do not run forever, and this can be a problem if you are trying to process data that may run over the allotted max of 15 minutes. But, depending on how you structure your processing, you can trigger another lambda to continue where the other left off.

This can be done by combining the following:

• AWS Lambda context

• AWS Lambda invoke

• Monitoring current index

Note: The examples below are using Javascript and the AWS NodeJS V3 SDK.

Lambda Context

Every lambda has a context (an object containing methods and properties about the lambda), from this context we can pull the time remaining. We do this by calling getRemainingTimeInMillis. In the example below we are checking if the time remaining is less than 10s. If that is true, we are triggering the function again.

const handler = async (event, context) => {
    let index = 0;

    for (index; index < lots.length; index++) {

        // Check time remaining
        if (context.getRemainingTimeInMillis() < 100000) {
            // Invoke another lambda to continue processing
        }

        // Processing
    }

    return processingResult
}

Invoking Lambda

The next step to continue processing is to invoke another lambda. We can do this by creating an invoke method that handles generating the invoke command needed by AWS.

Below is an example of this invoke method. It takes in a simple payload (we will define that in a bit), and configures a client and command. In our case, we are using the invocationType of Event this is because we don’t want to wait for a response from the lambda itself.

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

const asyncLambdaInvoke = async (payload) => {
  const client = new LambdaClient({
    region: 'us-east-1',
    endpoint: '<https://lambda.us-east-1.amazonaws.com>'
  });

  const params = {
    FunctionName: 'longtail-process-function-name',
    InvocationType: 'Event',
    Payload: JSON.stringify(payload)
  };

  const command = new InvokeCommand(params);
  const response = await client.send(command);
}

Now that we have created the invoke method, we need to supply it with the current processing index. This allows us to pick back up where we left off.

Monitoring Processing

Expanding on our initial lambda handler we are adding a check for currentIndex. If the event has one, use that, else set it to 0. This allows us to pick up where the processing left off. We are also adding in calling our invoke method and supplying it with the currentIndex as its payload. When the lambda is invoked it will now contain the index we need to pick up processing from.

const handler = async (event, context) => {
    // If event has currentIndex use that, else 0
    let index = event.currentIndex || 0;

    for (index; index < lots.length; index++) {

        // Check time remaining
        if (context.getRemainingTimeInMillis() < 100000) {
            // Invoke another lambda to continue processing
            await asyncLambdaInvoke({currentIndex})
        }

        // Processing
    }

    return processingResult
}

Conclusion

Using this technique of monitoring the time remaining and the current processing index you can run through larger long-tail processing efforts with AWS Lambda. This simple method can be applied in many other scenarios.

See the full example below

const { LambdaClient, InvokeCommand } = require('@aws-sdk/client-lambda');

const asyncLambdaInvoke = async (payload) => {
  const client = new LambdaClient({
    region: 'us-east-1',
    endpoint: '<https://lambda.us-east-1.amazonaws.com>'
  });

  const params = {
    FunctionName: 'longtail-process-function-name',
    InvocationType: 'Event',
    Payload: JSON.stringify(payload)
  };

  const command = new InvokeCommand(params);
  const response = await client.send(command);
}

const handler = async (event, context) => {
    // If event has currentIndex use that, else 0
    let index = event.currentIndex || 0;

    for (index; index < lots.length; index++) {

        // Check time remaining
        if (context.getRemainingTimeInMillis() < 100000) {
            // Invoke another lambda to continue processing
            await asyncLambdaInvoke({currentIndex})
        }

        // Processing
    }

    return processingResult
}

Happy coding!

David