What is AWS Lambda
AWS Lamda is a service offered by AWS whereby scripts can be executed serverlessly. Through the AWS Lambda function and the Cloudwatch function, you can schedule your scripts to be executed at a certain time. It is a good option if you do not want to pay for a server to run just few scripts or you have a shared server where you do not have control about libraries or language versions which are installed therein.
Lambda function vs. server
The lambda function has some cons and pros compared to having your own server where you can schedule your Python scripts with cron jobs.
- Cost: you only pay for the time which is used to run the scripts. Depending on the usage chances are that you do not exceed the AWS free tier.
- No maintenance: you do not need to worry about the maintenance of the server.
- Little scalability: if you would like to run a big number of scripts it is better to have your own server as the lambda function has some limitations such as only capacity to run a script maximum for 15 minutes or 300 MB of deployment package (using a layer and zipped resources).
How to upload your first scripts with the Lambda function and the zipped direct uploading?
First of all, we need to adapt our script to make it suitable to be executed with the Lambda function.
- Rename your python script to lambda_function.py.
- Modify your script and put your code under a function called lambda_handler: def lambda_handler(event,context). Only the library imports do not need to be included under this function. This should look like:
A python lambda needs the packages that the script uses. To make them available, we install them locally, zip the script with the packages and upload to lambda.
- We set the script in its own and individual folder.
- If this script needs any other resource such as CSV file, XLSX file, JSON file, etcetera, they need to be set in the same folder.
- We run pip install <package_name> –target=$(pwd) on our terminal in order to install locally on that folder the libraries we need to run that script.
- We zip the content of the folder with zip -r script.zip.
- You might have to run sudo pip install <package_name> –target=$(pwd) because your local machine won’t allow such scripts to be added to folder without root permissions.
- In the case the zipped function exceed the 50MB of weight, you might need to make use of layers to install some of the libraries instead of uploading them with the zipped function.
Once we have our zipped function, we only need to set up the project on AWS:
- In AWS main page, under “Find services”, search for lambda and select it.
- In Lambda’s main page, select Author from scratch and give a name to the function.
- Under Runtime, select the Python version you need to run your script.
- Click on “Create function”.
- In the function´s configuration screen, select “Upload a zip file” and upload the zipped file which has been created previously.
- At the top right corner, click on “Save”.
- Scroll down and adapt the timeout time depending on the time the script usually takes to be executed.
Note: chances are that if you are using an API which works with a Password you would like to store them as environment variables to not hardcode them on the script so that lambda will encrypt them automatically. In such a case, you would need to use the os.environ function on your script to reference them and use the environment variables.
Finally, if we would like to schedule this script to be ran in a regular basis:
- In AWS main page, under “Find services”, search for cloudwatch and select it.
- Inside Cloudwatch´s main page, on the left hand side menu, under events, click on “Rules”.
- Under event source, select “Schedule”.
- Define how often you would like to trigger the function by using cron job expressions or how many times you would like to trigger it daily.
- On the right side, click “Add target” and select lambda from the drop down menu. Select the lambda function which has been created previously.
- Click configure details, give a name for the rule and create the rule.