Handle your AWS Spot Instance Actions with terminate-notice

You’re running AWS Spot Instances, and suddenly your monitoring system notices that the spot instances are being terminated without workloads being effectively moved to another instance. How can we fix this?

AWS create an entry in the Metadata Service which the node can watch for. This script polls that Metadata entry, and if it finds it, it runs any responding scripts to cleanly move services elsewhere.

How to Install?

Individually

Get the latest release from terminate-notice/terminate-notice

You will also need to install the appropriate actions, which are available from:

Add a Debian Repository

Download the public key and put it in /etc/apt/keyrings/terminate-notice.gpg. You can achieve this with:

wget -qO- https://terminate-notice.github.io/terminate-notice.asc | sudo tee /etc/apt/keyrings/terminate-notice.asc >/dev/null

Next, create the source in /etc/apt/sources.list.d/

echo "deb [arch=all signed-by=/etc/apt/keyrings/terminate-notice.asc] https://terminate-notice.github.io/deb stable main" | sudo tee /etc/apt/sources.list.d/terminate-notice.list >/dev/null

Then run apt update && apt install -y followed by the names of the packages you want to install.

Add a RPM Repository

Download the repo file cd /etc/yum.repos.d ; curl https://terminate-notice.github.io/terminate-notice.repo -LO

Then you can do yum install -y followed by the names of the packages you want to install.

The packages you can install

Currently these are:

  • terminate-notice
  • terminate-notice-announce
  • terminate-notice-datadog
  • terminate-notice-shutdown
  • terminate-notice-slack

After installing the packages

You should configure settings that are installed into /etc/terminate-notice.conf.d/ and then run

systemctl enable --now terminate-notice.service

How to contribute?

Please contribute changes and bug reports in the relevant repository above.

Have a security issue? Please email Jon with details.

Have your own action? Please create a pull request on this repository.

You might find this document from AWS useful.

Thanks

This script was created during my first month at Dice.FM, who very generously permitted me to release this code publically.