Manual code-delivery

While dealing with a Python script for data migration and clean-up in Postgres at my company, I needed to shift it to the cloud on AWS. For tasks that don’t frequent my to-do list, I rely on notes. While I’m well-versed in the basics of ssh and scp, occasional lapses in memory prompt me to revisit my notes rather than resorting to online searches.

Side-notes

This is slighlty off-track, but I’d really like to mention couple of nifty life hacks I’ve accumulated over time. Here are a couple of tools I find useful:

  1. Cheat.sh: A quick curl cheat.sh/scp fetches a concise command explanation. It’s a convenient reference for essential information.

  2. ExplainShell.com: When StackOverflow snippets lack detailed explanations, ExplainShell breaks down command calls. If find it a swift alternative to diving into manuals for basic details.

Tools for the job

Returning to the AWS task, the objective was to upload my Python project to an EC2 instance without the complications of Docker containers, CI/CD and things like that.

scp

Initially, I opted for scp:

scp -i ~/.ssh/my_particular_key.pem \
    ~/code/smart_cleanup \
    [email protected]:~/

However, it proved to be a slow, file-by-file process. Also, the classic mistake – I forgot to exclude local directories.

rsync

A more efficient solution emerged with rsync. It offers faster operations due to special algorithm. Featuring options like -e for SSH, --exclude for bypassing unwanted folders, and -a for compression, the command looked like this:

rsync -av -e "ssh -i ~/.ssh/my_particular_key.pem"\
    --exclude "venv"\
    --exclude "__pycache__"\
    --exclude ".git"\
    ~/code/smart_cleanup\
    [email protected]:~/

This approach not only addressed the slow copying issue but also allowed for the exclusion of multiple unnecessary folders like venv. The reason? My development environment is on an M1 Mac (ARM), while the script runs on an amd64 EC2 instance, so different dependencies need to be installed there.

Conclusion

In conclusion, the shift to rsync proved to be a significant enhancement over scp, offering a more efficient and flexible file transfer solution for this particular task. The only drawback you may encounter is the requirement for rsync to be installed on the receiving end.

With these steps, my Python script now resides on the EC2 instance, ready for action. Time to execute and get things done.