Hugo and Fabric

Python Fabric is a denatured version of ansible and, if you enjoy python, fabric makes a small scale devops strategy a whole lot less painful. Combined with hugo static website generator, fabic makes testing and promoting a website a whole lot easier.

I have struggled in the past with one challenge that made promoting website code from my workbench to dev, staging, and ultimately production tricky. The matter of URL always played havoc when I sent code up from one server to another for testing. Hugo offers a simple solution. When I promote website code it goes through this workflow process:

  • requirements and design
  • code development
  • workbench (my laptop or desktop) testing
  • git add, git commit
  • promote code to dev server(s) - with rsync or git
  • test dev website
  • promote code to staging
  • test staging website
  • promote to production server(s)
  • test production website

If I am working on code for a website it resides in development directory with the website domain name for the base directory… for example:

domain = "companionway.net"
base_dir = $HOME/dev/www/{domain}/

This works cleanly for me. In hugo config.toml file you have to declare the url so I give it the full domain name ie:

baseURL = "http://companionway.net/"

The problem, of course, manifests with promoting code to another server at a level that is not the fully qualified domain (FQD) production url. That is where hugo allows the -b [your_base_url] option to save the day. To show an example, I have my development server document root as /var/www/html and each website I am working on has its name off of that. My development server(s) and raspberry pi (cheap) so they have clever names like rasp01, rasp02, etc. So the code for “companionway.net” resides at raspXX:/var/www/html/companionway.net. The hugo build command of hugo -d "https://raspXX/companionway.net/ allows me to go to that url to test the code and all the links will work properly. When the code goes to the production level with a fully qualified domain name with a DNS A record the hugo build command simply is hugo and the default baseURL is used from the config.toml file.

The fabric process I use has a lot of steps but might be worth something to someone out there… once gain it comes with my standard “it can be dome better’ clause:

Here is my fabfile.py that allows promoting … read’em and weap …

``` from fabric.api import env,run,local,lcd,cd,sudo,put,get,prompt,reboot,roles,settings,task,warn_only,abort,hide from fabric.colors import * from fabric.contrib.files import exists import subprocess import sys sys.path.append(“/home/geoffm/dev/python/gmodules/”) from dbug import dbug, trace # my own dbug tools import functools from datetime import datetime

dtime = datetime.now().strftime(“%Y%m%d-%H%M”)

env.user = “geoffm”

env.roledefs = { ‘rpis’:[‘rasp1’,‘rasp2’,‘rasp3’,‘rasp4’,‘rasp5’,‘rasp6’,‘rasp7’], ‘ubuntus’:[‘192.168.1.72’], ‘prsnl’:[‘artemis’], ‘rhat’:[‘f-node01’], ‘containers’:[‘u-node01’,‘u-node02’,‘f-node01’], ‘tst’:[‘rasp2’,‘rasp7’], ‘websites’:[‘cos.tst’,‘hype.companionway.net’,‘companionway.net’], ‘dev’:‘rasp5’, ‘stage’:‘rasp2’, ‘prod’:‘rasp1’ } all=env.roledefs[‘rpis’]+env.roledefs[‘containers’]+env.roledefs[‘prsnl’]

Note: @task is just a decorator that adds some features to the wrapped function - ie adds it to “fab -l”

this allows use of namespacing

env.hosts = ‘rasp1’,‘rasp2’,‘rasp3’,‘rasp4’,‘rasp5’,‘rasp6’,‘rasp7’

def help(): # ####### msg = “”” fab sethost:“rpis” udug # will select the host group “rpis” and run the task udug (update->upgrade dist) “”” print(msg) dbug(f”task:{task.doc}“) exit

def test_site(site): # ############## “”” test_site(site) # promts the user to run all tests on the site and respond to verifiy question “”” try: local(‘hugo server -v -b “http://localhost"') # forces the baseURL to be at this location except KeyboardInterrupt: print(f”\n\n\nInterrupted site test for: {site}.\n\n\n”) r = prompt(“Were the results of your testing complete and satisfactory?”, default=“n”) # if r == “y” or r == “Y”: if r: proceed = True else: proceed = False return proceed

def do_title(s,llen=50): # ###########
“”” prints a title block of length llen with supplied string s “”” slen = len(s) margin = “=” * (int((llen-slen)/2) - 1) title = (f”{margin}[ {s } ]{margin}“) title_len = len(title) print(”\n”*3) print(“=” * title_len) print(title) print(“=” * title_len) print(”\n”*1)

@task

def promote(to_level=“where”,stack=“websites”): # ##################################### “”” def promote(to_level=“where”,stack=“what”) default: stack=“websites” eg. def promote(“staging”,“websites”) fab promote:stage,websites # promotes websites to stage platform “”” user=env.user

# user can deploy just one site eg: fab promote:dev,companionway.net
sites_dir = f"/home/{user}/dev/www/"
if stack == "websites":
    sites=env.roledefs['websites']
else:
    sites = [ stack ]

# build the test_approved.flag filename, with basename and extention name for use later
taff_base = "test_approved" # needed for moving the file after a push
taff_ext = "flag"
taff = f"{taff_base}.{taff_ext}"  # alias test_approved.flag file

# loop through site(s)
for site in sites:
    dbug(f"site:{site} user:{user} sites_dir:{sites_dir}")
    do_title(site)
    site_dir = f"{sites_dir}/{site}/"
    with lcd(site_dir):  # change dir to the site to work on
        r = local('pwd')
        cmd = f"test -f {taff}"  # there is a way to do this with a fabric exists call BTW
        # ### turn errors off so script continues through the loop od sites
        with settings(warn_only=True):
            # ### test for existing test-approved.flag file (taff)
            r = local(cmd)
            # dbug(f"r:{r}")
            if r == 0:
                dbug(f"\nFound the file: {taff}\n")
                proceed=True
            else:
                # dbug(f"Failed to find file: {taff}")
                # dbug(f"need to run a site test")
                proceed = test_site(site)
                # ### create test_approved.flag file ### #
            if proceed:
                # ### create a new test_approved.flag file (taff)
                msg = f"\n\n\n{dtime} Test successful. Approved by: {user}"
                cmd = f"echo {msg} > {taff}"
                local(cmd)

                # ### rm public ### #
                local('rm -r ./public')  # remove the /public tree recursively to avoid any left over extraneous files

                # ### run git commands (after /public removed although /public is in .gitignore as well)
                local('git add -A')
                local('git commit')

                # ### hugo set vars then create public ###
                if to_level == 'dev': 
                    trgt = env.roledefs['dev']  
                elif to_level == 'stage': 
                    trgt = env.roledefs['stage']  
                elif to_level == 'prod': 
                    trgt = env.roledefs['prod']  
                else:
                    # trgt = "unknown":
                    print(red(f"Unsure of to_level [{to_level}]... exiting"))
                    exit()

                # hugo build (/public) with specified root/baseURL (domain) name
                # there is a better way to do this ... by declaring the a key:value pair for each to_level:site... this is an easy cop-out
                if to_level == 'dev' or to_level == 'stage':
                    cmd = f'hugo -b "http://{trgt}/{site}/"'  # overides the baseURL setting in config.toml
                elif to_level == 'prod':
                    cmd = 'hugo -v'  # This allows the declared baseURL in config.toml to take effect

                else:  # this should be dealt with above in creat /public block ... but just in case
                    cmd = ""  # clearing cmd just in case
                    print(red(f"Unsure of to_level [{to_level}]... exiting"))
                    exit()
                local(cmd)  # run whatever cmd has been set

                # ### xfer files to promote lvl ### #
                if to_level == 'dev' or to_level == 'stage':
                    # r = exists('/var/www/html/{site}')
                    # dbug(f"r:{r}"); exit()
                    cmd = f'rsync -av ./public/* {trgt}:/var/www/html/{site}/'  
                elif to_level == 'prod':
                    if site == "companionway.net":
                        cmd = f"git push origin master"
                    else:
                        cmd = f'rsync -av ./public/* {trgt}:/var/www/html/{site}/'  
                else:
                    print(red(f"Unsure of to_level [{to_level}]... exiting"));exit()
                    r = True
                    exit()

                r = local(cmd)

                if r.failed:
                    print("\n"*3)
                    print(red(f"Houston, there was a problem with: {cmd} result:{r}"))
                    print(f"output:{r.stdout}")
                    print(red(f"Please investigate... removing {taff}"))
                    local(f'rm ./{taff}')
                    print(red("Exiting..."))
                    exit()

                # ### add msg to test_approved.flag file of msg=f"{site} promoted to {to_level} {dtime}"
                msg = f"{site} promoted to {to_level} {dtime}"
                do_title(green(msg))
                cmd = f"echo {msg} >> ./{taff}"
                local(cmd)

                # ### move test_approved.flag file (taff) to {taff}.{dtime}
                cmd = f'mv {taff} {taff_base}-{dtime}.{taff_ext}'
                local(cmd) 

                # ### tell user to go test site on to_level
                print("\n"*3)
                print('----------------------------------------------------------')
                print(f"Please test [{to_level}]: http://{trgt}/{site}  # in tilix: highlight the link and hit Ctrl-click")
                print('----------------------------------------------------------')
                print("\n"*1)
                prompt("Continue? [Y]")

            else:
                print("\n"*3)
                print('----------------------------------------------------------')
                print(f"You need to research and re-test site: {site}.")
                print('----------------------------------------------------------')
            # end of: if proceed is true or false
        # end of with warn_only=True [encounterred error within script has exit() turned off] so that the loop continues
    # EOLoop
return

@task def sethosts(group=“list”): “”” sethosts(group=list) use: fab sethosts:\“rpis\” udug “”” if group == “all”: env.hosts = all dbug(f”hosts all:{env.hosts}“) return if group == “list”: for k,v in env.roledefs.items(): print(f”{k}={v}“) else: try: env.hosts=env.roledefs[group] print(f”Selected group:{env.hosts}“) except: dbug(f”Unrecognized group[{group}]…“)

@task def getos(): with settings( hide(‘warnings’, ‘running’, ‘stdout’, ‘stderr’), warn_only=True): if run(‘ls /etc/lsb-release’): os = ‘ubuntu’ elif run(‘ls /etc/redhat-release’): os = ‘redhat’ print(os) return os

@task def lxclist(): local(‘lxc list’)

@task def lxcstartall(): env.warn_only=True local(‘lxc start –all’) lxclist()

@task def lxcstopall(): env.warn_only=True local(‘lxc stop –all’) lxclist()

@task def lxcrestartall(): env.warn_only=True local(‘lxc restart –all’)

@task def createlxcnode(name): “”” Use fab createlxcnode:myname “”” cmd=“lsb_release -r | awk ‘{print $2}’” rel = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) print(rel.stdout.read()) # rel = subprocess.check_output(cmd.split(),stderr=subprocess.STDOUT) cmd=f”echo lxc launch ubuntu:{rel} {name}” local(cmd)

@task def updatehosts(): # use hand writtend script in this dir (for now) WIP local(‘./update-hosts.sh’)

def uname_a(): “”” runs: uname -a “”” with settings(warn_only=True): run(‘uname -a’)

def top(): run(‘top -b -n 1 | head’)

def df(): run(‘df -H | grep “dev”’)

def uptime(): run(‘uptime’)

@task def status(): with settings(warn_only=True): uname_a() uptime() df() top()

@task def apt_update(): sudo(‘apt update’)

@task def aptfullupgrade(): apt_update() sudo(‘apt -y full-upgrade’)

@task def udug(): apt_update() aptfullupgrade()

@task def instbasics(): with settings(warn_only=True): r = sudo(‘apt -y install vim git alpine inxi screen lynx xymon-client’) if r.failed: r = sudo(‘dnf install vim git alpine inxi screen ‘)

if you run “fab tst” then this next block fires off for all the hosts listed in containers

this just an example

@task @roles(‘containers’) def tst(): with cd(‘/tmp’): print(“Gathering file names from /tmp”) sudo(‘ls -l’)

@roles(‘rhat’) def dnfupgrade(): sudo(‘dnf upgrade’)

@task def adduser(node, user=“”): “”” use: fab adduser(nodename,username) WARNING: assumes (defaults) username is your username your pub key will be copied over to this account gives sudo privileges to the user adds ssh passwordless login with your puplic key “”” import getpass if user == “”: user=getpass.getuser() # sets user to current user cmd=f’lxc exec {node} – bash -c “adduser {user}”’ from fabric.contrib.console import confirm print(f”\n\n\nPlease confirm: node:{node} user:{user} “) if not confirm(“Continue? “): abort(“Aborting as requested…”) with warn_only(): local(cmd) cmd=f’echo “{user} ALL=(ALL) NOPASSWD: ALL” > ./{user}’ local(cmd) local(“chown root ./{user}”) cmd=f”lxc file push ./{user} {node}/etc/sudoers.d/{user}” local(cmd) cmd=f”cat /home/{user}/.ssh/id_rsa.pub >> ./{user}-key.pub” local(cmd) cmd=f”lxc file push {user}-key.pub {node}/root/{user}-key.pub” local(cmd) cmd=f’lxc exec {node} – bash -c “mkdir /home/{user}/.ssh”’ local(cmd) cmd=f’lxc exec {node} – bash -c “cat /root/{user}-key.pub >> /home/{user}/.ssh/authorized_keys”’ local(cmd)

@task def rpibasics(): “”” install basic apps specific to the needs of rpi like xrdp “”” sudo(‘apt install xrdp’)

@task def webit(): “”” get apache up and running with skel structure “”” sudo(‘apt -y install apache2 libapache2-mod-php ‘) # sudo(‘apt install php-pear php-fpm php-dev php-zip php-curl php-xmlrpc php-gd php-mysql php-mbstring php-xml libapache2-mod-php’) # if you want to rule the world # sudo(‘mkdir /var/www/html/chkit’) put(‘/home/geoffm/dev/www/skel/chkit’,‘/var/www/html/’,use_sudo=True) “’

This seems long and ugly but it works for me solidly.

Enjoy, -g-

Geoff McNamara

“Do not meddle in the affairs of wizards, for they are subtle and quick to anger.” J.R.R Tolkien

blah Elizabeth City, NC https://www.companionway.net