Fixing Flipped or Reversed Web Images

The tool of choice is mogrify which come from the imagemagic package on your linux platform.

I had one image that looked fine when I displayed it using display tools in linux but when placed into a webpage the image would be reversed.

»

Author image Geoff McNamara

Python's all() function

Python all() function Much of my coding efforts are directed towards analyzing financial data specifically targeting potential cash secured puts or covered calls. Recently, I had to refine some market overview displays. I needed a listing of the current market “gainers” and market “losers”. There is one site which I scrape using panda read_html function to grab a hadful of tables. My problem arises from two factors. One is there are no labels or titles to the tables and they can vary. »

Managing Vim and Essential Plugins

Our last post discussed using fisa-vimrc and appreciating the simple installation and ease of use. This post will explore, briefly, some tweaks and a short “how to” for each of the loaded plugins. This exercise will be worth your time if you have not explored these tools before. »

Vim-Plug Quick Setup

I want you to try this .vimrc setup.

I have had my own .vimrc for years. I recently came across fisa-vim-config and ended up dropping my extensive .vimrc, adopting the one above and adding a few tweaks to make it friendly to me. I was actually looking for more ALE-fixers when I found this gem.

It is simple, self installing and very powerful due to the vim Plugins that are installed and loaded.

»

Author image Geoff McNamara

Quick Note about fabric and pipenv

I have posted several articles about python fabric and yet I failed to warn you about the dangers of version and module chaos.

Initially, fabric came as a python2 only application. They added support for python3 later. Here is the version I am using in a virtual environment (more about that in a minute). From within my pipenv project directory with pipenv shell loaded this is what I get running fab –version

fab --version
Fabric3 1.14.post1
Paramiko 2.4.2

Here is how I created my virtual environment. I have used almost all of python’s virtual environment tools (and there a quite a few). And there has been a steady progression of improvements in those tools. My current goto for a python virtual environment is pipenv. It is simple, clean, and almost fool proof which is essential for me.

»

Author image Geoff McNamara

Python fabric - Dynamically Discover Hosts

Python fabric is a library tool for executing ssh commands remotely and responding as you desire. I use python fabric for promoting code and web content. Essentially it is my devops tool to satisfy the single most important devops rule: never log into a server! I use this over ansible or some of the other devop tools because fabric only requires knowning one langauge and that is python.

My network changes a lot as I take servers down or stand them up frequently. Normally that would require changing the environment roles (host lists) in my fabfile.py with all my changes. In the past I set aside certain IPs in my network for each role. For example all my raspberry pies get named before I ever introduce them to the network. Using the option –skip_bad_hosts with fabric is one way to do this but I really want more control.

»

Author image Geoff McNamara

Replace PHP with WSGI python

This post is a brief description of replacing PHP code with python using WSGI (Web Server Gateway Interface). The journey here comes from how I monitor my servers (mostly raspberry pi’s). I used a PHP script on all my servers that delivered the hostname and basic server information and I use xymon monitoring to poll that PHP script on all the servers and test to see if the hostname is contained in the returned page. My focus lately has been on python and my interest in PHP has waned. The goal was to replace my PHP script with a python script. The end result provides a simple means of delivering dynamic web pages. I will be using bottlepy as the wsgi web-framework. My development was done on an ubuntu 18.10 (“cosmic” release). »

Using a generic git pre-commit hook

test_it && commit_it || reject_it

Using a pre-commit script when committing code makes sense and fits with every respectable dev-ops model. The goal is to have respectable code before it lands in a repository. Git makes this very easy through the use of “hooks”. If you look in any of your git project directories you will find these files in the ./.git/hooks/ directory: »

Author image Geoff McNamara

Vim ALE

VIM is a standard for me. My UNIX career hung on my “vi” savvy which I learned appreciate from mentors who encouraged me to learn the tips and tricks of using vi. My learning curve with vi/vim has never leveled off, but rather, always gives me new features and tricks that energize me again. The latest vim “trick” is ALE (Asynchronous Lint Engine). »

Vim Scriptable Templates

Dynamic vim Templates

Vim (or vi in the anceint past) has been my partner in getting things done. Like an rich tool it has lots of features that go untapped or undiscovered. A few days ago I decided I wanted a script within a tempate to help build the “front matter” to my markdown (.md) files.

The front matter looks like this:

+++
author = ""
comments = true
date = "2018-07-15 14:45:19"
draft = false
image = ""
share = true
# slug = 
# tags = [ "tag1", "tag2" ]
title = "vim_scriptable_templates"

+++

The challenge was to have vim write in the date for me in the needed format. And if you use hugo as your static web site generator as I do you may have found that it will silently ignore a new markdown file if the date is not formated correctly. It some testing to discover that but the bottom line is that I needed a way to script it for to avoid mistakes.

Research to me to this tip source: vim embedded scripts

»

Hugo and Fabric

Python Fabric is a denatured version of ansible and, if you enjoy python, fabric makes a small scale devops strategy a whole lot less painful. Combined with hugo static website generator, fabic makes testing and promoting a website a whole lot easier.

I have struggled in the past with one challenge that made promoting website code from my workbench to dev, staging, and ultimately production tricky. The matter of URL always played havoc when I sent code up from one server to another for testing. Hugo offers a simple solution. When I promote website code it goes through this workflow process:

»

Author image Geoff McNamara

Fabric_vs_ansible

I have wrestled with ansible for too long. It is an awesome tool but you need to learn the syntax of three different languages (ansible, yaml, and maybe some python) to hold it down and control it. Every time I go to use it I have to learn how to do what I need to do. But the most frustrating aspect of ansible when learning to do something new is that your first attempt rarely succeeds - at least for me. So you resort to “force programming” to get through it. By “force programming” I mean you try this and that and then something else until it works, and I end up with a hodgepodge of debug lines commented out. »

Author image Geoff McNamara

How this theme evolved

Announcing a new project: newspaper-hugo-subtheme

First credit goes to Silke V at codepen.io name “silkine” [http://www.silkevoigts.de/]. Original source code can be found here: https://codepen.io/silkine/pen/jldif

I had a coworker who had a non-profit business blog that wanted to offer a newsletter to her clients. So I got interested in finding an HTML format that gave an old traditional newspaper style and Silke offered a solution that provided a strong simple css base. Currently (2017) I find the hugo cms system to be the easiest most portable static web building application freely available. The challenge was to create a full newsletter style theme as a subtheme within an existing theme structure. So I mocked up a ported version of Silke’s newspaper css into a hugo “section”.

An example of this subtheme can be seen here: [newspaper-hugo-subtheme]

This project is a drop in offering to any existing hugo themed structure which will produce a newsletter format in an “article(s)” hugo section. A user only needs to change the title, city, and state varbiable that are stored as front matter in the ./content/articles/_index.md file. Hugo uses this file natively to discover section (in this case “articles”) specific data. All posts to the newsletter subtheme should be created in this ./content/articles/ directory as markdown files. Running hugo new articles/mynewarticle.md will create a new file with the default front matter pulled from ./archtypes/articles.md.

This project becomes a drop in addition to any hugo theme to add a newspaper type subtheme section.

»

Author image Geoff McNamara

MagicMirro2 and fauxmo

Warning: what follows is a cascade of issues and lessons learned… but just plain fun!

The goal

This project is designed to provide a reliable MagicMirror2 on a Raspberry Pi fitted with a 5 inch touchscreen that doubles to provide browsing if needed. This comes from a desire to have a quick aide in the kitchen that reports standard information when needed or a browser for looking up and displaying recipes etc.

The result is a raspberry pi with one wire that stands on its edge without a case and has a 5 inch display. The display can show any of the modules provided for magicmirror2 on command to Alexa or stop magicmirror on command. The rasbian desktop along with an inboard keyboard pad and touch “pen” allows you to launch a browser (or any other program) and navigate to any url. When the desktop is idle a screensaver comes up with anynumber of pleasing animations - selectable and can be rotated with screen blanking possible as well. The backlight of the screen is enough to act as a soothing nightlight. All in all, I am very pleased (and so is my wife) with the final result of this project.

There is a LOT of information here and lots of steps but it is worth it and you will definately learn lots of useful tricks. I am sure there are many improvements that could be made.

Kitchen Pi Image:1

Sorting IPs

I scan my home network frequently. There are a good number of smarthome devices, raspberry pies, phones, laptops, PCs etc. Using nmap to quickly scan a network is too slow for me - and it is a challenge to parse the output for easy reading (of course I would use awk to do that). So I use arp-scan (available in most repositories). So output might look like this: »

Author image Geoff McNamara

Raspberry Pi and Xrdp

I decided to have some remote desktop fun with one of my raspberry pies. Remote desktop for the raspberry pi (or any linux machine) can be done so many different ways - but the easiest from my perspective is xrdp. It is available really on any linux distro and works with Windows remote desktop protocol (kinda). It requires that you install xrdp and tightvncserver (under the covers it uses a vnc server - thightvncserver is probably the best choice here). Then as a client on my laptop I use the very good remote desktop client called remmina; again it is availalbe on almost any linux distro. »

Sometimes It Is The Obvious

This was one of those “problems” that nagged me for many hours until the obvious dawned on me. I wrote a python script to grab the temperature and humidity from a DHT22 sensor and then write the output with the proper syntax to send to my monitoring program [xymon]. The wrapper bash script ran every 5 minutes and used a redirection to write out a file ie:

*/5 * * * * /usr/local/bin/temphum.sh >/tmp/temphum.dat

The wrapper script grabs output and sends it to the server. Having this wrapper script lets me run the python script independently for testing or for curiosity.

»

Always Use Visudo

“familiarity breeds contempt”

I get very comfortable with using the vim editor (take cover… flame way ensuing) so comfortable that using any other editor leaves me just a bit unnerved. I constantly use it to update sudoers. The best way to modify sudoers is to add a supplimental inclusion file under /etc/sudoers.d/whatevername - so the other day I needed to add an entry to allow the user for my simple monitoring program (xymon) to be able to run “sudo ufw status” without a password and report back to the monitor server.

I fired up

sudo vim /etc/sudoers.d/xymon

and started to add the needed line:

xymon ALL = NOPASSWD: /usr/sbin/ufw status

but I got halfway through and could not remember the exact location

»

Recursively Replace Text

Recently I had to do a find and replace text on multiple files recursively. I had to look up how to do it to remind myself of the exact syntax. If I had to look it up, you have to suffer seeing it in my blog post… [smile]

find ./ -name \*.md -print0 | xargs -0 sed -i 's/header-img/img/'

Be careful here. I used the -0 option to deal with any filename that have spaces in the name BUT it really requires that the find command contain the -print0 option otherwise things will not operate in the manner expected.

»

Switched to Hugo

I am swtiching from jekyll to hugo for flat file for a Content Managment System (CMS). alt text

Here is why:

  • It comes as a binary file that runs on pretty much any platform. A single file… without a list of dependencies
  • I could stop with the last reason and it would be enough… but I have to say that the file tree used in hugo is simple, and simple is beautiful. As Einstein once said:

“Make things as simple as possible, but not simpler.”

  • It is written in googles go language… so it is fast and portable.
  • It is closely tied to git so it can easily be used on github and for version maintence.
  • it builds a clean directory [./public/] for the full site deploy and this can be rsync’d to where ever you like. or … just git push it to hithub and use something like netlify.com to host from your github!
  • No frustrating version incompatiblities (or at least very minimal) - unlike jekyll which seemed to break on me every few months.
  • hugo is relatively mature and has a huge developer backing.
  • configurable and changed without challenges
  • lots of themes to choose from
  • as the Hugo websites states - it makes the web fun again…
  • Did I mention that it is a single executable file?

Reference: gohugo.io

Enjoy -g-

»

Converting Images

I have fought with trying to remember the syntax for Imagemagik’s convert command for years. I finaly caved in [from my normal position of “if you can’t do it from the commandline, then you are a wimp” and sought out a gui to help me quickly convert images to use on a website. Or to just roll a picture 90 or 180 degrees. The easy solution and a standard ubuntu repository file is “converseen”. »

shell/bash uppercase arguments

To make a long story short… #!/bin/bash PATTERN=$(echo $1 | tr [:lower:] [:upper:]) # or PATTERN=$(echo $1 | tr [a-z] [A-Z]) # or PATTERN=$(echo $1 | awk ‘{print toupper($0)}') echo "Your $1 argument is now uppercase: $1" To make a short story long… Just to let you know a little more about me and reveal a few more shells tricks along the way. I am a very conservative investor - always have been but the low interest rates over the recent past has forced me to try a squeeze a little more out of the stock market also known as a “a gentleman’s gambling parlor of greedy people”. »

The GrovePi kit

Check out GrovePi These kits make working with the Raspberry Pi GPIO pins so simple and clean. I just ordered one of these kits and I will try to write up a review after I work it over. If you are interested in building a quick and easy raspberry pi for experimenting with the “Internet of Things” (aka IoT) platform - this board and kit looks like it will get you there in minutes. »

How to print the remainder of a line using awk

#How to print remainder of line using awk

eliminate the first X columns…

Here is your challenge you have a list of cronjobs and you want to just execute the scheduled backup command with its arguments for a test run.

Using one command line, how would you do it?

Here is a sample root cronjob and I want to run the first entry with its arguments as a test run:

# m h  dom mon dow   command
 0 16  *   *   * /data/share/home/geoffm/dev/utils/bak-it.sh -i /usr/local/etc/include.lst -e /usr/local/etc/exclude.lst -t /data/bak/`hostname` -s 7 >/dev/null 2>&1
  5 0  *   *   * /data/share/motion/arch-files.sh >/dev/null 2>&1
*/15 *  *   *   * /data/share/home/geoffm/grab-pic.sh >/dev/null 2>&1
####### below are WIP or OLD ########
# 16 6  *   *   * /usr/sbin/lynis -c --auditor "automated" --cronjob --quiet >/dev/null 2>&1
#*/5 *  *   *   * /usr/local/bin/watch-proc.sh -f /usr/local/etc/watch-proc.conf -n geoffm@localhost >/dev/null 2>&1
#*/5 * * * * /data/share/home/geoffm/ping-pong.sh "ping -c2 192.168.1.55" "ssh 192.168.1.1 reboot" >/dev/null
#  7 * * * /usr/local/sbin/postfix_report.sh &> /dev/null
»

Author image Geoff McNamara on #awk,