Digital Insanity Magazine by Bruno Amaral

Import Instagram Posts Into Hugo

Thu, 25 Jun 2020 13:35:26 +0100

I do love Instagram and the cool ways we can connect through with, see other cities, people we miss. It’s just hard to deal with having my photos kind of trapped there.

It is possible to export them, but that process takes a while. So this last weekend I took some time to improve a script that I used to extract my photos from there and onto this site.

They are in the pages of the Photos section.

Here is a step by step on what I did for those more tech-inclined.

Some things to take into account before moving on:

We are assuming the website is ran by Hugo.
You need some knowledge in using the Command Line of your computer.
Some of the information below includes what is useful for me, your needs will vary.

Instaloader

Instaloader is a tool to download pictures (or videos) along with their captions and other metadata from Instagram.

This command line tool does the heavy lifting, it authenticates you to Instagram, queries your profile and posts and saves everything to a nice file structure.

What we are going to do is take those files, extract data from them, and save it as environment variables. Those will then be picked up by the Hugo Archetype we defined for new posts in that section.

Archetype for Instagram Posts

When you create new content, Hugo uses a template for that file. That template is called an Archetype.

There are some Hugo functions you can use in these templates, and we are using getenv that allows us to use environment variables in your new post.

Take a look:

---
categories:
date: {{ getenv "post_datetime"}}
description:
draft: false
resources:
- src: {{ getenv "post_image" }}
  name: "header"
layout: instagram
location:
  - link: {{ getenv "google_maps_link"}}
  - latitude: {{ getenv "latitude" }}
  - longitude: {{ getenv "longitude" }}
stories:
subtitle:
title: "{{ getenv "post_title" }}"
tags: {{ getenv "post_tags"}}
---

{{ getenv "post_content" }}

You can tweak this file to add more information, as long as you also do the same in the script that puts it all together. More on that later.

Import and update script

The script became a little more complex than what was to be expected. To keep things in order, it’s hosted on GitHub.

Steps

Edit the config section of the script
Run the setup with ./instagram-posts-from-instaloader.sh setup
Execute the update ./instagram-posts-from-instaloader.sh run-update

Running setup, you will be asked for your username and password. Instaloader will setup a directory with your username where it will store your photos alongside the metadata it found.

We are keeping this directory untouched, read from it, and save the information we need as an environment variable. The full code is below and on GitHub.

Inside the update-post function we are reading the json data with xzcat and jq. If you want to add more data to the exported post, this is where you should look.

After this is done, we check if the post exists and if not, run hugo new ... and copy any jpg or video files that may be included.

With this done, your new posts should be ready. Run hugo server to make sure it’s working as you expected.

If you wish to add or improve any bit of the script, feel free to submit a merge request on the GitHub page.

Let me know in the comments if this was useful. 😀

#!/bin/zsh
## USAGE

### 1. Run the setup with `./instagram-posts-from-instaloader.sh setup`
### 2. Execute the update `./instagram-posts-from-instaloader.sh run-update`

## Config

### Where is your blog located?
blog="/Users/brunoamaral/Labs/Digital-Insanity"
### What is your Instagram Username?
instagramUser="brunoamaral"
### Where are we saving the output from Instaloader? (do not include the username)
directoryWithInstagramPosts="/Users/brunoamaral/Labs/instagram/"

### Where do you want photos saved to? Relative to ./content 
instagramdir="instagram"

# Dependency check

command -v instaloader >/dev/null 2>&1 || { echo >&2 "I require Instaloader but it's not installed. Aborting."; exit 1 }
command -v xzcat >/dev/null 2>&1 || { echo >&2 "I require xzcat but it's not installed. Aborting."; exit 1 }
command -v jq >/dev/null 2>&1 || { echo >&2 "I require jq but it's not installed. Aborting."; exit 1 }
command -v hugo >/dev/null 2>&1 || { echo >&2 "I require hugo but it's not installed. Aborting."; exit 1 }

setup() {
  cd $directoryWithInstagramPosts; 
  instaloader --fast-update --geotags --login=$instagramUser $instagramUser
}

run-update(){
  find "$directoryWithInstagramPosts/$instagramUser" -maxdepth 1 -name '*.json.xz'  -exec ./instagram-posts-from-instaloader.sh update-post {} \;
}  >&2

update-post() {
  file=$(basename $1)
  dir=$(dirname $1)
  imagefinal=${file//'json.xz'/'jpg'}
  content=`xzcat "$1" | jq '.node.edge_media_to_caption.edges[0].node.text'`

  declare -a tags;
  tags=$(echo $content | grep -o '#[[:alpha:]|[:alnum:]]*')
  jsontags=$(printf '%s' "${tags[@]}" | jq -R . | jq -s .)
  jsontags=${jsontags//[$'\t\r\n'|$' ']}
  jsontags=${jsontags//'#'/}
  date=`cut -d'_' -f 1 <<< "$file"`
  time=`cut -d'_' -f 2 <<< "$file"`

  export post_image=$imagefinal;
  export post_datetime=$(echo $date"T"${time//-/:}"+00:00")
  export post_tags=$jsontags
  if [[ $content = null ]]; then
    export post_slug
    export post_title=$date
    export post_content=" "
  else
    slug=$(echo "$content" |  iconv -c -t ascii//TRANSLIT | sed -E 's/[~\^]+//g' | sed -E 's/[^a-zA-Z0-9]+/-/g' | sed -E 's/^-+\|-+$//g' | sed -E 's/-$//g' | tr A-Z a-z )
    export post_slug=${slug:0:25}
    title=${content:0:50}
    export post_title=${title//'"'}
    export post_content=$content
  fi

  locationfile=${file//.json.xz/_location.txt}

  grepLocation=$directoryWithInstagramPosts
  grepLocation+="$instagramUser/"$locationfile

  location=$(grep -s 'maps.google.com' $grepLocation)
  if [[ ! -z "$location"  ]]; then
  echo "found $location"
    export google_maps_link=$location 
    locationClean=${location/'&ll'/}
    export latitude=$(cut -d'=' -f 2 <<< "$locationClean" | cut -d'\' -f 1 | cut -d',' -f 1)
    export longitude=$(cut -d'=' -f 2 <<< "$locationClean" | cut -d'\' -f 1 | cut -d',' -f 2)
  fi

  destination="$blog/content/$instagramdir/$date-$time$post_slug/"
  if [[ ! -f "$destination/index.md"  ]]; then
    hugo new $instagramdir/$date-$time$post_slug/index.md
    cp ${1:r:r}(*.jpg|*.mp4) $destination
  else
    echo 'post exists: '$post_title
  fi

}  >&2

if [[ $1 = "setup" ]]; then
  setup
fi

if [[ $1 = "run-update" ]]; then
  run-update
fi

if [[ $1 = "update-post" ]]; then
  update-post $2
fi

Lisa Fotios

Creating a Gallery Component for the Hugo Static Site Generator

Sat, 24 Aug 2019 18:11:34 +0100

Some of the posts that you see in this blog feature an image gallery. If you want to see a working example, there is one in the post about the Portuguese National Tile Museum.

National Tile Museum

http://www.museudoazulejo.gov.pt/ Any place is a good place to start. I chose this one for several reasons, one of which my curiosity to see …

That gallery is using Photoswipe and simple css for the mansonry effect.

If you want to replicate, here’s how it works.

1. Set frontmatter

resources: 
- src: "gallery/*.jpg"
  name: gallery-:counter
  title: gallery-title-:counter

We are using the shortversion, where the name and title and simply derived from the number of images in the folder.

2. shortcodes/gallery.html

The shortcode is used in the following format:


{{< gallery folder="gallery" title="Escher in Lisbon" >}}

The shortcode’s logic is simple. First, use .Page.Resources.Match to get all the resources whose name matches the gallery folder string. It’s important the resource name starts with the folder’s name.

{{ with .Get "title" }}
  <h4>{{ . }}</h4>
{{ end }}
  <div class="gallery" itemscope itemtype="http://schema.org/ImageGallery">
  {{- range (.Page.Resources.Match (printf "%s*" (.Get "folder"))) }}
  {{ $thumbnail := .Resize "320x" }}
    <figure itemscope itemtype="http://schema.org/ImageObject" class="image gallery-item">
    <a href="{{ .Permalink }}" itemprop="contentUrl" data-size="{{ .Width }}x{{ .Height }}" >
      <img src="{{ $thumbnail.Permalink }}" itemprop="thumbnail" alt="galleryImage" class="galleryImage" />
    </a>
    <figcaption itemprop="caption description">
      <span itemprop="copyrightHolder"></span>
    </figcaption>
    </figure> 
  {{ end }}
  </div>

No gimmicks here, just the standard photoswipe html and making sure it is required using .HasShortcode.

{{- if .HasShortcode "gallery" -}}
  <!-- Root element of PhotoSwipe. Must have class pswp. -->
  <div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">
  <!-- Background of PhotoSwipe.
     It's a separate element, as animating opacity is faster than rgba(). -->
  <div class="pswp__bg"></div>
  <!-- Slides wrapper with overflow:hidden. -->
  <div class="pswp__scroll-wrap">
    <!-- Container that holds slides.
      PhotoSwipe keeps only 3 of them in DOM to save memory.
      Don't modify these 3 pswp__item elements, data is added later on. -->
    <div class="pswp__container">
      <div class="pswp__item"></div>
      <div class="pswp__item"></div>
      <div class="pswp__item"></div>
    </div>
    <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
    <div class="pswp__ui pswp__ui--hidden">
    <div class="pswp__top-bar">
      <!--  Controls are self-explanatory. Order can be changed. -->
      <div class="pswp__counter"></div>
      <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>
      <button class="pswp__button pswp__button--share" title="Share"></button>
      <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>
      <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>
      <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
      <!-- element will get class pswp__preloader--active when preloader is running -->
      <div class="pswp__preloader">
      <div class="pswp__preloader__icn">
        <div class="pswp__preloader__cut">
        <div class="pswp__preloader__donut"></div>
        </div>
      </div>
      </div>
    </div>
    <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
      <div class="pswp__share-tooltip"></div>
    </div>
    <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
    </button>
    <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
    </button>
    <div class="pswp__caption">
      <div class="pswp__caption__center"></div>
    </div>
    </div>
    </div>
  </div>
  <link rel="stylesheet" href="/plugins/photoswipe/photoswipe.css">
  <link rel="stylesheet" href="/plugins/photoswipe/default-skin/default-skin.css">

  <script src="/plugins/photoswipe/photoswipe.min.js"></script>
  <script src="/plugins/photoswipe/photoswipe-ui-default.min.js"></script>
  <script src="/plugins/photoswipe/initphotoswipe.js"></script>
  <script>initPhotoSwipeFromDOM('.gallery');</script>
{{- end -}}

4. CSS

I am using the Photoswipe’s CSS and columns to display thumbnails:

/**/
/* GALLERY */
/**/

.gallery{
  width: 100%;
  display: block;
  margin:0 auto;
}

/* Extra Small Devices, Phones */ 
@media only screen and (min-width : 480px) {
  .gallery {
    column-count: 1;
    column-gap: 3px;
  }
}
/* Small Devices, Tablets */
@media only screen and (min-width : 768px) {
  .gallery {
    column-count: 3;
    column-gap: 3px;
  }
}

/* Medium Devices, Desktops */
@media only screen and (min-width : 992px) {
  .gallery { /* Masonry container */
      column-count: 4;
      column-gap: 3px;
  }
}

/* Large Devices, Wide Screens */ 
@media only screen and (min-width : 1200px) {
  .gallery{
    column-count: 5;
    column-gap: 0.2rem;
    width: 100%;
  }
}


.gallery-item { /* Masonry bricks or child elements */
    background-color: #eee;
    display: inline-block;
    margin: 0 0 0.2rem;
    width: auto;
}

5. And that’s it. Hope it helps!

soragrit wongsa

Content Groups in Hugo

Sat, 03 Mar 2018 00:41:57 +0000

If you are not a webdesigner or programmer, you probably want to skip this post. Click on https://brunoamaral.eu/categories/ to find something that is more worthy of your time.

For everyone else, a little background first. My this blog is built using Hugo and has the following sections:

Posts
Pages
Stories
Instagrams

And I wanted to track these elements as content groups in Google Analytics, using Google Tag Manager. Because I am not at all sure which of these drives more traffic and interest to the site.

You can configure 5 levels of content groups, and I wanted to configure the following:

	Posts	Pages	Stories (Section)
Index 1	Post	Page	Stories
Index 2	nil	nil	StoryName
Index 3	PostTitle	PageTitle	StoryArticleTitle

Configure your Hugo theme

To get this information out of Hugo do this, I used the classes in the body tag.

_default/single.html

<body class="single contentgrouptype-{{ .Section }} contentgroupname-{{ with .Params.stories }}{{ index . 0 | urlize }}{{ end }} article article-title-{{ .Title |urlize }}">

_default/page.html

<body class="single page contentgrouptype-{{ .Section }} contentgroupname-{{ .Title | urlize }}">

Since the page for Instagrams has its own layout, I took advantage of that:

Instagrams/instagram.html

<body class="single instagram contentgrouptype-instagrams contentgroupname-{{ .URL }}" id="instagram">

I should be using the title of the instragram post, with urlize. The reason I am not, is because my script still does not collect that information.

taxonomy/category.html

<body class="page contentgrouptype-category contentgroupname-{{ .Title | urlize }}">

taxonomy/story.html

<body class="page subhomepage contentgrouptype-story contentgroupname-{{ .Params.story_title | urlize }}">

taxonomy/tag.html

<body class="page contentgrouptype-tag contentgroupname-{{ .Title | urlize }}">

Configure Google Tag Manager

First, we need to install Google Analytics from the Google Tag Manager tag.

What you are seeing in {{ }} are variables that we need to create and configure to tie this together.

In Google Tag Manager, click on “Variables” and then “New”. Select “Page Variables” > “Custom Javascript”

Then use this code for the variable “Content Group Type”, which will have a value of “tag

category
stories”, or “instagrams”

// Identify if content group is a tag, category, story etc.
function(){
var contentgroupclass = document.body.classList;
 var dem = contentgroupclass.length;
 for (var i = 0; i <= dem; i++) {
    if (contentgroupclass[i].match('contentgrouptype-')) {
      var contentgroup = contentgroupclass[i].replace(/contentgrouptype-/g, '');
      return contentgroup;
      break;
    }
 }
}

The code for “Content Group Name” will follow the same structure:

function(){
var contentgroupnameclass = document.body.classList;
 var dem = contentgroupnameclass.length;
 for (var i = 0; i <= dem; i++) {
    if (contentgroupnameclass[i].match('contentgroupname-')) {
      var contentgroupname = contentgroupnameclass[i].replace(/contentgroupname-/g, '');
      return contentgroupname;
      break;
    }
 }
}

And finaly, the code for “Content Group Article Title”

function(){
var articletitleclass = document.body.classList;
 var dem = articletitleclass.length;
 for (var i = 0; i <= dem; i++) {
    if (articletitleclass[i].match('article-title-')) {
      var articletitle = articletitleclass[i].replace(/article-title-/g, '');
      return articletitle;
      break;
    }
 }
}

You are done.

Go to the preview of tag manager and check that it’s working in the different pages. Soon, you will begin to see this data arrive in Google Analytics

Custom Report

For my use case, I prepared a small custom report using the drilldown of the different content groups:

And this in turn gives you this report:

And that’s it, you can now use the new Content Groups to see where your readers see the most value.

Hope this helps!

Under the Hood

Wed, 07 Feb 2018 10:44:23 +0000

What is a static site

Most sites work like errand boys. You go to them, ask for something and they will get it for you. The obvious problem here is that it may take longer to find and deliver your request.

The example is crude, but works. Most sites do have a database and each time you open a page, there is some behind the scenes work to gather and display what you have requested. It’s one of the reasons why some sites are slow. They may have a database that became too big, or too many visitors to be able to handle every request, some will have to wait.

This site works more like a store. When you get there, whatever you need is already in stock for you to read.

There is no database, every page here is like a PDF file: ready to deliver and open in any browser or device.

Speed comes at a cost

Most static sites are very bare bones, with little or no search, comments, automation of social media posts, etc.

The comments here are powered by an external website, disqus.com, and the search function only works by using a service from Algolia.com

It also means I need to make sure every image is small enough to keep the pages light. (More on that soon.)

The way I automate social media publishing is with Zapier and it’s integration with Buffer.com.

Flexibility in every page

Dynamic sites that use information from a database don’t allow for much in terms of flexibility.

For example, the story I called CryptoNovel uses javascript to encrypt and decrypt each chapter and adding this feature didn’t impact the whole site like it would have if it need some logic behind it.

It would go something like this:

If the story is called Crypto, then add this piece of javascript code and this form logic so that the user can decrypt the text.

It’s simple, but imagine asking this question once for each one of the 2000+ pages that make up this blog. At some point it would slow down the whole site.

Powered by a Community

The first incarnation of the “Digital Insanity” blog used the Future Imperfect theme, which I ported to use the Hugo static site generator.

It received a great deal of improvements since then. I have added the concept of Stories, made a good deal of SEO improvements, added structured data to the html, included my instagram feed and changed the way you see and use the search function.

None of this would have been possible if Hugo was not open source and supported by an amazing community who helps developers solve problems in the forum.

To everyone who participates there, please consider this my heartfelt way of thanking you for your time and dedication.

Tailored content and smart image crop

The flexibility of static files instead of a database is already good enough. I can go as far as have a fully different page if I like.

Recently, Hugo got a new feature thanks to the work by Bjørn Erik Pedersen: smart image cropping. This feature uses another open source project by Christian Muehlhaeuser, smartcrop.

I was already running optimizations during the generation of the static sites, but without any way of making sure I was keeping the subject of the photo intact:

# Optimize images

printf "\n${GREEN}### Build done, optimizing NEW images ###\n${NORMAL}"

if [ "$1" == "--all-images" ]
  then
      TAG=$(/usr/bin/git -C $DEFAULT_DIR rev-list --max-parents=0 HEAD)
  else
  # Find the images using the information in Git:
      TAG=$(/usr/bin/git -C $DEFAULT_DIR describe --tags --abbrev=0)
fi

# List the images to be optimized, using Git:
IMAGES=$(/usr/bin/git -C $DEFAULT_DIR diff --name-only $TAG | grep -E 'jpg|jpeg|JPG')

for i in $IMAGES; do
    NAME=$(basename $i)
  DIR=$(dirname $i)
  TARGET="${i/content/$WEBSITE_DIR}"
  printf "processing: $DIR | $TARGET"
  /usr/bin/jpegoptim --all-progressive -p -o $TARGET
  cd "$(dirname $TARGET)" && /usr/bin/convert $TARGET -verbose -resize 500x500 -set filename:original '%t' %[filename:original]_500.jpg ;

   /usr/bin/jpegtran -outfile $TARGET.out -copy none -optimize -perfect $TARGET && mv $TARGET.out $TARGET && chmod 644 $TARGET
done

Security and Simplicity

Since every file is already present in the server, there is no processing and less chances of the server being hacked.

Another thing is it makes me question: “Do I really need this feature?” Most of the times no, I do not need a fancy contact form or flashy pop up with information to subscribe.

Contact Form

The contact form page is powered by a service called FormSpree. It’s first tier is free and will deliver any input to the email address specified.

Again, less configuration and a better experience for the user.

TinyLetter

I wanted a no-thrills way of sending a newsletter and this was perfect. It allows for simple formatting of titles, lists and links.

To save time, I have Zapier keeping watch over the blog and preparing the content for me. All that is left for me to do is tweak the formatting and press send. What’s best? I can place the subscribe form anywhere.

Why keep a blog or a Site?

I still believe in good long form content. Specially if you mix it up between images, videos, gifs and whatever else is relevant. Lately I have even been thinking about migrating old content onto new sections of this site.

Here, anything can be posted. I did tiny experiments with video, little clips with less than a minute that YouTube insisted to take down because I didn’t own the rights for the music I used.

This site can also be made more personal to the point of sending a page to a single person or group. In fact, it’s riddled with hidden sections, draft projects, personal notes, and self reminders.

This would be harder to do with a Dynamic Site since the complexity behind the scenes is always too much for me to grasp and be 100% sure it’s working.

It’s about the story, it’s about being different

Sharing things with friends and family on Facebook and instagram can be cool, but it’s ephemeral. Facebook and Instagram’s Algorithms only give you a certain measure of exposure and once that’s gone, it’s forever.

Here, I hope to drive people to stumble upon more posts and stories to read. The Photo Story Telling Challenge is one of those things.

And does Facebook represent the whole range of interests you have? Here I can post my notes of UXLx and keep it in context. I can go as far as translate one of my favourite poems and display it in both languages I speak: English and Portuguese.

We need better ways to share what is important to us, and to have as much control as possible over the content we produce. And also, to mix and match the things we find along the way into a (half) coherent exposé of why we take the time to develop and write a blog.

A blog or a website is not the medium by excellence, sure I don’t think anything really is. What I do think is that so far this is the best solution to talk about the different thinks that interest me. It’s also future proof, since this site does not need to host a database or support a programming language like PHP. It will certainly be available to be read in 10 years time. I also get the luxury of a nice daily backup on my computer and on GitHub.

It comes with the added bonus of freedom. I don’t have to annoy you with ads, I don’t have to post everyday or worry about likes. I can relax and just make this virtual place into whatever I like, while at same time can write this article sitting in the sofa, on my computer or phone.

Isn’t this what technology should do? Take away restrains and give us freedom to connect with others in different and creative ways.

Blake Connally