Digital Insanity Magazine by Bruno Amaral

Import Instagram Posts Into Hugo

Thu, 25 Jun 2020 13:35:26 +0100

I do love Instagram and the cool ways we can connect through with, see other cities, people we miss. It’s just hard to deal with having my photos kind of trapped there.

It is possible to export them, but that process takes a while. So this last weekend I took some time to improve a script that I used to extract my photos from there and onto this site.

They are in the pages of the Photos section.

Here is a step by step on what I did for those more tech-inclined.

Some things to take into account before moving on:

We are assuming the website is ran by Hugo.
You need some knowledge in using the Command Line of your computer.
Some of the information below includes what is useful for me, your needs will vary.

Instaloader

Instaloader is a tool to download pictures (or videos) along with their captions and other metadata from Instagram.

This command line tool does the heavy lifting, it authenticates you to Instagram, queries your profile and posts and saves everything to a nice file structure.

What we are going to do is take those files, extract data from them, and save it as environment variables. Those will then be picked up by the Hugo Archetype we defined for new posts in that section.

Archetype for Instagram Posts

When you create new content, Hugo uses a template for that file. That template is called an Archetype.

There are some Hugo functions you can use in these templates, and we are using getenv that allows us to use environment variables in your new post.

Take a look:

---
categories:
date: {{ getenv "post_datetime"}}
description:
draft: false
resources:
- src: {{ getenv "post_image" }}
  name: "header"
layout: instagram
location:
  - link: {{ getenv "google_maps_link"}}
  - latitude: {{ getenv "latitude" }}
  - longitude: {{ getenv "longitude" }}
stories:
subtitle:
title: "{{ getenv "post_title" }}"
tags: {{ getenv "post_tags"}}
---

{{ getenv "post_content" }}

You can tweak this file to add more information, as long as you also do the same in the script that puts it all together. More on that later.

Import and update script

The script became a little more complex than what was to be expected. To keep things in order, it’s hosted on GitHub.

Steps

Edit the config section of the script
Run the setup with ./instagram-posts-from-instaloader.sh setup
Execute the update ./instagram-posts-from-instaloader.sh run-update

Running setup, you will be asked for your username and password. Instaloader will setup a directory with your username where it will store your photos alongside the metadata it found.

We are keeping this directory untouched, read from it, and save the information we need as an environment variable. The full code is below and on GitHub.

Inside the update-post function we are reading the json data with xzcat and jq. If you want to add more data to the exported post, this is where you should look.

After this is done, we check if the post exists and if not, run hugo new ... and copy any jpg or video files that may be included.

With this done, your new posts should be ready. Run hugo server to make sure it’s working as you expected.

If you wish to add or improve any bit of the script, feel free to submit a merge request on the GitHub page.

Let me know in the comments if this was useful. 😀

#!/bin/zsh
## USAGE

### 1. Run the setup with `./instagram-posts-from-instaloader.sh setup`
### 2. Execute the update `./instagram-posts-from-instaloader.sh run-update`

## Config

### Where is your blog located?
blog="/Users/brunoamaral/Labs/Digital-Insanity"
### What is your Instagram Username?
instagramUser="brunoamaral"
### Where are we saving the output from Instaloader? (do not include the username)
directoryWithInstagramPosts="/Users/brunoamaral/Labs/instagram/"

### Where do you want photos saved to? Relative to ./content 
instagramdir="instagram"

# Dependency check

command -v instaloader >/dev/null 2>&1 || { echo >&2 "I require Instaloader but it's not installed. Aborting."; exit 1 }
command -v xzcat >/dev/null 2>&1 || { echo >&2 "I require xzcat but it's not installed. Aborting."; exit 1 }
command -v jq >/dev/null 2>&1 || { echo >&2 "I require jq but it's not installed. Aborting."; exit 1 }
command -v hugo >/dev/null 2>&1 || { echo >&2 "I require hugo but it's not installed. Aborting."; exit 1 }

setup() {
  cd $directoryWithInstagramPosts; 
  instaloader --fast-update --geotags --login=$instagramUser $instagramUser
}

run-update(){
  find "$directoryWithInstagramPosts/$instagramUser" -maxdepth 1 -name '*.json.xz'  -exec ./instagram-posts-from-instaloader.sh update-post {} \;
}  >&2

update-post() {
  file=$(basename $1)
  dir=$(dirname $1)
  imagefinal=${file//'json.xz'/'jpg'}
  content=`xzcat "$1" | jq '.node.edge_media_to_caption.edges[0].node.text'`

  declare -a tags;
  tags=$(echo $content | grep -o '#[[:alpha:]|[:alnum:]]*')
  jsontags=$(printf '%s' "${tags[@]}" | jq -R . | jq -s .)
  jsontags=${jsontags//[$'\t\r\n'|$' ']}
  jsontags=${jsontags//'#'/}
  date=`cut -d'_' -f 1 <<< "$file"`
  time=`cut -d'_' -f 2 <<< "$file"`

  export post_image=$imagefinal;
  export post_datetime=$(echo $date"T"${time//-/:}"+00:00")
  export post_tags=$jsontags
  if [[ $content = null ]]; then
    export post_slug
    export post_title=$date
    export post_content=" "
  else
    slug=$(echo "$content" |  iconv -c -t ascii//TRANSLIT | sed -E 's/[~\^]+//g' | sed -E 's/[^a-zA-Z0-9]+/-/g' | sed -E 's/^-+\|-+$//g' | sed -E 's/-$//g' | tr A-Z a-z )
    export post_slug=${slug:0:25}
    title=${content:0:50}
    export post_title=${title//'"'}
    export post_content=$content
  fi

  locationfile=${file//.json.xz/_location.txt}

  grepLocation=$directoryWithInstagramPosts
  grepLocation+="$instagramUser/"$locationfile

  location=$(grep -s 'maps.google.com' $grepLocation)
  if [[ ! -z "$location"  ]]; then
  echo "found $location"
    export google_maps_link=$location 
    locationClean=${location/'&ll'/}
    export latitude=$(cut -d'=' -f 2 <<< "$locationClean" | cut -d'\' -f 1 | cut -d',' -f 1)
    export longitude=$(cut -d'=' -f 2 <<< "$locationClean" | cut -d'\' -f 1 | cut -d',' -f 2)
  fi

  destination="$blog/content/$instagramdir/$date-$time$post_slug/"
  if [[ ! -f "$destination/index.md"  ]]; then
    hugo new $instagramdir/$date-$time$post_slug/index.md
    cp ${1:r:r}(*.jpg|*.mp4) $destination
  else
    echo 'post exists: '$post_title
  fi

}  >&2

if [[ $1 = "setup" ]]; then
  setup
fi

if [[ $1 = "run-update" ]]; then
  run-update
fi

if [[ $1 = "update-post" ]]; then
  update-post $2
fi

Lisa Fotios

Under the Hood

Wed, 07 Feb 2018 10:44:23 +0000

What is a static site

Most sites work like errand boys. You go to them, ask for something and they will get it for you. The obvious problem here is that it may take longer to find and deliver your request.

The example is crude, but works. Most sites do have a database and each time you open a page, there is some behind the scenes work to gather and display what you have requested. It’s one of the reasons why some sites are slow. They may have a database that became too big, or too many visitors to be able to handle every request, some will have to wait.

This site works more like a store. When you get there, whatever you need is already in stock for you to read.

There is no database, every page here is like a PDF file: ready to deliver and open in any browser or device.

Speed comes at a cost

Most static sites are very bare bones, with little or no search, comments, automation of social media posts, etc.

The comments here are powered by an external website, disqus.com, and the search function only works by using a service from Algolia.com

It also means I need to make sure every image is small enough to keep the pages light. (More on that soon.)

The way I automate social media publishing is with Zapier and it’s integration with Buffer.com.

Flexibility in every page

Dynamic sites that use information from a database don’t allow for much in terms of flexibility.

For example, the story I called CryptoNovel uses javascript to encrypt and decrypt each chapter and adding this feature didn’t impact the whole site like it would have if it need some logic behind it.

It would go something like this:

If the story is called Crypto, then add this piece of javascript code and this form logic so that the user can decrypt the text.

It’s simple, but imagine asking this question once for each one of the 2000+ pages that make up this blog. At some point it would slow down the whole site.

Powered by a Community

The first incarnation of the “Digital Insanity” blog used the Future Imperfect theme, which I ported to use the Hugo static site generator.

It received a great deal of improvements since then. I have added the concept of Stories, made a good deal of SEO improvements, added structured data to the html, included my instagram feed and changed the way you see and use the search function.

None of this would have been possible if Hugo was not open source and supported by an amazing community who helps developers solve problems in the forum.

To everyone who participates there, please consider this my heartfelt way of thanking you for your time and dedication.

Tailored content and smart image crop

The flexibility of static files instead of a database is already good enough. I can go as far as have a fully different page if I like.

Recently, Hugo got a new feature thanks to the work by Bjørn Erik Pedersen: smart image cropping. This feature uses another open source project by Christian Muehlhaeuser, smartcrop.

I was already running optimizations during the generation of the static sites, but without any way of making sure I was keeping the subject of the photo intact:

# Optimize images

printf "\n${GREEN}### Build done, optimizing NEW images ###\n${NORMAL}"

if [ "$1" == "--all-images" ]
  then
      TAG=$(/usr/bin/git -C $DEFAULT_DIR rev-list --max-parents=0 HEAD)
  else
  # Find the images using the information in Git:
      TAG=$(/usr/bin/git -C $DEFAULT_DIR describe --tags --abbrev=0)
fi

# List the images to be optimized, using Git:
IMAGES=$(/usr/bin/git -C $DEFAULT_DIR diff --name-only $TAG | grep -E 'jpg|jpeg|JPG')

for i in $IMAGES; do
    NAME=$(basename $i)
  DIR=$(dirname $i)
  TARGET="${i/content/$WEBSITE_DIR}"
  printf "processing: $DIR | $TARGET"
  /usr/bin/jpegoptim --all-progressive -p -o $TARGET
  cd "$(dirname $TARGET)" && /usr/bin/convert $TARGET -verbose -resize 500x500 -set filename:original '%t' %[filename:original]_500.jpg ;

   /usr/bin/jpegtran -outfile $TARGET.out -copy none -optimize -perfect $TARGET && mv $TARGET.out $TARGET && chmod 644 $TARGET
done

Security and Simplicity

Since every file is already present in the server, there is no processing and less chances of the server being hacked.

Another thing is it makes me question: “Do I really need this feature?” Most of the times no, I do not need a fancy contact form or flashy pop up with information to subscribe.

Contact Form

The contact form page is powered by a service called FormSpree. It’s first tier is free and will deliver any input to the email address specified.

Again, less configuration and a better experience for the user.

TinyLetter

I wanted a no-thrills way of sending a newsletter and this was perfect. It allows for simple formatting of titles, lists and links.

To save time, I have Zapier keeping watch over the blog and preparing the content for me. All that is left for me to do is tweak the formatting and press send. What’s best? I can place the subscribe form anywhere.

Why keep a blog or a Site?

I still believe in good long form content. Specially if you mix it up between images, videos, gifs and whatever else is relevant. Lately I have even been thinking about migrating old content onto new sections of this site.

Here, anything can be posted. I did tiny experiments with video, little clips with less than a minute that YouTube insisted to take down because I didn’t own the rights for the music I used.

This site can also be made more personal to the point of sending a page to a single person or group. In fact, it’s riddled with hidden sections, draft projects, personal notes, and self reminders.

This would be harder to do with a Dynamic Site since the complexity behind the scenes is always too much for me to grasp and be 100% sure it’s working.

It’s about the story, it’s about being different

Sharing things with friends and family on Facebook and instagram can be cool, but it’s ephemeral. Facebook and Instagram’s Algorithms only give you a certain measure of exposure and once that’s gone, it’s forever.

Here, I hope to drive people to stumble upon more posts and stories to read. The Photo Story Telling Challenge is one of those things.

And does Facebook represent the whole range of interests you have? Here I can post my notes of UXLx and keep it in context. I can go as far as translate one of my favourite poems and display it in both languages I speak: English and Portuguese.

We need better ways to share what is important to us, and to have as much control as possible over the content we produce. And also, to mix and match the things we find along the way into a (half) coherent exposé of why we take the time to develop and write a blog.

A blog or a website is not the medium by excellence, sure I don’t think anything really is. What I do think is that so far this is the best solution to talk about the different thinks that interest me. It’s also future proof, since this site does not need to host a database or support a programming language like PHP. It will certainly be available to be read in 10 years time. I also get the luxury of a nice daily backup on my computer and on GitHub.

It comes with the added bonus of freedom. I don’t have to annoy you with ads, I don’t have to post everyday or worry about likes. I can relax and just make this virtual place into whatever I like, while at same time can write this article sitting in the sofa, on my computer or phone.

Isn’t this what technology should do? Take away restrains and give us freedom to connect with others in different and creative ways.

Blake Connally

#cid20 #geek

Sat, 27 Sep 2014 07:52:35 +0000

“#cid20 #geek”

X-Ray of a broken political system

Sun, 29 Dec 2013 22:55:19 +0000

The picture in this post is an infographic built by Pedro Cruz. The circles represent portuguese companies. The bigger the circle, the more politicians where involved in managing it. You can find it here: https://pmcruz.com/eco/

He called it “An ecosystem of Corporate Politicians“, I call it an X-Ray of a broken political system.

The dataset includes data from 1975 to 2013. Which shows that this is not just a problem to be fixed, it’s a whole tradition of mixed interests and loyalties. I, for one, have long ago lost all faith in the political parties available. Whenever there is an election, I get myself out of the house to stand in line, show my ID card and push a blank vote down the ballot box. It’s my way of saying that I care, but that none of the options available make any sense to me.

Now, seeing the way major companies are organised around the political system, I can’t help but feel a bit more of dismay and, this is new, a lot more anger towards the political class.

Three players and no communication

Just keep this in mind, our PM is a social-democrat faced with a country taking financial hits from all sides, he is therefore forced to resolve all these issues of alliances between companies and opposing parties to get anything done. While he does that, we get news from him saying that things are looking up with no clear signal that they in fact are. Taxes are still high, starting a profitable company sounds like pulling off a miracle and some families are at the brink of poverty.

On the other side of things, the Socialist Party is a joke. Their communication strategy is to wait for any action by the social democrats and to counter it with an ideological speech. There is no clear plan, no actionable ideas, just shouts and soundbites against the prime minister.

The communist parties do pretty much the same, trying to rally up the middle and lower class towards demands of better quality of life and most of the time undermining any scarce effort for a joining of hands towards a solution.

As for the President, well, there is nothing I can say about him because I haven’t seen any action on his part. Ever. So he’s the non-playing character of this system.

In this scenario I don’t trust any of the remaining members of our government to be virtuous and make the right decision if they have to choose between the people or their own companies (and well-being).

Hacking it to fix it

Have you ever worked with an old computer? Or even an old software at your job that everybody hates and that stopped being efficient a million years ago?

I am going to take a leap here and say that if you answered any of those questions affirmatively that computer or software was soon replaced with a newer model. Sadly we can’t just throw our politicians out of a window like we did in the past. Also, I don’t expect to see this government passing laws that enforce political independence from private companies.

The solution I see is to raise the bar on political scrutiny. In a balanced social system, this scrutiny would come from journalist and media companies. In this status quo journalists have little resources and whether they want to or not, their publications are in some way tied to some of the major portuguese companies.

In Portugal, Political Scrutiny, good or bad, needs to come from hackers and geeks. People eager to learn and eager to build things that help others. And there are already some good examples out there about what can be done.

One of them is a site where people can view what political promises were made by their mayor.

To help fix this system, we could take that idea one step further and list relevant political actions and laws passed, with names of those involved and the private companies they work with. The goal would be to bring just a bit more transparency to politics. Of course it wouldn’t work forever, but it would at least serve as a deterrent.

What’s the use ?

Probably nothing. This idea will by no means result in a complete fix for the broken system. The only good fix would be a full reboot but we simply can’t afford it. We have however entered a time when we must take matters into our own hands.

Patch it wherever possible,
Build new tools to empower citizens,
Make it more transparent,
Throw away ideology in favour of practical and effective projects.

We have sat on our hands and the political class had their opportunity to show results. It’s about time we take action.

A New Perspective

Mon, 23 Sep 2013 12:20:22 +0000

This blog was a bit stuck in the dark ages of blogging and was in need of a redesign, in all definitions of the word: a new look and a new reading experience, as well as a revised editorial line.

Fortunately I know some amazing people, one of them being Diana Costa.

With her help, there is now a new logo and a new design. It is more focused on the content and the reading experience and adapts gently to any screen size.

I liked her solution so much that instead of applying it to an existing wordpress template I went the extra mile and coded it all from scratch. It is using the latest Twitter Bootstrap Framework with a 16 column grid instead of the usual 12 columns. This saved several days of coding to get the first prototype.

And because the search function is paramount on any website, it is there at the header for when you need it taking up the whole screen so you can focus and find what you are looking. For an added bonus, check out what happens when you search and don’t get results.

But there is more.

To make the blog a more compelling experience I reviewed the whole editorial line I was following. So now you have the following categories, or perspectives:

Communication; Communication and Web

Not all communication is about the web, but the web is always about communication. If you want to check out the ideas and opinion, papers and articles on Communication and Relationship Management, then you need can follow the communication part which will include the web perspective as a sub-category.

Buf If all you care about is what’s new online and how it can help you communicate or how it changes the way we relate with each other, then you should read the Web perspective in its own single category.

This is a split-hair. I was inclined to just keep the Web category and drop it in favor of Communication or vice versa. But the web has always been social, always been about communicating and collaborating with people across the globe. Therefore they need to be present and intertwined.

Business

With so many things that change online and offline, what does that mean for business? This is where I want to share what perils and what opportunities show up whenever a major player like facebook, twitter, or google makes changes or when we as users switch from one platform to the other. This is also where I am going to expand on the idea of strategy and around the need for a concept of communication strategy, either digital or analog.

Geek

I don’t hide that I am a Geek and in fact I am quite proud of that. So it was only fitting that for a personal perspective I would embrace this trait. This is where I will share what is happening with me, the bits and pieces of code I come up with, the pet projects and aspirations. This is the window where you will have a chance to peak into my world of madness and coffee.

Exporting Things.app tasks to Todo.txt format

Wed, 03 Jul 2013 10:15:32 +0000

Warning, there is some serious geek talk coming up.

Long story short I have been using Things as my task management for a long time. What is bad about these tools is that they rarely allow you to export or save your tasks into a universal format that can be read by any app.

Fortunately, Gina Trapani came up with Todo.txt calling it a Future-proof task tracking in a file you control.

Since it works with a simple text file you can use it together with Dropbox and sync it across devices.

I think you can see where I am getting at…

If Todo.txt is so simple and useful it makes perfect sense to use it as the universal format to save/export/import tasks. To make this perfect, there are a few task management tools that support the todo.txt syntax.

So I dove into the Things.app applescript documentation and after a while came up with a little script that you can use to export your tasks and break free so you can try out other task management solutions or even come up with your own.

Everything is available for free in my github account, I hope you find it useful.