Insights


<p>We recently built a tool to visit every page in a website and collect the resulting data into a JSON file, a process known as web <strong>crawling</strong> and <strong>scraping</strong>. The JSON file was then used to create a search feature on the site, no amendments were needed to the code that powered the site itself. Here&#8217;s how we did it, and why we&#8217;ve open sourced the code to do it.</p>



<h2 id="h-the-decision-making-process">The decision making process</h2>



<p>There are a lot of solutions for crawling/spidering/scraping websites. A lot. And so obviously we decided to write our own so that it may become the <a href="https://xkcd.com/927/">universal standard</a>! Actually, no. We built what we needed to get the job done well using an open source <a href="http://go-colly.org/">scraping framework called Colly</a>. But why couldn&#8217;t we just use something that already existed, one of the many tools designed for this purpose?</p>



<p>Firstly, there is no one-size-fits-all solution for crawling. Every website is different. No matter which tool is used, you will have to specify what information you need, from which pages, and how you&#8217;d like it presented. Secondly, while we&#8217;re not adverse to using SAAS solutions, in this case we wanted to be able to control our costs and run the crawler on our schedule without worrying about spiraling costs. These two issues mean that we need an easily configurable tool that we can run without jumping through too many hoops.</p>



<p>We&#8217;d already had experience with scraping tools that could be configured with a GUI and so strongly favoured something which would let us capture our configuration as code &#8211; therefore allowing us to store it under source control.</p>



<h2 id="h-colly-web-crawling-and-scraping-in-go">Colly &#8211; Web Crawling and Scraping in Go</h2>



<p>After much research, Colly ticked our boxes: we could run it ourselves, we could configure it with code, and it was simple to set up and understand. Let&#8217;s take a look at an example, which is written in Go as you might expect:</p>



<pre class="wp-block-code line-numbers"><code>c := colly.NewCollector(
    colly.AllowedDomains("gotripod.com"),
)

c.OnRequest(func(r *colly.Request) {
    fmt.Println("Visiting", r.URL.String())
})

c.Visit("https://gotripod.com/")</code></pre>



<p>This has three distinct parts:</p>



<ol><li>We create a &#8220;collector&#8221;. It is only allowed to visit gotripod.com</li><li>On each request, it should print the current URL</li><li>Visit the Go Tripod homepage, starting the site crawling process</li></ol>



<p>All this will do is hit the GT homepage and stop, because we haven&#8217;t told it what to do afterwards. And since we&#8217;re writing a <em>crawler</em> and not a <em>visitor</em>, we need to tell it where to crawl to next. After creating the collector, we can add the following code:</p>



<pre class="wp-block-code"><code>c.OnHTML("html", func(e *colly.HTMLElement) {
    e.ForEach("a&#91;href]", func(_ int, el *colly.HTMLElement) {
        e.Request.Visit(el.Attr("href"))
    })
}</code></pre>



<p>When Colly&#8217;s collector detects that the requested URL is an HTML page, it&#8217;ll trigger this callback and loop through every <code>&lt;a&gt; </code>element with an <code>href</code> attribute. It&#8217;ll then visit the URL in the <code>href</code>. That&#8217;s really all we need to <em>crawl</em> through a website with Colly! This example code doesn&#8217;t do much, so let&#8217;s have a look at how we can <em>scrape</em> information from the page.</p>



<p>The <code>OnHTML</code> method takes two parameters, a selector and a callback. That selector is anything supported by <a href="https://github.com/PuerkitoBio/goquery">goquery</a> and if you&#8217;re comfortable with CSS selectors then you&#8217;ll be good. This selector specifies the element(s) that will be passed as the second argument to OnHTML, which has a type of <a href="https://pkg.go.dev/github.com/gocolly/colly#HTMLElement">colly.HTMLElement</a>. With the methods on that type you can further focus down your selection:</p>



<pre class="wp-block-code"><code>c.OnHTML("article", func(e *colly.HTMLElement) {
    exportArticle(
        e.ChildText("news__meta"), e.ChildText("news__title")
    )
}</code></pre>



<p>Every time Colly sees an article element on the current page it&#8217;ll pass it to the callback, where we use the ChildText method to select metadata and title for export. Colly has <a href="https://github.com/gocolly/colly/tree/master/_examples">loads of examples</a> which go into further depth, so we&#8217;ll leave our quick overview here.</p>



<h2 id="h-all-aboard">All aboard!</h2>



<p>Having been suitably impressed by Colly&#8217;s simplicity, we completed our project without fuss. It&#8217;s not the first time we&#8217;ve written a crawler and so we wanted to see if we could easily use Colly in future projects without rewriting the same code. We wondered if we could extend Colly&#8217;s use of selectors so that we could just reconfigure the crawler to run on a completely different project. It turned out we could!</p>



<p>Super Simple Scraper takes a JSON configuration file and pushes it into Colly&#8217;s crawl engine. Out pops a JSONL file. Let&#8217;s take a look at a configuration snippet. Web crawling and scraping ahoy!</p>



<pre class="wp-block-code"><code>"html": {
    "selectors": {
        "title": "title",
        "h1": "h1",
        "url": "{{ .Request.URL }}"
    }
}</code></pre>



<p>This is the configuration used when scraping HTML pages. We specify that there should be a selector called &#8220;title&#8221; which grabs the contents of the title tag. Likewise a selector called &#8220;h1&#8221; that grabs the combined contents of all the h1 tags on the page. The last selector is a templated selector which grabs the URL of the page. </p>



<p>We&#8217;ve provided a Docker image to run SSScraper, which means that you can run it with a single command:</p>



<pre class="wp-block-code"><code>docker run -it -v pwd:/go/src/app -e "CONFIG=$(cat ./path/to/your/config.json | jq -r tostring)" gotripod/ssscraper:main</code></pre>



<h2 id="h-ss-scraper-provides-results">SS Scraper provides results</h2>



<p>And that&#8217;ll produce a JSONL file, where the selector names are used as the JSON keys:</p>



<pre class="wp-block-code"><code>{"h1":"Want the internet to work for you? Got an idea for a project?","title":"Contact Go Tripod | website \u0026 web application development in Falmouth, Cornwall","url":"https://gotripod.com/contact/"}
{"h1":"Topic: Git Got an idea for a project?","title":"Git Archives - Go Tripod","url":"https://gotripod.com/topics/git/"}</code></pre>



<p>That&#8217;s some Super Simple Scraping! The &#8220;title&#8221; and &#8220;h1&#8221; selectors are straightforward, but there&#8217;s some interesting stuff happening in the &#8220;url&#8221; selector. Let&#8217;s break that down to understand it a bit more.</p>



<p>We&#8217;re using Go&#8217;s text/templates package, and passing in the current request as a value. The selector {{ .Request.URL }} therefore becomes: &#8220;give me the value of the URL field on the Request variable&#8221;. </p>



<p>Templates also have access to <a href="https://masterminds.github.io/sprig/">sprig</a> helpers, which means you can get fancy with something like:</p>



<pre class="wp-block-code"><code>{{ .Request.URL.Path | regexFind \"\/(insights|work)\/\" | replace \"/\" \"\" }}</code></pre>



<p>That would return either &#8220;insights&#8221;, &#8220;work&#8221;, or an empty string, depending on which appeared in the URL of the current page.</p>



<figure class="wp-block-pullquote"><blockquote><p>Why JSONL? Because we can then import it straight into <a href="https://typesense.org/">Typesense</a> to search it!</p></blockquote></figure>



<h2>Custom software for custom requirements</h2>



<p>SS Scraper doesn&#8217;t do anything that someone familiar with Go couldn&#8217;t do, but it does mean that we can quickly deploy a crawler/scraper on a completely new project without having to write code. We can have different config files for different projects rather than having to start afresh each time. <a href="https://github.com/gotripod/ssscraper/">SS Scraper is open source</a> on GitHub, and if you&#8217;d like us to help you deploy a crawler or scraper for your site, please <a href="https://gotripod.com/contact/">get in touch</a>.</p>


Web Crawling and Scraping Made Super Simple

<p>Automotive industry manufacturer <a href="https://gotripod.com/work/sanden/">Sanden</a> recently asked us to import a new set of data into <a href="https://sandenaftermarket.com/">their site</a>, which we built on WordPress several years ago. While we knew this was going to be a one-off custom WordPress data import, there were a few things to consider when deciding how to implement it.</p>
<ul>
<li>The import would take a while to run, and we needed to see some kind of progress indicator</li>
<li>Because the import time will exceed the allowed request timeout, we&#8217;ll need to process the import in the background</li>
<li>If anything went wrong during the import we&#8217;d need a log</li>
<li>It&#8217;d be good to do a &#8220;dry run&#8221; to check for problems without writing to the database</li>
</ul>
<p>This was one of those situations where WordPress is a little bit more of a hindrance than another system might be. Because the structure of the WordPress database is one-size-fits-all, it can be a bit tricky to do the kind of SQL INSERT you might do with other systems, and so we needed to jump through a few hoops. With the requirements asking for a dry run toggle, a progress indicator and a log display, it made sense to build a custom WordPress data import plugin which would show up in the WP admin and let us build a user interface. It could also use the WordPress authentication mechanism to restrict access to authorised users.</p>
<p>Rather than reinvent the wheel, we used <a href="https://actionscheduler.org/">Action Scheduler</a> as the backbone for our import processing. It also provided hooks to allow us to generate log files. It comes with its own dashboard to allow you to diagnose any issues, which we used during development of our plugin. Once that had been selected, the core of the import process was very simple &#8211; checking that records didn&#8217;t already exist and using the <a href="https://www.advancedcustomfields.com/">ACF</a> API to create them where missing.</p>
<p>As well as these benefits, building a custom WordPress plugin gave us a means to logically separate our code. We could package it up in a zip file for easy installation and updates, making testing a breeze.</p>


Custom WordPress data import – solving business problems with plugins


<p>&#8220;Software dependency management&#8221; just sounds so glam, doesn&#8217;t it? If we rephrase it to say &#8220;do less, get more&#8221; it starts to sparkle a little. Let&#8217;s take a look at this essential topic and some rules that Go Tripod follow in our <a href="https://gotripod.com/work/" target="_blank" rel="noreferrer noopener">projects</a>.</p>



<p>A &#8220;dependency&#8221; refers to a unit of code that the rest of your code relies on. The most common way of incorporating it is to use the package manager associated with your language of choice. And so; in Ruby you might add a <a href="https://rubygems.org/" target="_blank" rel="noreferrer noopener">gem</a> to your project, Rust uses <a href="https://crates.io/" target="_blank" rel="noreferrer noopener">crates,</a> JavaScript has <a href="https://npmjs.org" target="_blank" rel="noreferrer noopener">npm</a>. Easy peasy.</p>



<p>Except <a href="https://www.youtube.com/watch?v=7mAFiPVs3tM" target="_blank" rel="noreferrer noopener">it isn&#8217;t</a>. It&#8217;s telling that searching for the title of this post on Google turns up the Wikipedia page for &#8220;<a href="https://en.wikipedia.org/wiki/Dependency_hell" target="_blank" rel="noreferrer noopener">dependency hell</a>&#8220;. Adding dependencies is one thing but we&#8217;re talking software dependency <em>management</em> here, and that involves not only keeping them up to date, but making sure you actually need those dependencies you&#8217;re adding.</p>



<h2>Rule 1 of software dependency management: do you need it?</h2>



<p>Let&#8217;s talk about that last point first. Here we go, we&#8217;re adding a dependency:</p>



<pre class="wp-block-code"><code>npm install stringconcat</code></pre>



<p>In our theoretical JavaScript project, we&#8217;re running a command to add a dependency called &#8220;stringconcat&#8221; so that we can join strings together. But do we need that? In JavaScript, we certainly don&#8217;t:</p>



<pre class="wp-block-code"><code>const joinedString = 'my' + 'string';</code></pre>



<p>This is a contrived example, but back in 2016  <a href="https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/" target="_blank" rel="noreferrer noopener">a very simply library called left-pad was removed from NPM</a> and caused a great deal of consternation.</p>



<p>So before you run that command willy-nilly, think: do you need it? Or can you just write it yourself?</p>



<h2>Rule 2: is it any good?</h2>



<p>Before adding the dependency, you need to make sure if it&#8217;s of sufficient quality. This doesn&#8217;t just mean that it has the features you need, or that you need to do an in-depth source code analysis, but you do need to perform some due diligence:</p>



<p><strong>How popular is your dependency? Are there more popular alternatives?</strong></p>



<p>Popularity doesn&#8217;t guarantee quality, but it&#8217;s a good indicator that the community relies on it. That should mean that people are willing to maintain it.</p>



<p><strong>How many maintainers does it have?</strong></p>



<p>Is there only one person working on this code? What happens if they get really busy?</p>



<p><strong>Are there lots of open issues or pull requests?</strong></p>



<p>If the code&#8217;s on GitHub, take a look at the issues and pull requests. Does it seem like these are neglected, or is the project team responsive and present?</p>



<p>There&#8217;s no hard and fast rule for assessing the health of a project, but thinking along these lines should allow you to get a feel for whether code&#8217;s going to be well maintained.</p>



<h2>Rule 3: do you <em>still</em> need it?</h2>



<p>We might find that our code evolves. We might find that <a href="https://caniuse.com/" target="_blank" rel="noreferrer noopener"><em>platforms</em> evolve</a>. In <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/padStart" target="_blank" rel="noreferrer noopener">those cases</a> we need to keep on top of our dependencies to make sure that we&#8217;re not bringing anything in unnecessarily. Particularly in cases where native implementations become available, we should use that to get a speed boost for free.</p>



<h2>Rule 4 of software dependency management: keep up to date!</h2>



<p>Rules 1 &amp; 2 are mere preludes to the killer pain of dependency management: updates. Let&#8217;s look at an example using Ruby on Rails 6, which introduced Action Cable to integrate WebSockets into your application. You try and upgrade Rails as follows:</p>



<pre class="wp-block-code"><code>bundle update rails</code></pre>



<p>And see the dreaded:</p>



<pre class="wp-block-code"><code>Bundler could not find compatible versions for gem "actionpack"
&lt;INSERT ENDLESS MESSAGES&gt;</code></pre>



<p>This is the reality of dependency hell. A gem relies on actionpack 5.x, so when you try and upgrade actionpack to 6.x you can&#8217;t, because of that one gem. So, you update that gem, but you can&#8217;t, because either it hasn&#8217;t been updated for Rails 6 or it in turn depends on <em>another</em> gem which is tied to Rails 5. Repeat this all the way down the dependency tree or until you have removed all of your hair and nails, whatever comes first.</p>



<p>There is no foolproof way around this, but the easiest way is to follow the other rules: keep your dependencies limited to ones that you really need and ones which are well maintained. Secondly, don&#8217;t pin to some random broken git revision, and don&#8217;t use your own forks of code. You&#8217;re asking for trouble. Thirdly, make updating your dependencies routine. Do it regularly and you&#8217;ll avoid having to wade through dependency hell and migration instructions in changelogs. Not only will your hair and nails thank you, but your project will too.</p>




Software dependency management – a primer

<p>At Go Tripod, we&#8217;ve been in the business of software development for a long time. Although Go Tripod was officially formed in 2009, we&#8217;ve got blog posts going back to 2007 which describe the technologies we were interested in at the time. My <a href="https://colinramsay.co.uk">personal blog</a> goes back to 2003! Here&#8217;s a quick list of the things we&#8217;ve been into during that time:</p>
<ul>
<li>ASP.NET (with MVC using Castle Project or SubSonic)</li>
<li>PHP &amp; WordPress</li>
<li>Obj-C</li>
<li>Adobe Flex &amp; Flash</li>
<li>Ruby &amp; Ruby on Rails</li>
<li>JavaScript (React, React Native, Ext JS &amp; Sencha Mobile)</li>
</ul>
<p>That&#8217;s a whole host of disparate technologies. Can&#8217;t we just settle on one thing?</p>
<h2>Backend Dealings</h2>
<p>ASP.NET is still a thing today, just as it was when we wrote about it in 2007. But in 2007, the web was changing rapidly. ASP.NET was clunky, relying on something called ViewState, a giant blob of data that got transferred on every page request. While it enabled rapid application development, the result wasn&#8217;t always maintainable or conducive to quick loading. I was an <a href="https://colinramsay.co.uk/ruby%20on%20rails/2005/04/06/ruby-on-rails-a-dabble.html">early fan of Ruby on Rails</a> and its organised approach to development (known as MVC), and it seemed like a breath of fresh air compared to the complexity of ASP.NET. We still use Ruby on Rails for our backend systems today, 14 years after we first started researching it.</p>
<p>On the way to 2019 though, we continued evaluating new software. Microsoft got on the bandwagon with the mostly-excellent ASP.NET MVC framework, but the Rails community was cranking out loads of great libraries to support an excellent foundation. PHP did the same, with frameworks like CakePHP but, at the time it was released, PHP had a reputation for being a bit of a mess.</p>
<h2>Front &amp; Centre</h2>
<p>The same applies on the frontend, though to a lesser extent. JavaScript&#8217;s always been the <a href="https://en.wikipedia.org/wiki/Lingua_franca">lingua franca</a> of the web, powering rich interactions in the browser. But to get round some of its quirkiness, we had <a href="http://prototypejs.org/">PrototypeJS</a> in 2005, <a href="http://prototypejs.org/">MooTools</a> in 2006, and <a href="http://jquery.com/">jQuery</a> in 2007.</p>
<blockquote><p>jQuery inspired me to create <a href="https://colinramsay.co.uk/css/projects/c%23/fizzler/2008/10/19/fizzler-a-css-selector-engine-for-c-sharp.html">Fizzler, a selector engine for C#</a> in 2008. It was improved and redesigned by <a href="http://www.raboof.com/projects/fizzler/">Atif Aziz</a>, and he still maintains it today.</p></blockquote>
<p>A big one for me was the release of <a href="https://en.wikipedia.org/wiki/Ext_JS">Ext JS</a> in 2007/2008, and I went on to author and co-author <a href="https://www.packtpub.com/books/info/authors/colin-ramsay">several books</a> on this rich framework.</p>
<p>In 2013, <a href="https://reactjs.org/">ReactJS was released</a> and was as much of a game-changer for frontend development as Ruby on Rails had been for backend. We jumped on it, and its sibling for apps, <a href="https://facebook.github.io/react-native/">React Native</a>, to let us build complex user interfaces in a way that was maintainable and testable.</p>
<h2>Why tho?</h2>
<p>What makes us continually search for a <a href="https://en.wikipedia.org/wiki/Build_a_better_mousetrap,_and_the_world_will_beat_a_path_to_your_door">better mousetrap</a>? Looking at the bigger picture gives a clear reason: the difference in developer experience between jQuery in 2007 and React in 2019 is huge. A better developer experience means that we can provide an end product that is better tested and is more usable. The gap between ASP.NET and Ruby on Rails is a bit more muddled; we could use ASP.NET MVC now, which is a much better product. But Ruby on Rails wins out, because it&#8217;s simpler to use and has a huge lead in community-provided libraries to help us turn out top-notch projects.</p>
<h2>But&#8230;</h2>
<p>There&#8217;s an outlier here. Something that has remained constant, like a comforting blanket. WordPress was released in 2003 and remains incredibly popular, powering over half the world&#8217;s websites. It&#8217;s not perfect, but it&#8217;s developed in the open and has a massive support community. We&#8217;ve used it for years and hope to keep on doing so for years to come. The recent release of Gutenburg gives us the chance to let users build amazing page layouts with ease, and using partners such as WPEngine means we can keep things secure and backed up without any worries.</p>
<h2>Conclusion</h2>
<p>The world of software moves quickly, but the goals remain the same: make great experiences for our clients and their customers. The technology should fade away into the background, but we know it&#8217;s there, quietly powering and empowering.</p>


Evolving our technologies and learning how to learn

<p><a href="https://content.gotripod.com/images/2019/01/photo-1531030874896-fdef6826f2f7.jpg" data-size="1350x900"><img loading="lazy" class="alignnone size-large wp-image-471" src="https://content.gotripod.com/images/2019/01/photo-1531030874896-fdef6826f2f7-1024x683.jpg" alt="" width="1024" height="683" srcset="https://content.gotripod.com/wp-content/uploads/2019/01/photo-1531030874896-fdef6826f2f7-1024x683.jpg 1024w, https://content.gotripod.com/wp-content/uploads/2019/01/photo-1531030874896-fdef6826f2f7-300x200.jpg 300w, https://content.gotripod.com/wp-content/uploads/2019/01/photo-1531030874896-fdef6826f2f7-768x512.jpg 768w, https://content.gotripod.com/wp-content/uploads/2019/01/photo-1531030874896-fdef6826f2f7.jpg 1350w" sizes="(max-width: 1024px) 100vw, 1024px" /></a></p>
<p>Just interested in the code? <a href="#migrator-code">Cut to the chase</a>.</p>
<p>GitHub and GitLab are competing services providing source control products based on Git. Source control enables us to keep track of all of the code changes we make to a project so that we can roll them back in the event of a problem. Seeing the history of a project can also help understand why decisions were made, years after they happened.</p>
<p><a href="https://blog.github.com/2019-01-07-new-year-new-github/">GitHub have just announced</a> that they&#8217;re providing free private repositories for individuals. This is great news for the community at large, but since we run an &#8220;Organisation&#8221; account on GitHub, it didn&#8217;t directly affect us. However, it did give us a chance to re-evaluate how we host our source code. We previously self-hosted GitLabCE, which is a fine piece of software. However, it had two drawbacks:</p>
<p>1. We&#8217;re hosting another server, which requires updating and auditing.<br />
2. There are some third-party services which only interact with GitHub and not GitLab</p>
<p>We decided to close our GitLab server and move to GitHub, and the first step of that is moving our repositories. We had 132 hosted on that server, so moving them one-by-one simply wasn&#8217;t practical. Fortunately, both GitLab and GitHub have APIs which we could use to migrate automatically.</p>
<p>We&#8217;re going to use a Ruby script to call the APIs. The gitlab and octokit gems wrap the GitLab and GitHub APIs respectively, so once we&#8217;ve installed those we need to configure them:</p>
<pre><code class="language-ruby">Gitlab.configure do |config|
  config.endpoint       = GL_ENDPOINT
  config.private_token  = GL_PRIVATE_TOKEN
end

gh_client = Octokit::Client.new(:access_token =&gt; GH_PRIVATE_TOKEN)</code></pre>
<p>Having done so we can fetch all of our GitLab projects:</p>
<pre><code class="language-ruby">gl_projects = Gitlab.projects.auto_paginate</code></pre>
<p>Auto-pagination means we can grab all of them at once. Now, we need to do something with each project:</p>
<pre><code class="language-ruby">gl_projects.each do |gl_project|
# do something in here
end</code></pre>
<p>Firstly, we specify the destination repository in GitHub:</p>
<pre><code class="language-ruby">destination_repo = "#{GH_ORG_NAME}/#{gl_project.name}"</code></pre>
<p>GH_ORG_NAME is a variable containing the organisation which will contain this repository, in our case &#8220;gotripod&#8221;. You&#8217;ll see where to specify this in the full source code later.</p>
<p>Before we can grab the repo from GitLab, we need to make sure we&#8217;re a member of its project. That&#8217;s straightforward:</p>
<pre><code class="language-ruby">begin Gitlab.add_team_member(gl_project.id, 4, 40)
  puts "You've been successfully added as a maintainer of this project on GitLab."
rescue Gitlab::Error::Conflict =&gt; e
  puts "You are already a member of this project on GitLab."
end</code></pre>
<p>And we also need to create the destination repo on GitHub:</p>
<pre><code class="language-ruby">begin gh_client.create_repository(gl_project.name, organization: GH_ORG_NAME, private: true)
  puts "New repo created on GitHub."
rescue Octokit::UnprocessableEntity =&gt; e
  # If error everything else could fail, unless the error was that the repo already existed
  puts "Error creating repository on GitHub: #{e.message}"
end</code></pre>
<p>Finally, we can trigger the import!</p>
<pre><code class="language-ruby">gh_client.start_source_import(
  destination_repo,
  gl_authed_uri(gl_project),
  vcs: "git",
  accept: Octokit::Preview::PREVIEW_TYPES[:source_imports]
)</code></pre>
<p><code>gl_authed_uri</code> is a method defined as:</p>
<pre><code class="language-ruby">def gl_authed_uri(gl_project)
  gl_repo_uri = URI.parse(gl_project.http_url_to_repo)

  "http://oauth2:#{GL_PRIVATE_TOKEN}@#{GL_SERVER}#{gl_repo_uri.path}"
end</code></pre>
<p>If we tried to import from the GitLab repository URL without doing this, it would throw an error telling us we don&#8217;t have authorisation to access it. With this, method, we&#8217;re using our private GitLab token to gain access.</p>
<p>For extra bonus points, we can get a report of the progress of each import using Octopoller:</p>
<pre><code class="language-ruby">Octopoller.poll(timeout: 15000) do
  result = gh_client.source_import_progress(destination_repo, accept: Octokit::Preview::PREVIEW_TYPES[:source_imports])

  print "r#{result.status_text}"

  if result.status_text == "Done"
    nil
  else
    :re_poll
  end 
end</code></pre>
<p>This keeps checking the import until it returns a status of &#8220;Done&#8221;, at which point the import is complete! This makes the migration process a lot slower though, because it&#8217;ll wait until GitHub confirms that it&#8217;s done its work. If you comment out this block, then you can speed through all of the repos and let GitHub do the work in the background. It&#8217;ll email you when an import has completed.</p>
<p>We&#8217;ve made <a id="migrator-code" href="https://github.com/gotripod/gitlab-to-github-migrator" target="_blank" rel="noopener">the full source code</a> of this basic GitLab to GitHub migrator available on GitHub (unsurprisingly!). There&#8217;s some additional configuration code and a simple feature to allow resuming interrupted migrations. We don&#8217;t provide support, but you&#8217;re welcome to open issues and pull requests to collaborate on improving it. There are certainly a number of improvements that could be made (for example allowing an option to disable the progress report without having to comment it out), but we&#8217;ll leave that for another post.</p>
<p>Grab our <a href="https://github.com/gotripod/gitlab-to-github-migrator" target="_blank" rel="noopener">GitLab to GitHub migrator</a> now!</p>


Migrate from GitLab to GitHub

<p><a href="https://content.gotripod.com/images/2019/01/photo-1519635694260-6af6fc200c89.jpg" data-size="1942x1310"><img loading="lazy" class="alignnone size-large wp-image-461" src="https://content.gotripod.com/images/2019/01/photo-1519635694260-6af6fc200c89-1024x691.jpg" alt="" width="1024" height="691" srcset="https://content.gotripod.com/wp-content/uploads/2019/01/photo-1519635694260-6af6fc200c89-1024x691.jpg 1024w, https://content.gotripod.com/wp-content/uploads/2019/01/photo-1519635694260-6af6fc200c89-300x202.jpg 300w, https://content.gotripod.com/wp-content/uploads/2019/01/photo-1519635694260-6af6fc200c89-768x518.jpg 768w, https://content.gotripod.com/wp-content/uploads/2019/01/photo-1519635694260-6af6fc200c89.jpg 1942w" sizes="(max-width: 1024px) 100vw, 1024px" /></a></p>
<p>As 2018 hurtles away from us in the rear-view mirror, 2019 leaps upon us with its loving embrace. It&#8217;s time for us to look back, and forward, and take stock of where Go Tripod has been and where it&#8217;s going.</p>
<h2>Double Down on Software</h2>
<p>Last year we focused on the things we do well. We started off by <a href="https://gotripod.com/insights/a-new-look-for-go-tripod/">launching our new website</a> which saw us double our visits and increase our visitor count. We added a host of <a href="https://gotripod.com/work/">case studies</a> (with many more to come!) and, by becoming the <a href="https://gotripod.com/insights/jack-trades-master-one/">master of one trade</a>, we let those new visitors know exactly what we&#8217;re about. We made inroads into working with the larger <a href="https://gotripod.com/insights/a-caring-community/">Cornish software community</a>, something that was long coming, and we finally settled on a range of <a href="https://gotripod.com/insights/how-we-host/">hosting options that provide security</a> and peace of mind for our clients. We worked with WordPress a lot, improving <a href="https://gotripod.com/insights/ensuring-security-wordpress-websites/">security</a>, <a href="https://gotripod.com/insights/enqueue-defer-scripts-wordpress/">performance</a>, and our general <a href="https://gotripod.com/insights/managing-change-requests-within-the-development-process/">development process</a>.</p>
<h2>Kick On</h2>
<p>2018 felt like a year of consolidation, of certainty, and of improvement. In 2019, we&#8217;ll continue to review all aspects of our business, but we&#8217;ll be looking to do more research into techniques to take us to the next level. We&#8217;re working with a range of clients, new and old:</p>
<ul>
<li><a href="https://www.keysafetv.com/">Keysafe Tenant Vetting</a> to revamp their platform for a changing marketplace</li>
<li><a href="http://www.aftermarket-sanden-europe.com">Sanden International</a> to provide a new product catalogue infrastructure</li>
<li><a href="http://creativeengland.co.uk/investment/epic">The EPIC fund</a> to work on digital health products for Cornish business</li>
</ul>
<p>There are some exciting technologies coming to maturity that we&#8217;d like to leverage. <a href="https://firebase.google.com/">Google Firebase</a> has the <a href="https://firebase.google.com/products/firestore/">Firestore</a> real-time database in beta, allowing us to provide resilient offline-capable databases to our projects. Facebook continues to develop the React library, with exciting new features such as <a href="https://reactjs.org/docs/hooks-overview.html">Hooks</a> and <a href="https://github.com/facebook/react/issues/13206">Suspense</a> coming soon, and those will help us improve performance when we create complex user interfaces. WordPress has released their long-anticipated version 5.0, and we&#8217;ll be looking to see how the new <a href="https://wordpress.org/gutenberg/">Gutenberg</a> editor can improve the experience for our clients.</p>
<p>We&#8217;re also implementing some of the ideas raised during our <a href="https://gotripod.com/insights/how-we-host/">hosting research</a> and moving <a href="https://gotripod.com/work/tdk/">TDK-Lambda</a>&#8216;s infrastructure to Microsoft Azure. This will provide greater resilience for their database and application, as well as opening up exciting options for future development.</p>
<p>We&#8217;re really excited about 2019. If you&#8217;d like to hear what we could offer you in the year to come, please <a href="https://gotripod.com/contact/">get in touch</a>!</p>
<p>&nbsp;</p>


A Happy Go Tripod New Year

<p>As part of our drive to <a href="https://gotripod.com/insights/jack-trades-master-one/">double-down on software</a> and provide a great experience for our customers, we&#8217;ve been reviewing our hosting infrastructure. We recently talked about how our WordPress clients enjoy <a href="https://gotripod.com/insights/ensuring-security-wordpress-websites/">security and redundancy</a>, and we want to continue to provide that same great experience for the various web apps that we host. In this post, we&#8217;ll look at the three providers we&#8217;ve chosen to partner with in order to make this happen.</p>
<h2>Heroku</h2>
<p>Heroku is known as a &#8220;<a href="https://en.wikipedia.org/wiki/Platform_as_a_service">Platform as a Service</a>&#8221; or PAAS, which means that it provides infrastructure to run an application while abstracting things like operating system updates and configuration of the underlying system components. It integrates with source control to allow us to deploy versioned code (and roll it back in case of issues!), and the <a href="https://www.heroku.com/dynos">dyno</a> model lets us easily deal with high-traffic scenarios. Because Heroku is a cloud-based system, there&#8217;s no single point of failure, which means that if something does go wrong, chances are Heroku will automatically switch things behind the scenes so visitors never see an issue.</p>
<p>Heroku is built on Amazon&#8217;s AWS platform and, as such, it complies with accreditations such as PCI Level 1 and ISO 27001. This, on top of the huge number of Heroku addons and the excellent administration tooling, means that Heroku is our primary choice when we need to host Ruby on Rails or PHP applications.</p>
<h2>Microsoft Azure</h2>
<p>We built a bespoke ASP.NET content management system for <a href="https://gotripod.com/work/tdk/">one of our clients</a>, and since Heroku doesn&#8217;t formally support ASP.NET, we needed to look elsewhere. Azure is Microsoft&#8217;s answer to Amazon&#8217;s AWS and their <a href="https://docs.microsoft.com/en-us/azure/app-service/app-service-web-overview">Web Apps</a> provide a similar deployment model to that provided by Heroku. We can leverage Azure Storage to store uploaded documents in redundant fashion, and use Azure&#8217;s SQL Server support to deliver scalable and highly available databases. While Heroku is perhaps easier to use, Azure is more flexible and lets us work at a lower-level, and as such we&#8217;ll be moving our client&#8217;s infrastructure to Azure in 2019.</p>
<h2>Netlify</h2>
<p>For simpler sites which require little-to-no dynamic content, we turn to Netlify. It offers the same source control-based deployment model as Heroku and Azure, but it doesn&#8217;t support projects that are powered by a programming language or framework like the previously mentioned Rails, PHP or ASP.NET. Instead, its strength lies in making static sites as good as they possibly can be, with free TLS/SSL, caching support, and a redundant delivery network. If needs be, we can augment the static site with some simple dynamic features such as forms, login &amp; registration, and <a href="https://www.netlify.com/features/">custom functions</a>.</p>
<h2>Host with Go Tripod</h2>
<p>You may notice that when we talk about these providers, we&#8217;re reusing certain key terms such as versioning and redundancy. That&#8217;s because these are features that we think should be included as standard by any responsible hosting provider. By setting ourselves up for success, we can focus more on building software and less on maintaining infrastructure. If you&#8217;re looking for peace of mind when hosting your business-critical applications then <a href="https://gotripod.com/contact/">get in touch</a>.</p>


How we host

<p>We&#8217;ve recently been extending our reach into the local area by approaching organisations that help connect businesses together. The first of these is <a href="https://www.falmouth.co.uk/shopsandbusinesses/go-tripod/">Falmouth&#8217;s Business Improvement Distract (BID)</a>, who help boost the town by investing in marketing, business support, events and a variety of improvement mechanisms. The second is <a href="https://www.softwarecornwall.org/member/gotripod/">Software Cornwall</a>, who connect and support Cornwall’s digital techology community. These are the first two in a series of partnerships we&#8217;re entering into in order to better understand the needs of the local community.</p>


A Caring Software Community

<p>We recently posted about <a href="/insights/improved-font-loading/">the font loading solution we use on our websites</a>. Further to implementing this technique we noticed that, on this very site, some of the fonts weren&#8217;t quite rendering correctly.</p>
<p>Look at the examples below. The font rendering issues were slight but there was a definite problem with uneven character weighting in the Google fonts we were using (Noto Sans and Cabin).</p>
<p>It&#8217;s particularly noticeable on the crossbars of the capital Hs but you can also see that some of the ascenders are too bold. Capital Ss also show problems where their terminals on the cap line are bolder than their finials on the base line. There is a similar issue with the bold lower case C. Finally, the apices of the lower case Ts aren&#8217;t really rendering at all!</p>
<p>(Also note that the rendering issues are only obvious in the first paragraph of text and in the bold title. The second paragraph of text appears to be rendering correctly.)</p>
<p><div id="attachment_443" style="width: 1034px" class="wp-caption alignnone"><a href="https://content.gotripod.com/images/2018/07/font-comparison.png" data-size="1920x636"><img aria-describedby="caption-attachment-443" loading="lazy" class="wp-image-443 size-large" src="https://content.gotripod.com/images/2018/07/font-comparison-1024x339.png" alt="Comparative screenshots demonstrating font rendering" width="1024" height="339" srcset="https://content.gotripod.com/wp-content/uploads/2018/07/font-comparison-1024x339.png 1024w, https://content.gotripod.com/wp-content/uploads/2018/07/font-comparison-300x99.png 300w, https://content.gotripod.com/wp-content/uploads/2018/07/font-comparison-768x254.png 768w, https://content.gotripod.com/wp-content/uploads/2018/07/font-comparison.png 1920w" sizes="(max-width: 1024px) 100vw, 1024px" /></a><p id="caption-attachment-443" class="wp-caption-text">Font rendering issues (left) vs. correct font display (right). Click or tap image to enlarge.</p></div></p>
<p>Tying down what was causing this proved tricky for the following reasons:</p>
<ul>
<li>The problems are only apparent on Windows machines</li>
<li>The problems are more apparent in IE than Chrome</li>
<li>The problems are only apparent with certain font families</li>
<li>The problems are only apparent at certain font size/weight combinations</li>
</ul>
<p>So, to get to the bottom of this we needed to unpick our font loading solution&#8230; We use <a href="https://www.fontsquirrel.com/tools/webfont-generator" target="_blank" rel="noopener">Font Squirrel&#8217;s Webfont Generator</a> to generate compressed <abbr data-tooltip="Web Open Font Format">WOFF</abbr> and WOFF2 files from the original <abbr data-tooltip="TrueType Font">TTF</abbr>s.</p>
<p>The generator offers three modes: Basic, Optimal, Expert. We&#8217;d entrusted it to &#8216;Optimal&#8217; and it transpires that this was causing our problems.</p>
<p>Font Squirrel&#8217;s &#8216;Expert&#8217; mode allows you to adjust a whole host of settings. (See below for a subset of these.)</p>
<p><div id="attachment_441" style="width: 685px" class="wp-caption alignnone"><img aria-describedby="caption-attachment-441" loading="lazy" class="wp-image-441 size-full" src="https://content.gotripod.com/images/2018/07/font-suirrel-generator-settings.gif" alt="Font Squirrel generator settings" width="675" height="374" /><p id="caption-attachment-441" class="wp-caption-text">Font Squirrel webfont generator settings</p></div></p>
<p>It turns out that the culprit in this instance is Font Squirrel&#8217;s <strong>Truetype Hinting </strong>algorithm. <a href="https://en.wikipedia.org/wiki/Font_hinting" target="_blank" rel="noopener">Font hinting</a> is a complicated matter, <a href="https://blog.typekit.com/2010/12/14/a-closer-look-at-truetype-hinting/" target="_blank" rel="noopener">the nuances of which</a> are beyond the scope of this post, however, suffice to say that automating the hinting process via the default &#8216;Font Squirrel&#8217; setting proved problematic.</p>
<blockquote><p>When we set Truetype Hinting to &#8216;Keep Existing&#8217; or &#8216;TTFAutohint&#8217; our problems vanished.</p></blockquote>
<p>It&#8217;s worth noting that we&#8217;ve not seen this issue on any other projects so this setting must only affect certain fonts negatively. However, as a bonus, this change also involves a file size saving for the variants of Noto Sans our website uses. See the comparison below:</p>
<p><div id="attachment_442" style="width: 1034px" class="wp-caption alignnone"><a href="https://content.gotripod.com/images/2018/07/font-file-sizes.gif" data-size="1440x287"><img aria-describedby="caption-attachment-442" loading="lazy" class="wp-image-442 size-large" src="https://content.gotripod.com/images/2018/07/font-file-sizes-1024x204.gif" alt="Comparative font sizes showing that 'Keep Existing' hinting option results in ~10% saving " width="1024" height="204" srcset="https://content.gotripod.com/wp-content/uploads/2018/07/font-file-sizes-1024x204.gif 1024w, https://content.gotripod.com/wp-content/uploads/2018/07/font-file-sizes-300x60.gif 300w, https://content.gotripod.com/wp-content/uploads/2018/07/font-file-sizes-768x153.gif 768w" sizes="(max-width: 1024px) 100vw, 1024px" /></a><p id="caption-attachment-442" class="wp-caption-text">Noto Sans font file size comparisons with different hinting options (&#8216;Font Squirrel&#8217; on the left, &#8216;Keep Existing&#8217; on the right)</p></div></p>
<p>As such we&#8217;ve now added a config file which includes this change (<code>"tt_instructor": "keep"</code>) to our boilerplate repo.</p>
<p>Moral of the story &#8211; leave font hinting to the font designer!</p>


WOFF font rendering issue on Windows, fixed

<p>As developers, we have a seemingly endless supply of tools at our disposal thanks to the hard work and generosity of the wider development community who, by and large, share their hard work in the hope of making their peers&#8217; lives that little bit easier.</p>
<p>At Go Tripod we&#8217;re always on the lookout for new ways to streamline our processes and improve the quality of our work. Task runners are a great example of this.</p>
<p>Utilising JavaScript, programs like <a href="https://gulpjs.com/" target="_blank" rel="noopener">Gulp</a> and <a href="https://gruntjs.com/" target="_blank" rel="noopener">Grunt</a> can automate all kinds of labour-intensive tasks including, for example:</p>
<ul>
<li>compiling, concatenating and compressing assets (eg. images, <abbr data-tooltip="Cascading Style Sheets">CSS</abbr>, <abbr data-tooltip="JavaScript">JS</abbr>)</li>
<li>live browser reloading for speedy development and testing</li>
<li>auto-prefixing for optimal browser support</li>
</ul>
<p>With these jobs running automatically in the background we&#8217;re able to focus entirely on the task at hand. This increases productivity and makes sure the code we&#8217;re writing is as performant and compatible as it can be.</p>
<p>Our current build tool of choice is a modified version of the excellent <a href="https://github.com/ahmadawais/WPGulp" target="_blank" rel="noopener">WPGulp</a> by WordPress aficionado, <a href="https://ahmadawais.com/" target="_blank" rel="noopener">Ahmad Awais</a>. It does most of the things we need out of the box but since we use <a href="https://www.upstatement.com/timber/" target="_blank" rel="noopener">Timber</a> to separate logic from styling I&#8217;ve amended it to watch <code>.twig</code> files <em>as well</em> as <code>.php</code> files.</p>
<p><div id="attachment_434" style="width: 640px" class="wp-caption alignnone"><img aria-describedby="caption-attachment-434" loading="lazy" class="size-full wp-image-434" src="https://content.gotripod.com/images/2018/07/Screen-Shot-2018-07-17-at-09.29.26.png" alt="" width="630" height="411" srcset="https://content.gotripod.com/wp-content/uploads/2018/07/Screen-Shot-2018-07-17-at-09.29.26.png 630w, https://content.gotripod.com/wp-content/uploads/2018/07/Screen-Shot-2018-07-17-at-09.29.26-300x196.png 300w" sizes="(max-width: 630px) 100vw, 630px" /><p id="caption-attachment-434" class="wp-caption-text">WPGulp compressing assets and watching for changes</p></div></p>
<p>Along with a few other minor adjustments, the result is a fast and feature-packed workflow which allows us to build WordPress sites of the highest quality in record time. 💪</p>


The best Gulp build tool for WordPress

<p>Ten years ago, just before Go Tripod was formed, I was an independent contractor with a strong eye on my bottom line, so <a href="https://colinramsay.co.uk/personal/projects/php/freshbooks/plastiscenic/2008/11/27/freshbooks-invoice-widget.html" target="_blank" rel="noopener">I created a way to track my income</a> at-a-glance. This was extremely useful at the time, as it helped a young Colin make sure he had enough pennies at the end of each month, and we now use a much more advanced version at GT to make sure we&#8217;re bringing in some good dollar every year.</p>
<h2>The business challenge</h2>
<p>We&#8217;d like to be able to quickly get information on our current financial performance, including monthly tallies and current debtors and creditors. This data should be available in a frictionless manner &#8211; without needing to log in to accounting software, crunch numbers or run reports.</p>
<h2>The solution (In a nutshell)</h2>
<p>In my decade-old solution, I was using Freshbooks for accounting. At GT we now use Xero, a cloud-based accounting program that makes the whole process of invoicing, filing returns and tracking payments much easier. We worked out a way to pull our financial information out of Xero and have it posted into Slack, the chat service that our team uses each day. We wrote a chatbot, called Yus, which we can use to request vital information about how well Go Tripod is performing. Yus will also give us a breakdown of bills that we need to pay and invoices that are outstanding.</p>
<h2>The solution (The software side)</h2>
<p>We built Yus using a combination of <a href="https://botkit.ai/" target="_blank" rel="noopener">Botkit</a> and <a href="https://xeroapi.github.io/xero-node/" target="_blank" rel="noopener">xero-node</a>. It runs on <a href="https://www.heroku.com/home" target="_blank" rel="noopener">Heroku</a>. Here&#8217;s a sample of what it can do:</p>
<p><div id="attachment_426" style="width: 680px" class="wp-caption alignnone"><a href="https://content.gotripod.com/images/2018/06/yus-1.gif" data-size="670x368"><img aria-describedby="caption-attachment-426" loading="lazy" class="wp-image-426 size-full" src="https://content.gotripod.com/images/2018/06/yus-1.gif" alt="Invoking '@yus invoicing this month' in Slack" width="670" height="368" /></a><p id="caption-attachment-426" class="wp-caption-text">Yus giving us invoicing figures</p></div></p>
<p>We can send Yus simple phrases and it&#8217;ll respond accordingly. This time round it&#8217;s decided to exaggerate somewhat 🤑</p>
<p><div id="attachment_427" style="width: 680px" class="wp-caption alignnone"><a href="https://content.gotripod.com/images/2018/06/yus-2.gif" data-size="670x368"><img aria-describedby="caption-attachment-427" loading="lazy" class="wp-image-427 size-full" src="https://content.gotripod.com/images/2018/06/yus-2.gif" alt="Invoking '@yus who owes us' in Slack" width="670" height="368" /></a><p id="caption-attachment-427" class="wp-caption-text">Yus shows us which clients have outstanding invoices over 30 days old</p></div></p>
<p>In this case we get a list of clients who we might otherwise not have chased for payment until our bookkeeper flagged this up. Now, Yus can help us liaise with our clients to make sure invoices don&#8217;t slip through the cracks.</p>
<h2>Conclusion</h2>
<p>Transparency is key. Low friction transparency is even better. By using chatbot software we can listen to the heartbeat of Go Tripod and take immediate steps to perform course corrections. Fancy building some clever technology to improve your business? <a href="https://gotripod.com/contact/">Get in touch</a>.</p>
<h2>A little word about Yus</h2>
<p>The name &#8220;Yus&#8221; comes from two letters in the Cyrillic script: Little yus (Ѧ ѧ) and big yus (Ѫ ѫ). We use this character to represent a tiny tripod in our email footers.</p>


Using chatbot software to improve transparency – introducing Yus

<p>An early temptation when setting up a development shop is to diversify: to provide logo design, SEO assistance, print work, marketing. The ability to sell yourself to clients as a one stop shop is compelling, with the idea being that you can capture more work and provide a better experience for customers.</p>
<p>At Go Tripod, we don&#8217;t work that way.</p>
<p>We&#8217;re software developers and web experts. We like focus. By being up front and telling clients that we concentrate on what we do best, we can avoid spreading ourselves too thinly. The flip side of being a one stop shop is that there&#8217;s a lack of group knowledge about software best practices, such as using source control, automated testing, or keeping up to speed with the latest tools.</p>
<p>What happens when our clients do need other services? Then we hit our network. We can reach out to those who are using our approach: focusing on what they do best. That means that we can call upon the outstanding people in a particular field, bringing in specialists where needed. We&#8217;ll use Teamwork to bring everyone together to make sure the lines of communication remain clear, and in doing so can create projects that are greater than the sum of their parts.</p>
<p>We know code, and we know the web. <a href="https://gotripod.com/contact/">Let us show you</a>.</p>


Jack of all trades? Or master of one?

<p>We&#8217;re in the early days of planning out a web design-and-build project with a new client. A key part of this process is the creation of a site map.</p>
<h3>The Site Map</h3>
<p>In its simplest form, a site map is a pictorial representation of the pages required of a website. With smaller websites this list of pages is easy to define and it&#8217;s not much more of a stretch to group them into a sensible order.</p>
<p>With larger web projects this task can be much more involved and producing a sensible site map relies on a host of other disciplines (like information architecture and user experience design) but the requirements of the job at hand fall mostly into the former camp.</p>
<p>It&#8217;s worth demonstrating that a sensible site map doesn&#8217;t <em>merely</em> need to show a breakdown of a website&#8217;s pages. Instead it should also show how they fit together, where multiple pages might be grouped via dynamic data sources and what kind of elements might be common to every page.</p>
<p>The trick is fitting all of this information into a straight forward infographic that stakeholders can immediately understand (and preferably print).</p>
<p>Enter our case in point (click to expand):</p>
<p><a href="https://content.gotripod.com/images/2018/04/site-map.png" data-size="1262x883"><img loading="lazy" class="alignnone wp-image-274 size-full" src="https://content.gotripod.com/images/2018/04/site-map.png" alt="Example website site map" width="1262" height="883" srcset="https://content.gotripod.com/wp-content/uploads/2018/04/site-map.png 1262w, https://content.gotripod.com/wp-content/uploads/2018/04/site-map-300x210.png 300w, https://content.gotripod.com/wp-content/uploads/2018/04/site-map-768x537.png 768w, https://content.gotripod.com/wp-content/uploads/2018/04/site-map-1024x716.png 1024w" sizes="(max-width: 1262px) 100vw, 1262px" /></a></p>
<p>Note that I have used the homepage to define the site-wide header and footer information as well as the main navigation options. It is clear each navigation option represents a top tier page and it is immediately obvious which option includes additional features, pages or sub-sections.</p>
<p>(Strictly speaking the header and footer should be represented on every page, rather than the homepage alone, but to harness the power of an infographic one must expect some inferral ability on the part of our audience.)</p>
<h3>The User Flow Diagram</h3>
<p>User flow diagrams differ somewhat from site maps in so far as they attempt to demonstrate how a user interacts with a website rather than how the website&#8217;s contents are hierarchically broken down.</p>
<p>Naturally there is some crossover between these two approaches and again the complexity of a user flow diagram will depend on the complexity of the website (or process) concerned.</p>
<p>One way to think of these things is that whilst both diagrams might utilise boxes and arrows, the user flow diagram is more likely to take the shape of a flow chart.</p>
<p>With our current project, there was a loosely defined need for a restricted client area that would allow logged in users to access (mostly product based) information unavailable to regular visitors.</p>
<p>Whereas our client had been quite clear about what content was required of the front-end, defining this &#8216;restricted area&#8217; proved much more elusive. Instead we were given a long list of bullet point thoughts and ideas regarding things they might like.</p>
<p>As it happens, attempting to formalise these loosely defined requirements suits the user flow approach better than the site map approach.</p>
<p>And here is the case in point (click to expand):</p>
<p><a href="https://content.gotripod.com/images/2018/04/user-flow-diagram.png" data-size="1262x883"><img loading="lazy" class="alignnone wp-image-275 size-full" src="https://content.gotripod.com/images/2018/04/user-flow-diagram.png" alt="Example website user flow diagram" width="1262" height="883" srcset="https://content.gotripod.com/wp-content/uploads/2018/04/user-flow-diagram.png 1262w, https://content.gotripod.com/wp-content/uploads/2018/04/user-flow-diagram-300x210.png 300w, https://content.gotripod.com/wp-content/uploads/2018/04/user-flow-diagram-768x537.png 768w, https://content.gotripod.com/wp-content/uploads/2018/04/user-flow-diagram-1024x716.png 1024w" sizes="(max-width: 1262px) 100vw, 1262px" /></a></p>
<p>Note that this diagram demonstrates how a user might interact with the website and that the flow, in this instance, is shown from left to right.</p>
<p>Starting with the login process, the customer has a number of options; accessing important information available from their dashboard, amending their account details, or drilling down into specific product information.</p>
<p>Again, stakeholders can immediately make sense of what&#8217;s been proposed. This diagram should, after all, reflect a sensible interpretation of the client&#8217;s hastily compiled requirement list.</p>
<p>But also, importantly, it serves as an early conceptual blueprint that encourages client feedback. The flow diagram, once finalised, will additionally help formalise the development process.</p>
<p>In conclusion &#8211; a collection of carefully thought out infographics can serve as a powerful tool in terms of both defining a new project and improving an existing website or set of software processes.</p>
<p>&nbsp;</p>


Project planning with site maps and user flow diagrams

<p>Back in February <a href="https://gotripod.com/insights/improving-site-response-times-deferring-downloading-web-fonts/">I wrote about a font-loading technique I&#8217;d implemented on a new project</a>, but there was a problem with it &#8211; the script ran on every page load meaning the browser cache was never used to serve the fonts. There are many variations on the same theme with some using session cookies to determine if it&#8217;s a user&#8217;s first visit to the site, but they all come with their own quirks to overcome. After further research and testing I decided that, while not fully supported yet, the most effective and future-proof solution to this issue is not a JavaScript solution, but a CSS one.</p>
<h2>Enter our old friend, <code>@font-face</code></h2>
<p>The <code>@font-face</code> rule has been around as long as I can remember but went steadily out of favour as the popularity of <abbr data-tooltip="Flash Of Unstyled Text">FOUT</abbr> techniques grew. This is because its default behaviour results in a <abbr data-tooltip="Flash Of Invisible Text">FOIT</abbr> while the font is downloaded and with the rise of mobile devices on sluggish connections users can be left with a blank page for seconds or even minutes. This is no longer the case thanks to a new attribute, <code>font-display</code>.</p>
<p><a href="https://caniuse.com/#search=font-display" target="_blank" rel="noopener">Browser support for <code>font-display</code> is growing</a> with the latest versions of Firefox, Chrome and Safari all playing ball. It degrades gracefully for all other browsers, falling back to its default behaviour and, as an added perk, we don&#8217;t need to worry about users with JavaScript disabled.</p>
<h2>Optimising fonts</h2>
<p>Here at Go Tripod we use <a href="https://fonts.google.com/" target="_blank" rel="noopener">Google Fonts</a> for most projects but in their default format (<abbr data-tooltip="TrueType Font">TTF</abbr>) file sizes can be much larger than is required. On slower connections this means the web fonts will take noticeably longer to swap with the system fonts and it&#8217;s best we do everything to deliver these assets to the user as quickly as possible. We use <a href="https://www.fontsquirrel.com/tools/webfont-generator" target="_blank" rel="noopener">Font Squirrel&#8217;s Webfont Generator</a> to convert our TTFs from Google into <abbr data-tooltip="Web Open Font Format">WOFF</abbr> and WOFF2 formats which often reduces their size by over 50% meaning the time it takes to download them is significantly decreased. This coupled with the <code>font-display: swap;</code> attribute means we&#8217;re providing most users with a superior experience which doesn&#8217;t rely on JavaScript, falls back gracefully and will soon be supported by all major browsers.</p>


Improved font loading

<p>Go Tripod is based at Tremough Innovation Centre in Penryn, Cornwall, but our team members work remotely from locations of their choosing. By giving our staff the freedom to create their own environment, they&#8217;re able to work smarter and in the way they prefer.</p>
<p>How do we make sure that everyone&#8217;s on the same page, providing our clients with the best service we can? We&#8217;ve got a suite of tools that help us coordinate and communicate.</p>
<h2>Email</h2>
<p>It may be seen as old hat but for most companies, including us, email is still a huge part of our operation. It&#8217;s often the first point of contact for new enquiries, and it&#8217;s an important mechanism for us to keep in contact with our clients.</p>
<h2>Slack</h2>
<p>We use <a href="https://www.slack.com">Slack</a> as our main means of communicating internally. It&#8217;s a chat system that allows us to have distinct channels for a variety of different topics and typically we have one for each project. This gives our discussions focus, and allows us to have a history of what&#8217;s been said (which is searchable for future reference). If we need to bring in a partner, such as a client or a contractor, then we can invite them to a channel as a guest so we can collaborate more easily.</p>
<h2>Teamwork</h2>
<p>As a project progresses, we track the outstanding items and the evolution of features using a project management tool called <a href="https://www.teamwork.com/project-management-software">Teamwork</a>. Our clients (<a href="https://gotripod.com/contact/">you could be one too!</a>) get their own login so they can participate in discussions, provide feedback and help us understand any issues. We can also track our time to invoice accurately, and we can see, at-a-glance, the status of any of our current projects.</p>
<h2>Appear.in</h2>
<p>No matter how fancy our chatroom and project management tools are, sometimes a phone call is the best way of getting a point across. We&#8217;ll often use <a href="https://www.appear.in">appear.in</a> to share a screen between the team to demonstrate a new feature or discuss an implementation, but we&#8217;ll also use it to let our team catch up. It supports webcams too so we can even look at each other&#8217;s ugly mugs!</p>
<h2>Yodel.io</h2>
<p>Since we spend so much time on Slack, we wondered if we could hook phone calls into it. It turns out we can with <a href="https://www.yodel.io/">Yodel.io</a>. When someone calls our office number, if no-one&#8217;s there to pick up, it&#8217;ll be routed into Slack where it can be answered in the browser with the press of a button. We can even redirect the call to the correct person&#8217;s mobile or send it to a customised voice message if we&#8217;re all unavailable.</p>
<p>By using this set of tools we can work more efficiently and deliver better value to our clients. Working remotely is becoming increasingly popular, and at Go Tripod we&#8217;ve been remote-working advocates since our formation in 2009. In fact, we think that working remotely enables us to provide a better service. Why not <a href="https://gotripod.com/contact">try us out</a>?</p>


We work remotely – here’s why and here’s how

<p>We&#8217;ve worked with various content management systems (<abbr>CMS</abbr>) over the years but <a href="https://wordpress.org/" target="_blank" rel="noopener">WordPress</a> is our favourite. Thanks to its powerful user interface and open source architecture, it&#8217;s the most popular <abbr>CMS</abbr> on the planet. (In fact <a href="https://w3techs.com/technologies/history_overview/content_management" target="_blank" rel="noopener">its market share</a> is 10 times the size of its closest rival!)</p>
<p>Whilst WordPress empowers millions of users to publish their own content, it also attracts hackers eager to find and exploit vulnerabilities in the WordPress system itself and (more often) in the myriad 3rd party WordPress plugins. As such, security issues are found on a regular basis and plugged just as regularly with frequent updates.</p>
<p>At Go Tripod, we thoroughly test any updates in a staging environment to ensure the new code doesn&#8217;t cause any problems before it&#8217;s pushed live to our clients&#8217; websites. We maintain the same care and attention throughout our development practice, investing time into researching the best tool for the job at hand.</p>
<p>So, to ensure our WordPress clients don&#8217;t need to worry about the security of their own websites, we have recently partnered with a dedicated WordPress hosting company.</p>
<p>This means our WordPress clients enjoy:</p>
<ul>
<li>daily backups restored with a single click</li>
<li>automated <abbr data-tooltip="Secure Sockets Layer">SSL</abbr>/<abbr data-tooltip="Transport Layer Security">TLS</abbr> certificates via <a href="https://letsencrypt.org/" target="_blank" rel="noopener">Let&#8217;s Encrypt</a></li>
<li>unlimited development/staging environments</li>
<li>rapid environment updates via <a href="https://git-scm.com/" target="_blank" rel="noopener">Git</a></li>
<li>a built-in one-click <abbr data-tooltip="Content Delivery Network">CDN</abbr></li>
</ul>
<p>Do you have a WordPress site? Is it secure? <a href="/contact/" rel="noopener">Get in touch</a> if you&#8217;d like to know more about what we can offer.</p>


Ensuring the security of WordPress websites

<p>At Go Tripod, there are many ways in which we improve our working efficiency. We pick the best technologies, such as WordPress, React or Ruby on Rails. We pick the best services, such as Teamwork, Slack or Xero. But we spend our days sat in front of a computer and the way we interface with these technologies and services is just as important as how we select them.</p>
<p>When coding, the primary interface with code is via an editor, be that an IDE such as <a href="https://www.jetbrains.com/webstorm/" target="_blank" rel="noopener">WebStorm</a> or a classic editor like <a href="https://en.wikipedia.org/wiki/Vi" target="_blank" rel="noopener">vi</a>. My current editor of choice is <a href="https://code.visualstudio.com/" target="_blank" rel="noopener">Visual Studio Code</a>, built from the ground up by Microsoft to be a fast and extensible editor for coders. In this post I&#8217;m going to show a couple of ways in which VS Code improves my day-to-day productivity.</p>
<h2>Multiple Cursors</h2>
<p>If I need to repeat something, it&#8217;s going to quickly become tedious, and I&#8217;m more likely to make mistakes. How about if I can add magical automatic cursors that replicate my every keystroke?</p>
<p><img loading="lazy" class="alignnone wp-image-249 size-full" src="https://content.gotripod.com/images/2018/04/multi.gif" alt="a demonstration of multiple cursors at work" width="353" height="86" /></p>
<p>In this GIF, I wanted to amend the hyphen to a colon. I could use find and replace, but it&#8217;s also a good opportunity to try out multiple cursors. I select all lines, then press CMD+Shift+L to add cursors at the end of each line. Then I  hold down Alt and move left to skip over the last word, and then begin editing the hyphens on all lines at once. Knowing the Alt-skip trick is invaluable, because it means you can move your cursors over a whole word at once and keep them aligned with punctuation over multiple lines. Without this, your cursors will become out of sync as you move over characters one-by-one.</p>
<h2>Git Merge Conflicts</h2>
<p>We love <a href="https://git-scm.com/" target="_blank" rel="noopener">git</a>, but hate resolving merge conflicts. These occur when two people have worked on the same file and the work needs to be combined, or merged, into a single version. VS Code&#8217;s interface for resolving these is quick and intuitive:</p>
<p><a href="https://content.gotripod.com/images/2018/04/merge-conflict.png" data-size="842x508"><img loading="lazy" class="alignnone wp-image-254 size-medium" src="https://content.gotripod.com/images/2018/04/merge-conflict-300x181.png" alt="resolving merge conflicts in VS Code is intuitive" width="300" height="181" srcset="https://content.gotripod.com/wp-content/uploads/2018/04/merge-conflict-300x181.png 300w, https://content.gotripod.com/wp-content/uploads/2018/04/merge-conflict-768x463.png 768w, https://content.gotripod.com/wp-content/uploads/2018/04/merge-conflict.png 842w" sizes="(max-width: 300px) 100vw, 300px" /></a></p>
<p>As well as the UI in the editor pane that enables you to choose which change to select, the file explorer on the left on VS Code will show any files in your project with outstanding merge conflicts, and the gutter on the right-hand side of the editor will show any merge conflicts in the currently open file.</p>
<h2>VS Code Extensions</h2>
<p>While VS Code isn&#8217;t unique in allowing extensions, I would say that it is unique in the quantity and quality of extensions that the community has created. Language support for VS Code is detached from the editor itself, instead being supplied by extensions. This means that languages such as go, python, and rust can be written in VS Code and use editor features such as autocomplete and refactoring. Here&#8217;s a selection of the extensions I have installed:</p>
<h3>Bookmarks</h3>
<p><a href="https://github.com/alefragnani/vscode-bookmarks.git" target="_blank" rel="noopener">https://github.com/alefragnani/vscode-bookmarks.git</a></p>
<p>When I have multiple files open, I want to be able to quickly navigate between them. This extension lets me set bookmarks in a file, which I can skip back to at the press of a button.</p>
<h3>WordCounter</h3>
<p><a href="https://github.com/Kirozen/vsce-wordcounter" target="_blank" rel="noopener">https://github.com/Kirozen/vsce-wordcounter</a></p>
<p>This extends the VS Code status bar to include the number of words and characters in a document or a selection.</p>
<h3>Prettier</h3>
<p><a href="https://github.com/prettier/prettier-vscode" target="_blank" rel="noopener">https://github.com/prettier/prettier-vscode</a></p>
<p>Prettier is a code formatting tool which I primarily use for JavaScript and TypeScript. This extension will highlight any problems with my code formatting and even correcting it when I type or save.</p>
<h2>Conclusion</h2>
<p>VS Code brings together many different features to create a tool that improves my productivity. Take a look at the full documentation on the <a href="https://code.visualstudio.com/docs" target="_blank" rel="noopener">VS Code website</a> and learn how to work smarter.</p>


Work smarter, not harder, with VS Code

<p>Software design is a complicated business. To illustrate this point, here is a user flow diagram from a previous project of ours. This level of detail is useful for a number of reasons; it gives the client a real feel for the end product before the need for any coding, it aids in the design process and it helps to formalise pre-development requirements.</p>
<p><img loading="lazy" class="aligncenter wp-image-243 size-full" src="https://content.gotripod.com/images/2018/03/userflow-01b.jpg" alt="User flow example image" width="738" height="932" srcset="https://content.gotripod.com/wp-content/uploads/2018/03/userflow-01b.jpg 738w, https://content.gotripod.com/wp-content/uploads/2018/03/userflow-01b-238x300.jpg 238w" sizes="(max-width: 738px) 100vw, 738px" /></p>
<p>Complications can arise when, down the line, it becomes apparent that a functional change is necessary. With all but the most menial of software, this scenario is inevitable.</p>
<p>By practising agile software development we allow for multiple changes during the design and build processes. These changes still need defining, but needn’t derail a project or upset its budget. In the example below what had been specified as an ‘About page’ within the app later became an ‘About section’. (Click the image to enlarge.)</p>
<p><a href="https://content.gotripod.com/images/2018/03/userflow-02.jpg" data-size="1463x994"><img loading="lazy" class="aligncenter wp-image-244 size-full" src="https://content.gotripod.com/images/2018/03/userflow-02.jpg" alt="User flow change request example image" width="1463" height="994" srcset="https://content.gotripod.com/wp-content/uploads/2018/03/userflow-02.jpg 1463w, https://content.gotripod.com/wp-content/uploads/2018/03/userflow-02-300x204.jpg 300w, https://content.gotripod.com/wp-content/uploads/2018/03/userflow-02-768x522.jpg 768w, https://content.gotripod.com/wp-content/uploads/2018/03/userflow-02-1024x696.jpg 1024w" sizes="(max-width: 1463px) 100vw, 1463px" /></a></p>
<p>We do our utmost to clearly define our software at the start of a project, but we have a well defined process for managing changes as and when the need arises. This enables us to crack on with development without getting bogged down in specification minutiae.</p>
