Optimize Your Site’s Speed: Convert Images to WebP Format

upgrading image files

In a recent post I discussed auditing existing websites for potential performance enhancements. In the process I discovered issues with the AntPace.com Lighthouse estimates (a topic I covered briefly in a post about progressive web apps).

Lighthouse performance estimate

My performance only scored 50 out of 100. One “opportunity” (estimated to save 2.16s) was to “serve images in next-gen formats”. AVIF (based on the AV1 video codec) and WebP are considered next generation because of “superior compression and quality characteristics compared to their older JPEG and PNG counterparts“. I chose to upgrade my images WebP (instead of AVIF) mostly due to greater browser support.

Before I upload images to this site, I usually run them through TinyPNG. Sometimes, I’ll also do some manual pre-processing in the GIMP, like cropping or scaling images down. I know that there  must be a better way to do all of that through a proper CI pipeline – but for now that is my low tech solution. Now that I need to come up with a new process for next-gen image formats, it might be a good time to explore a more automated solution.

Properly Size an Image for Your Website

Site speed is important for user experience and SEO. Having an image with unnecessarily large dimensions eats up bandwidth and leads to slow load times. You can open an image file in the GIMP and use the scale tool to reduce the size while maintaining the ratio. Once it is reduced, you can copy and paste it to it’s own canvas and save the file.

I write more about using this image manipulation program in another article about design for programmers

Convert multiple images to next-gen WebP file format

For converting single images, manually, one at a time I could use the GIMP or an online service. I used a command line tool on my mac called webp tools to bulk upgrade all of the files in an image directory.

brew install webp

For a single file I can use this command:

cwebp -q 80 software-education.png -o software-education.webp

To hit all of my image files in that folder I ran loops over the existing file formats. You should change this to include any other file formats used in your project.

for i in *.jpg; do cwebp -q 80 "$i" -o "${i%.jpg}.webp";done
for i in *.png; do cwebp -q 80 "$i" -o "${i%.png}.webp";done

I saved them to a new folder, so that I could easily delete all of the old files at once and then replace them. Alternatively, I can save them in the same directory and then run a command to delete all of the .png files: rm *.png

New folder amongst legacy image files

My website has been around for a while, so there are unused legacy media files. I manually searched my code base (ctrl + shift + f) for the file names listed in the various /image directories throughout my project, and deleted them if I found no references. (Maybe having so many disjointed image folders is part of a bigger problem with this project). If I did find it, I updated the file extension to webp. Later, I wrote a script to automatically find and delete unused image files from a project.

This change increased my Lighthouse performance score by ten points (I gained an additional two but upgrading MariaDB from version 10.2 to 10.4). I did this process is a few other directories of my project to complete the upgrade. I was going to convert my Apple touch icon files to a next-gen format too, but the documentation specify the use of the PNG format (good thing I checked).

WordPress and WebP

Having adopted WebP as the new standard file format for AntPace.com, I decided to upload .webp images to my posts – starting with this one. When I tried, WordPress complained: “This image cannot be processed by the web server. Convert it to JPEG or PNG before uploading.”

error from wordpress when trying to upload a next generation image file

This surprised me. I upgraded WordPress to the latest version (6.3.1 at that time), but it didn’t help. After further investigation, it looked like the problem had to do with the PHP image module gd.

I SSH’d into my EC2 instance to see what I could do. When I tried to install gd for my PHP version 7.4, I got a version mismatch error. It had to due with Amazon Linux 2 (the OS I run on AWS EC2) not supporting PHP 7.4 as it approaching or have passed their end-of-support dates. Every time I tried to installthe gd module, Amazon Linux Extras (the default Amazon Linux 2 package mechanism) would try to pull the version compatible with PHP 7.2. In an attempt to make it work, I manually disabled ‘amazon-linux-extras’, installed the Remi repository and made sure it was prioritized as my package manager. Still, “Packages skipped because of dependency problems”.

Amazon Linux 2 is at EOL.

The same thing happened when I tried using ImageMagick instead. This made me consider my Linux distribution. Not having gd installed what causing other problems when uploading media through WordPress (responsive image sizes are not being generated).

I had been wanting to upgrade the size of my EC2 instance anyway, so this might be the right time. I am considering Amazon Linux 2023 or a Bitnami image.  As you know, I’ll write a blog post about which I choose and the implementation details

Look-and-Say in PHP

The look-and-say sequence is a series of integers. It can grow indefinitely. It is generated by reciting a number phonetically, and writing what you spoke numerically. Its popularity is attributed to famed cryptographer Robert Morris. It was introduced by mathematician John Conway. It looks like this:

1
11
21
1211
111221
312211
13112221

The first line would be pronounced as “one 1”, and then written as “11” on the second line. That record would be spoken as “two 1’s”,  giving us the third line “21”. The greatest individual symbol you’ll ever find in this consecution is a 3.

This topic has lots of trivia, variations, and history that could be dug up and expounded upon. Here, I’ll explain a solution written in PHP to produce this chain of numerals. The input will be the count of how many lines, or iterations, in the series to generate. Below is the code:

<?php

echo "Count And Say: \n";

function countAndSay($count=0){
	$value = 1; // initial seed
	for($i=1;$i<=$count;$i++){
		echo $value . "\n";
		$value = calcOutput($value);
	}
}
function calcOutput($value){
	$value = "$value";  // change it into a string, so we can iterate over each character
	$current = $value[0]; // first character
	$count = 1;
        $return = '';
	for ($i = 1; $i <= strlen($value); $i++) { // keep going until we get through the whole string
		if ($current != $value[$i] || $i == strlen($value)) { // found a different character, or end of the input string
			$return .= "$count$current";
			$count = 1; // reset count
			$current = $value[$i]; // set new current character
		} else {
			$count++;
		}
	}
	return $return;
}

countAndSay(7);

echo "\n\n";

?>

I separated my code into two functions. I think this is the best approach. As an exercise, see if you can figure out how to refactor it into one. This could help you to internalize the logic as you write it out for yourself.

The initial seed value is “1”, and that is hard-coded at the top. The for-loop iterates based on the count input parameter. That means the code circles back and re-runs, with updated values, until its internal count (represented by the variable $i ) matches the $count variable passed into countAndSay($count).

The code that we loop over outputs the current sequence value (starting with 1) as its own line (“\n” will output a new line in most programming languages) , and then calculates the next. The function that determines the next line of output, calcOutput($value), takes the current value as an argument.

The first thing we do is cast the integer value passed along into a string. This lets us refer to each character by index – starting at zero – and save it to a variable $current. We start a new $count, to keep track of how many times we see the same digit.

The next for-loop executes for the length of the $value string. On each loop, we check if the $current character we saved matches the subsequent one in that $value string. It is again referenced by index, this time based on the for-loop’s iteration count represented by the variable $i.

If it does match, one is added to the $count variable that is keeping track of how many times we see the same character is a row. If it doesn’t match (or we’ve reached the end of the input), the $count and $current number are concatenated to the $return element. At that point, the $count is reset to 1, and the $current value is updated.

Writing an algorithm to generate the look-and-say (also known as, count-and-say) sequence is a common coding puzzle. You might run into it during a job interview as a software engineer. As practice, see if you can simplify my example code, or even write it in a different programming language than PHP.

Develop Apps and Explore the World Wide Web

BJJ Tracker

World Wide Web

The web, as a platform, is open and free. Unlike native app markets, we don’t have to wait for software to be approved by any third-party. It works across any device or operating system that has a web browser. (Which is why standards across browsers is so important). But, until recently web-apps faced limitations. Not having full access to a device’s hardware and operating system was an issue – but that’s being fixed as more native APIs are being added to modern web browsers.

A disadvantage of having a web-only app was losing out on the discoverability that comes with having it listed in a searchable marketplace. Adding a web-app to your device home screen, from a web browser, is not intuitive to average users. Fortunately, the Google Play Market allows us to upload an app file that links to a progressive web app.

This involves a new protocol, Trusted Web Activities, as “a way to integrate your web-app content such as your PWA with your Android app“. The PWA leverages Digital Asset Links to “declare that it is associated with a specific Android app.

Progressive web apps

I decided to try this out with one of my web-apps, BJJ Tracker. You can read about how I first built it on another blog post.

I had to make sure it qualified as a PWA. It needed offline support, as well as any other features that would make it feel like a native app. Google Chrome’s developer tools has a section called “Audits” that helped me identify such opportunities.

progressive web app audit

The first step was to create a “service worker” JavaScript file, and register it when BJJ Tracker loads.

if('serviceWorker' in navigator) {
  navigator.serviceWorker
           .register('/serviceWorker.js')
           .then(function() { console.log("Service Worker Registered"); })
           .catch(error => {
	        	console.log(error.message)
	    	})
}

I added the above code to a shared file that loads on every page of my app.  Below is an example service worker file. This file downloads any vital assets to a user’s device, and later loads them from the cache. Including a polyfill ensures that the cache methods exist (in case the browser does not support them natively). “We need to use the polyfill because the Cache API is not yet fully supported in all browsers.

importScripts('/cache-polyfill.js');

self.addEventListener('install', function(e) {
 e.waitUntil(
   caches.open('bjjtracker').then(function(cache) {
    return cache.addAll([
       '/',
       '/index',
       '/index?login',
       '/create-record?class',
       '/create-record',
       '/create-record?competition',
       '/view-record',
       '/view-month',
       '/privacy-policy',
       '/contact',
       '/view-more-data',
       '/account',
       '/css/bootstrap.min.css',
       '/css/bootstrap.min.css',
       '/css/bootstrap-theme.min.css',
       '/css/main.css',
       '/simpleMobileMenu/styles/jquery-simple-mobilemenu.css',
       'https://use.fontawesome.com/releases/v5.3.1/css/all.css',
       'https://fonts.googleapis.com/css?family=Roboto|Eczar&display=swap',
       'https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js',
     ]);
    }).catch(error => {
        console.log(error.message)
    })
 );
});

self.addEventListener('fetch', function(event) {
	event.respondWith(
		caches.match(event.request).then(function(response) {
			return response || fetch(event.request);
		}).catch(error => {
	        console.log(error.message)
	    })
	);
});

Read the documentation on Google’s developer portal.

Next, I created a “manifest” file. This file is written in JSON format. It helps describe how the web-app behaves once “installed”. It handles things such as app icon images and meta data.

{
  "name": "BJJ Tracker",
  "lang": "en-US",
  "short_name": "BJJ Tracker",
  "start_url": "/",
  "display": "standalone",
  "background_color": "#2a4d69",
  "theme_color": "#2a4d69",
  "description": "Track Brazilian Jiu Jitsu progress and fitness goals.",
  "icons": [{
    "src": "img/homescreen48.png",
    "sizes": "48x48",
    "type": "image/png"
  }, {
    "src": "img/homescreen72.png",
    "sizes": "72x72",
    "type": "image/png"
  }, {
    "src": "img/homescreen96.png",
    "sizes": "96x96",
    "type": "image/png"
  }, {
    "src": "img/homescreen144.png",
    "sizes": "144x144",
    "type": "image/png"
  }, {
    "src": "img/homescreen168.png",
    "sizes": "168x168",
    "type": "image/png"
  }, {
    "src": "img/homescreen192.png",
    "sizes": "192x192",
    "type": "image/png"
  }, {
    "src": "img/homescreen512.png",
    "sizes": "512x512",
    "type": "image/png"
  }]
}

I created the image assets using open source software.

Image assets created with GIMP
Image assets created with GIMP

The manifest needs to be referenced by the app. I added a link tag to a shared <head> file. Additionally, I included a few other meta tags that let browsers know to treat this website as an app.

<link rel="manifest" href="/manifest.json">
<meta name="theme-color" content="#005b96"/>
<meta name="mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="msapplication-starturl" content="/">

Android Studio

A signed app bundle is generated from Android Studio.  I use a sample project from Google Chrome Labs as a template. We can clone that repository, and update the “/svgomg-twa/app/build.gradle” settings to point to our PWA.

app gradle settings
TWA to wrap SVGOMG in an Android App

The app’s icon files can be generated using an online tool. The downloadable bundle can be dropped into “/svgomg-twa/app/src/main/res/“.

icon generator
https://romannurik.github.io/AndroidAssetStudio/icons-launcher.html

When creating the app bundle (“Build > Generate Signed Bundle/APK”) we’ll need a signing key. I created a new one, and named the file mykeystore.keystore.

key signing on mac

An “assetlinks.json” file needs to be uploaded to the web app’s host to satisfy the Digital Asset Links requirement.  “The Digital Asset Links protocol and API enable an app or website to make public, verifiable statements about other apps or websites.” This confirms ownership of the PWA so that it can be linked to our app in the Play Store. To generate this file, first we’ll need to get the fingerprint from the signing key we used:

keytool -list -v -keystore mykeystore.keystore -alias mykeystore -storepass password-here  -keypass password-here

That command shows us the certificate fingerprints. Copy the SHA256 value. It is used with Google’s Statement List Generator to create the contents of the assetlinks.json file. The statement file is then placed in a “.well-known” directory on the root of our PWA domain (eg. https://www.bjjtracker.com/.well-known/assetlinks.json)

Finally, I visited the Google Play Console. Besides uploading the .apk file, I also needed to include screenshots, featured image files, and complete a content rating survey – amongst other things. Since my app has been approved, you can now find it in the Google Play Market.

BJJ Tracker in the Google Play Store.

This app is a side project I use to toy with new web technologies. I’m trying to drive traffic to it so that I can experiment with optimizing conversions. I’m using it as a trial grounds for another software service called SplitWit. SplitWit is focused on optimizing conversions for the web, and helping digital marketers reach their goals. You can read about it on another post from this blog.

bjj tracker app

Technology and Consciousness

Throughout philosophical discussions concerning technology the concept of “human nature,” and its influence, are often referenced. Upon examining consciousness within a technological context the idea of a loss of humanity or individuality continually arises. Curiously, this seems to imply something very strange – specifically that the major trends of human nature are ultimately leading to its own demise. The abstraction of human characteristics and qualities, such as talent and emotion, which emerges from the influence of human nature on technology, causes a reasonable sense of fear and unnaturalness in most people. It’s clear and obvious that the world is rapidly changing in a way that history has never felt before. Socially, environmentally, and even spiritually, humankind is experiencing a metamorphosis. Issues that have stirred the minds of fanatics and dreamers for centuries are finally coming to a boil. Despite this, we may find consolation in the idea that perhaps human beings are simply a stepping-stone in the bigger picture, and that quality which we refer to as ‘humanity,’ actually derives from the whole of the cosmos and will be survived regardless of the fate of the 46 chromosomed machines that claim it as their exclusive birthright.

The twenty-first century is understood to be a pivotal moment in the history of humankind. Through technology human nature is being altered and we begin to face issues that never before existed. In a talk given by Sir Martin Rees, it is argued that this may be our last century on Earth (1). Discussing the immense future lying ahead of us, Rees explains that complexity and intelligence have far to go even here still on earth, not to mention into the depths of space. A main tenet of his argument entails that for the first time humans are able to materially change themselves, and the planet on a global scale. With the arrival of the internet, complexity and abundance of information has sky-rocketed. Slowly, digital information is becoming more important than material things (consider cash versus electronic banking, etc.). Perhaps this transition is also affecting humanity itself. Our own ability, and desire, to change ourselves may in the end result in the loss of ourselves. Bioengineering and bionics aside, I assert that most vital role will be played by the systems we create in this ‘infinite game.’ Humanity itself is based on information systems, in various regards. Our physical selves result from genetic information. Our minds, our consciousness, are all essentially information processed through a system. Everything that defines humanity seems to be compatible.

Interestingly enough, there is already a website that is devoted to “the putative future process of copying one’s mind from the natural substrate of the brain into an artificial one, manufactured by humans.” “The Mind Uploading Webpage, (2)” also details a list of various issues and questions that seem to arise from the concept, including personal identities, brain enhancements, and artificial realities. What future does a website like this promise in the developing context of web 3.0 and beyond? Imagine once something like this invades our everyday lifestyle – the explicit and intentional outsourcing of the human brains. The reason why I have focused so much on this mental outsourcing and expansion of humanity is because it points to result in something even more complex than the sum of its parts. The various digital systems that humanity has begun to embrace, interconnected within a system itself (which will ultimately be a descendant of today’s internet), will itself eventually develop into a conscious, sentiment being. As described in an article from the New York Times by Jim Holt, something as simple as a rock may “be viewed as an all-purpose information processor.”

I recently enjoyed a cartoon strip by Scott Adams that maintains great relevance to the topic (3). Its title, “Supreme Being: Our Future, Not Our Origin,” accurately describes its argument. Its initial four slides explain how complex things result from the combination of simpler, less capable components. Its sixth slide then says: “What if ‘God,’ is the consciousness that will be created when enough of us are connected by the internet?!!” Interestingly, I feel that this idea generally pointing in the right direction. A ubiquitous, unseen entity that connects everything, huh. Not that I’m trying to bring up a theological argument (perhaps God does exist, and what the cartoon refers to would simply be a manifestation of such), but the idea does seem remarkable.

This post was originally written for my first blog that has since been discontinued.

The Future of Education

flowers

In today’s world it is conventional wisdom that a college education is necessary to excel as a professional. Times are said to have changed, and without proper schooling one is doomed to a life of either hard labor or low-paying pencil pushing. And if you’re planning on paying for an education there is no escaping the fact that college costs are rising. Besides the hefty price tag, traditional schooling is consuming, socially and mentally, forcing a particular lifestyle upon the student. Further, the relationship between the educator and the educated maintains a certain depravity, as a professor holds a figurative gun to the student’s head (any false moves may lead to a career crippling F). But is there an alternative?

In a recent editorial featured in the New York Post (April 23, 2008) Thomas Sowell attributes the high cost of college to two reasons: “People will pay what the colleges charge, and colleges have little incentive to reduce tuition.” He explains that unlike most markets, where lowering prices attracts business, in the academic world the government is ready to step in to pick up the slack. A university would loose millions per year in government money if they lowered tuition. Considering the position that today’s young people are placed, where the arduous task of completing a degree is coupled with unfair prices and a dire necessity, which will affect the rest of their life, it is fair to say that they have us by the proverbial balls.

In an article which I recently compiled I attempt to imagine the direction of coming educational paradigms. It quickly becomes obvious how the talent of great minds may be ignored due to lack of proper credentials. Our current scholastic system bespeaks the Tory elitism representative of Western culture. Perhaps the stereotypical role of an experimental, bohemian college student is effected by the sharp contrast of the academic organization. While it is clear that the classroom is continuing to evolve, it will be necessary for the vintage activist spirit of the student to lend guidance to new educational trends that shifts to a liberal method of intellectual maturation.

So where is the classroom going? I can say with a great deal of confidence that virtual technology will play a leading role in the future of education. Already most colleges and universities offer distance learning programs (online classes). Some colleges, such as the University of Phoenix offer completely virtual degrees. Hybrid courses, in which physical meetings compose only a third of the course time, are also becoming popular. This model moves the educator from the head of the classroom, handing knowledge down, to a guiding medium. This new role forces a teacher to not merely present knowledge, but to be sympathetic in facilitating its acquisition.

Despite the advantages of a virtual classroom, the heavy price still lingers overhead. In overcoming this obstacle towards an open, intellectually progressive society we must embrace the idea of autodidactism.

Being self-educated sounds harder than it is. Some of the most important figures in history have been non-traditionally educated (including Socrates, Benjamin Franklin, Alan Watts, and Mark Twain). It means having a choice in subject matter, moving at your own pace, and it’s free. Its relevance towards the shifting educational paradigm can be attributed to the dawn of the information age, coupled with the open content movement. Considering resources available today, it has never been easier to be self taught. Wikipedia alone serves as an ocean of open knowledge. Various colleges, including MIT, offer ‘open-courseware,’ which include lectures, videos, and notes for entire courses for free. E-books, language courses, podcasts, and dictionaries have all become openly available in a spectrum wide enough to cover anyone’s interests. Even aspiring musicians can learn basics of instruments, theory, and entire songs through online tablatures, sheet music, and video lessons. Rather than growing around current structures, we should move to evolve the system to fit our needs and goals.

Additional Resources:

“I have never let school interfere with my education”
-Mark Twain

“In the first place God made idiots. This was for practice. Then He made school boards.”
-Mark Twain

The PH.D Octopus by William James

“It is interesting to imagine the direction of the classroom and forthcoming educational paradigms. I imagine that within the next twenty years the physical classroom will become obsolete, only to be replaced by an autodidactic virtual environment. The role of the teacher will shift from dictating at the head of the class, to more of a supervising librarian that directs the flow of the program in a very yang manner.”

Open Source Society

paintings

Sometime during the 1950’s television sets had begun to become widely available and fairly affordable. Noting this point as the advent of entertainment focused telecommunications; art had begun to be understood as a trivial distraction to conservative intellectuals. The vegetating trance of the television, typically allowing the mind to enter a state of ‘cruise control,’ could be attributed to the public’s low level of input towards programming. Although the Internet seems to descend from this legacy of infotainment, something quite different is going on. While television preaches endless forms of false happiness through consumerism, the design of the contemporary web aims to facilitate “creativity, information sharing, and, most notably, collaboration among users (Wikipedia – Web 2.0)” This arrangement comprises the core methodology of the Open Source Revolution that is beginning to reshape our traditions and lead us towards a new renaissance of gnosis.

Philosopher Terence McKenna, reminding us that “culture is not your friend (1999),” advises to resist the epistemological disease of autocratic content by creating our own art. Through venues such as Youtube and MySpace, music, art, film, and photography, even mildly entertaining, is now able to draw large audiences and develop into a well received meme. Further, the Wiki archetype has effectively turned the amateur into the expert. McKenna often refers to a revival of the archaic, which is set to take place as a reaction to the patriarchal model of the elite handing knowledge down, and forsaking personal revelation. The proliferation of open source programs, granting users the ability to freely edit and redistribute computer software, manifests the artistic position towards which society must move. The world is shrinking in to a global village thru mass media, and the common majority must take direct control in order to reconnect and reconcile into cosmic consciousness.

The term ‘global village’ is often used metaphorically to describe the internet and World Wide Web. Philosopher Marshall McLuhan predicted that a global village would be tribal in character. The open source operating system Ubuntu, a distribution of Linux (a prime example of free software and open source development), derives its name from the South African philosophical notion of humanitarianism. Interestingly, this juxtaposition of concept and utility represents the new archetype of culture towards which the Open Source Revolution is driving. Projects such as Wikipedia, an open content encyclopedia, are able to maintain their integrity, accuracy, and scope through an effort of community and collaboration. The Open Source Revolution lends new drive to innovation, epistemology, peer support, and ultimately an altruism that trumps the capitalist agenda of elitism. I feel that we must adopt the concept of Open Source as a new organizational model for society.

Should the open source paradigm stay confined to computer software and the internet, or should we move to adopt it as a new model for social organization?

“In a global village where we have instant access to innumerable beliefs around the world, we have come to realize the relativity of what we think”

-Walter Anderson