Blog

I recently rebuilt jwage.com using Sculpin and GitHub Pages. It was previously hosted on Tumblr and over time I grew frustrated with the lack of control. I wanted a mobile friendly design and I wasn't looking forward to working with HTML & CSS in the Tumblr browser user interface so I decided to give Sculpin a try. I was immediately happy with the choice because it is built using familiar technologies like Symfony and Twig.

Here are a few handy utilities I hacked together to help with my development.

build.sh

For easily building the site in the way that GitHub pages expects:

./build.sh dev

Or to build it for prod:

./build.sh prod

Here is the source:

#!/bin/bash

vendor/bin/sculpin generate --env=$1
if [ $? -ne 0 ]; then echo "Could not generate the site"; exit 1; fi

mv docs/CNAME output_$1/CNAME
rm -rf docs
mv output_$1 docs

publish.sh

To make it easy to publish and deploy a new version of the site:

#!/bin/bash

./build.sh prod

git add --all .
git commit -m "New version of jwage.com"
git push origin master

# Put the dev version back after deploying prod
./build.sh dev

watch.php

For watching changes to your source and automatically triggering build.sh:

<?php

$buildScriptPath = __DIR__.'/build.sh dev';

$startPaths = [
    __DIR__.'/app/config/*',
    __DIR__.'/source/*',
];

$lastTime = time();

while (true) {
    $files = recursiveGlob($startPaths);

    foreach ($files as $file) {
        $time = filemtime($file);

        if ($time > $lastTime) {
            $lastTime = time();

            echo sprintf("%s was changed. Building...\n", $file);

            echo shell_exec($buildScriptPath)."\n";

            file_put_contents(__DIR__.'/docs/changed', time());
        }
    }
}

function recursiveGlob(array $paths)
{
    $files = [];

    foreach ($paths as $path) {
        $files = array_merge($files, glob($path));

        foreach ($files as $file) {
            if (is_dir($file)) {
                $dirPath = $file.'/*';

                $dirFiles = recursiveGlob([$dirPath]);

                $files = array_merge($files, $dirFiles);
            }
        }
    }

    return $files;
}

Run this script in the root of your project:

php watch.php

Then save a file in your project and it should rebuild automatically:

/data/jwage.com/source/_posts/2018-03-25-sculpin-watch-rebuild.md was changed. Building...
Detected new or updated files
Generating: 100% (249 sources / 0.01 seconds)
Converting: 100% (272 sources / 0.14 seconds)
Formatting: 100% (272 sources / 0.03 seconds)
Processing completed in 0.24 seconds

Browser Refresh

Everytime watch.php detects a change, it writes an updated timestamp to a file named changed with the current timestamp. Now you can poll this file for changes in your site to automatically refresh the browser. Put this in your layout.

{% if site.env == 'dev' %}
    <script type="text/javascript">
        var changedUrl = '{{ site.url }}/changed';
    </script>
    <script src="{{ site.url }}/js/watch.js"></script>
{% endif %}

Then in source/js/watch.js place the following code:

function setCookie(cname, cvalue, exdays) {
    var d = new Date();
    d.setTime(d.getTime() + (exdays*24*60*60*1000));
    var expires = "expires="+ d.toUTCString();
    document.cookie = cname + "=" + cvalue + ";" + expires + ";path=/";
}

function getCookie(cname) {
    var name = cname + "=";
    var decodedCookie = decodeURIComponent(document.cookie);
    var ca = decodedCookie.split(';');
    for(var i = 0; i <ca.length; i++) {
        var c = ca[i];
        while (c.charAt(0) == ' ') {
            c = c.substring(1);
        }
        if (c.indexOf(name) == 0) {
            return c.substring(name.length, c.length);
        }
    }
    return "";
}

var lastTimestamp = getCookie('lastTimestamp');

var watchCallback = function(timestamp) {
    if (lastTimestamp != timestamp) {
        lastTimestamp = timestamp;

        setCookie('lastTimestamp', lastTimestamp, 1);

        window.location.reload();
    }
};

setInterval(function() {
    $.get(changedUrl, watchCallback);
}, 500);

Maybe eventually I will put this together in to proper code and packaging so that other people can use it more easily. For now, feel free to copy and paste this in to your project.

Recently at OpenSky we migrated our e-mail service provider to Oracle Responsys and since then we’ve had issues with Gmail trying to translate our e-mails from random languages. Over the last month it has tried to translate from the following languages:

• Finnish
• Afrikaans
• Catalan
• German
• Hungarian
• Italian
• Latvian
• Norwegian
• Polish
• Slovak
• Vietnamese

We have tried to find someone at gmail support to help but we haven’t had any success. If we did have something in our HTML or e-mail service provider that was making gmail think our emails are in another language, I would think it would be the same language consistently. I’ve seen gmail try to translate the same e-mail to multiple different languages. This feels like a bug in gmail causing false positives.

Here is another Responsys customer that is experiencing the same thing here.

Here is a post on the gmail help forum about the issue

Here is a screenshot of an e-mail coming from one gmail user going to another gmail user. This rules out the problem being anything at all related to our e-mail service provider. I am confident that Gmail language detection is broken:

Here are some screenshots:

Finnish

Afrikaans

Catalan

German

Hungarian

Italian

Latvian

Norwegian

Polish

Slovak

Vietnamese

Deciding when to inject the container in Symfony is a frequent topic of discussion. Many would have you believe that you should NEVER inject the container because it breaks the “rules” and is an anti-pattern. This is not always true and, just like most things, it should not be applied blindly to everything you do. This post aims to demonstrate cases where injecting the container makes sense.

The problem with unused dependencies

Imagine you have a listener that records some query string parameters and you’d like this to run on each request:

<?php

use Doctrine\DBAL\Connection;
use Symfony\Component\HttpKernel\Event\GetResponseEvent;

class RequestParameterLoggerListener
{
    private $requestParameterLogger;

    public function __construct(RequestParameterLogger $requestParameterLogger)
    {
        $this->requestParameterLogger = $requestParameterLogger;
    }

    public function onKernelRequest(GetResponseEvent $event)
    {
        $request = $event->getRequest();

        if (!$request->query->has('utm_origin')) {
            return;
        }

        $this->requestParameterLogger->logUtmOrigin($request);
    }
}

class RequestParameterLogger
{
    private $connection;

    public function __construct(Connection $connection)
    {
        $this->connection = $connection;
    }

    public function logUtmOrigin(Request $request)
    {
        // record utm origin to the database using the connection
    }
}

As you can see, logUtmOrigin is only called when the Request’s query string contains the parameter named utm_origin. This means that even on requests where utm_origin does not exist, we are constructing the RequestParameterLogger and injecting it to RequestParameterLoggerListener.

This is a simple example; however, in a large application with many such listeners, you can imagine the application constructing potentially dozens of services like RequestParameterLogger, which would be injected but never used.

Injecting the Container

This problem can be easily fixed by injecting the container and lazily requesting the service from the container when it is needed. Here is the same listener above, but rewritten with container injection.

<?php

use Symfony\Component\DependencyInjection\ContainerAware;
use Symfony\Component\HttpKernel\Event\GetResponseEvent;

class RequestParameterLoggerListener extends ContainerAware
{
    public function onKernelRequest(GetResponseEvent $event)
    {
        $request = $event->getRequest();

        if (!$request->query->has('utm_origin')) {
            return;
        }

        $this->container->get('request.parameter_logger')->logUtmOrigin($request);
    }
}

Now, when the query parameter utm_origin does not exist, the RequestParameterLogger will not be constructed.

In the Wild

At OpenSky we have always injected services to listeners, security voters, security providers, etc. As a result, every request would eagerly construct many services that would never be used. By rewriting these services to use the strategy demonstrated above, I was able to shave 10-20 milliseconds off of every request and significantly reduce the number of services constructed to handle a request that serves a blank controller and template.

To make it easy to rewrite all of our prior art, I wrote a simple class named LazyService.

<?php

use Symfony\Component\DependencyInjection\ContainerAware;

abstract class LazyService extends ContainerAware
{
    protected $propertyMap = array();
    protected $values = array();

    public function __get($key)
    {
        if (!isset($this->propertyMap[$key])) {
            throw new \InvalidArgumentException(sprintf('Could not find service for key %s', $key));
        }

        if (!isset($this->values[$key])) {
            if ($this->propertyMap[$key][0] === '%') {
                $this->values[$key] = $this->container->getParameter(trim($this->propertyMap[$key], '%'));
            } else {
                $this->values[$key] = $this->container->get($this->propertyMap[$key]);
            }
        }

        return $this->values[$key];
    }
}

Here is our original example, but rewritten to use this LazyService class:

class RequestParameterLoggerListener extends LazyService
{
    protected $propertyMap = array(
        'requestParameterLogger' => 'request.parameter_logger',
    );

    public function onKernelRequest(GetResponseEvent $event)
    {
        $request = $event->getRequest();

        if (!$request->query->has('utm_origin')) {
            return;
        }

        $this->requestParameterLogger->logUtmOrigin($request);
    }
}

This made it easy for me to make dozens of services extra lazy without having to rewrite too much of the service themselves or the associated tests.

Why not use lazy services provided by Symfony?

I chose not to use lazy services provided by Symfony because I didn’t want to add yet more complexity and weight to our application. Even if you mark a service as lazy, a proxy still has to be instantiated and injected. I wanted to completely eliminate the construction of these classes.

That is it! I hope this post was helpful in realizing when to inject the container and not blindly follow design theory. Happy Thanksgiving!