Programming

Writing a vibe.d app

In this tutorial we’ll write a vibe.d web app that links bugzilla issues to github pull requests.

Let’s start by creating a vibe.d project,

dub init dlang-bot --type=vibe.d
git init dlang-bot

which generates a few files and folders.

dub.sdl      # the dub package file
source/app.d # a hello world vibe.d app
public       # folder for assets
views        # folder for frontend templates

Let’s edit app.d and use vibe.d’s URLRouter to add an endpoint for github webhooks, so that the app can receive notifications about new or updated pull requests.

auto router = new URLRouter;
router
    .get("/", (req, res) => res.redirect("https://github.com/dlang-bot?tab=activity"))
    .post("/github_hook", &githubHook)
    ;
listenHTTP(settings, router);

void githubHook(HTTPServerRequest req, HTTPServerResponse res)
{
    if (req.headers["X-Github-Event"] == "ping")
        return res.writeBody("pong");
    assert(req.headers["X-GitHub-Event"] == "pull_request");

    // vibe.d parses application/json post bodies by default
    auto action = req.json["action"].get!string;
    logDebug("#%s %s", req.json["number"], action);
    switch (action)
    {
    case "opened", "closed", "synchronize":
        auto commitsURL = req.json["pull_request"]["commits_url"].get!string;
        auto commentsURL = req.json["pull_request"]["comments_url"].get!string;
        runTask(toDelegate(&handlePR), action, commitsURL, commentsURL);
        return res.writeBody("handled");
    default:
        return res.writeBody("ignored");
    }
}

The hook can handle ping and pull_request events and will process interesting pull request actions asynchronously in a separate task.

void handlePR(string action, string commitsURL, string commentsURL)
{
    auto comment = getBotComment(commentsURL);
    auto refs = getIssueRefs(commitsURL);
    logDebug("%s", refs);
    if (refs.empty)
    {
        if (comment.url.length) // delete any existing comment
            deleteBotComment(comment.url);
        return;
    }
    auto descs = getDescriptions(refs);
    logDebug("%s", descs);
    assert(refs.map!(r => r.id).equal(descs.map!(d => d.id)));

    auto msg = formatComment(refs, descs);
    logDebug("%s", msg);

    if (msg != comment.body_)
        updateBotComment(commentsURL, comment.url, msg);
}

The processing is fairly straightforward. First we get a list of issues references from any of the PR’s commit messages, then ask Bugzilla for a short description of those issues, format the information, and finally create, update, or delete a comment on the PR.

struct IssueRef { int id; bool fixed; }
// get all issues mentioned in a commit
IssueRef[] getIssueRefs(string commitsURL)
{
    // see https://github.com/github/github-services/blob/2e886f407696261bd5adfc99b16d36d5e7b50241/lib/services/bugzilla.rb#L155
    enum issueRE = ctRegex!(`((close|fix|address)e?(s|d)? )?(ticket|bug|tracker item|issue)s?:? *([\d ,\+&#and]+)`, "i");
    static auto matchToRefs(M)(M m)
    {
        auto closed = !m.captures[1].empty;
        return m.captures[5].splitter(ctRegex!`[^\d]+`)
            .map!(id => IssueRef(id.to!int, closed));
    }

    return requestHTTP(commitsURL, (scope req) { req.headers["Authorization"] = githubAuth; })
        .readJson[]
        .map!(c => c["commit"]["message"].get!string.matchAll(issueRE).map!matchToRefs.joiner)
        .joiner
        .array
        .sort!((a, b) => a.id < b.id)
        .release;
}

This code heavily uses std.algorithm and std.range for pipeline style operations. It’s pretty terse but much more robust (and simple) than writing explicit nested loops. The pipeline fetches all commits, matches all referenced issues in commit messages, converts the matches, joins all references, joins all references of all commits, and sorts them by issue id.

struct Issue { int id; string desc; }
// get pairs of (issue number, short descriptions) from bugzilla
Issue[] getDescriptions(R)(R issueRefs)
{
    import std.csv;

    return "https://issues.dlang.org/buglist.cgi?bug_id=%(%d,%)&ctype=csv&columnlist=short_desc"
        .format(issueRefs.map!(r => r.id))
        .requestHTTP
        .bodyReader.readAllUTF8
        .csvReader!Issue(null)
        .array
        .sort!((a, b) => a.id < b.id)
        .release;
}

The code to query Bugzilla is fairly similar but uses csv instead of json.

struct Comment { string url, body_; }

Comment getBotComment(string commentsURL)
{
    auto res = requestHTTP(commentsURL, (scope req) { req.headers["Authorization"] = githubAuth; })
        .readJson[]
        .find!(c => c["user"]["login"] == "dlang-bot");
    if (res.length)
        return deserializeJson!Comment(res[0]);
    return Comment();
}

I’ll spare the formatting code, you can find it in the full source code.

Getting an existing comment simply searches for the first comment posted by the dedicated dlang-bot user and uses deserializeJson to convert json to a struct.

void sendRequest(T...)(HTTPMethod method, string url, T arg)
    if (T.length <= 1)
{
    requestHTTP(url, (scope req) {
        req.headers["Authorization"] = githubAuth;
        req.method = method;
        static if (T.length)
            req.writeJsonBody(arg);
    }, (scope res) {
        if (res.statusCode / 100 == 2)
            logInfo("%s %s, %s\n", method, url, res.bodyReader.empty ?
                    res.statusPhrase : res.readJson["html_url"].get!string);
        else
            logWarn("%s %s failed;  %s %s.\n%s", method, url,
                res.statusPhrase, res.statusCode, res.bodyReader.readAllUTF8);
    });
}

void deleteBotComment(string commentURL)
{
    sendRequest(HTTPMethod.DELETE, commentURL);
}

void updateBotComment(string commentsURL, string commentURL, string msg)
{
    if (commentURL.length)
        sendRequest(HTTPMethod.PATCH, commentURL, ["body" : msg]);
    else
        sendRequest(HTTPMethod.POST, commentsURL, ["body" : msg]);
}

And eventually we need some code to create, update, and delete comments. Notice that requestHTTP is synchronous and uses callbacks only to prevent you from escaping the request/response object.

Now that we’ve implemented our app and tested it locally (using ngrok for example) we can deploy it on Heroku. Though first we have to bind to an external network interface and make the listening port configurable.

auto settings = new HTTPServerSettings;
settings.port = 8080;
settings.bindAddresses = ["0.0.0.0"];
readOption("port|p", &settings.port, "Sets the port used for serving.");

Unfortunately openssl on heroku is configured to use custom config and certificate paths, so we have to explicitly specify Ubuntu’s default CA bundle.

// workaround for openssl.conf on Heroku
HTTPClient.setTLSSetupCallback((ctx) {
    ctx.useTrustedCertificateFile("/etc/ssl/certs/ca-certificates.crt");
});

After that we use the Heroku CLI to create a new app, push+deploy our code, set our github token, and start a single dyno to serve the app.

heroku create dlang-bot --buildpack http://github.com/MartinNowak/heroku-buildpack-d.git
git push heroku master
heroku config:set GH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
heroku ps:scale web=1

https://dlang-bot.herokuapp.com/

https://github.com/MartinNowak/dlang-bot

Programming

Other articles

Reducing vibe.d turnaround time (Part 2 Less Compiling)

Reducing vibe.d turnaround time (Part 1 Faster Linking)

Other articles

blogroll

social