I have a lot of scripts that automate the boring parts of my life. Recently
an article was posted about one system administrator who automated everything
down to [scripting the coffee maker to produce cups of coffee for him](https://www.jitbit.com/alexblog/249-now-thats-what-i-call-a-hacker/).
I don't go quite that far with my automation, but it inspired me to write about a
lesser known tool in OSX - `launchd` and its interfacing agent: `launchctl`.

You may be familiar with `launchctl` if you've ever run a database or cache server
on a Mac locally. Or, you may know it indirectly through the use of a wrapper
called [lunchy](https://github.com/eddiezane/lunchy), which makes working with
`launchctl` a much more sane experience.

At a high level, `launchctl` aims to make it easy to run daemonized processes
on Mac. It can manage these processes for a user or for the system as a whole.
Many of the services on Mac that you interface with every day are managed by
`launchd` such as Spotlight and AirPlay.

To explore the world of `launchd` and `launchctl`, I'm going to examine a specific
use case I had for using the service: updating [brew](http://brew.sh/). I have
a bad habit of forgetting to update my brew formulas for weeks (sometimes months)
at a time. The reminder usually comes in the form of an error when trying to
install a formula that mysteriously doesn't exist. I thought: "This is a perfect
use case for cron!" and I was probably right except for the minor (read: show-stopping)
issue of being on a laptop.

I needed a way to be able to update my brew formulas automatically in the background
on a rough schedule. If my laptop was off when the update was scheduled to run,
I would expect it to run the next time I turned on my laptop. On a more fine level,
I would like to retry the update if it failed and I would like the output to be
logged to a file in order to check on the status of previous runs. Enter `launchd`.

Every `launchd` service is configured using a plist file. Multiple
options can be specified in the plist file. The documentation for those options
can be found in the man pages (gasp!) under the `launchd.plist` entry. These configuration
files usually live in `~/Library/LaunchAgents`. If your user has no custom services
yet running via `launchd`, you will need to create that folder. A benefit of this
is that files for all of the daemons live in one location, which makes it easy to
organize and keep track of the different services.

Generally, the configuration files are named using a combination of your domain and
the name of the service. I decided to be creative and call this service brewd.
Below is the plist file that satisfied my requirements located at
`~/Library/LaunchAgents/me.jcomo.brewd.plist`.

``` xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>me.jcomo.brewd</string>
    <key>KeepAlive</key>
    <dict>
      <key>SuccessfulExit</key>
      <false />
    </dict>
    <key>ThrottleInterval</key>
    <integer>60</integer>
    <key>StartCalendarInterval</key>
    <dict>
      <key>Minute</key>
      <integer>0</integer>
      <key>Hour</key>
      <integer>10</integer>
      <key>Weekday</key>
      <integer>2</integer>
    </dict>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/bin/brew</string>
      <string>update</string>
    </array>
    <key>StandardOutPath</key>
    <string>/usr/local/var/brewd/output.log</string>
    <key>StandardErrorPath</key>
    <string>/usr/local/var/brewd/error.log</string>
  </dict>
</plist>
```

I'm not going to go into too much depth on each of the options since the descriptions
and parameters can all be found in the man page. However, the crucial part in
this file, and the point of this post, is under the key `StartCalendarInterval`.
Here, you can define a cron-like schedule to run the service. According to the docs,

> Unlike cron which skips job invocations when the computer is asleep, launchd will
> start the job the next time the computer wakes up.  If multiple intervals transpire
> before the computer is woken, those events will be coalesced into one event upon wake
> from sleep.

Bingo! The other options in the configuration specify where to direct stderr and
stdout, how often to retry if the program fails, and finally, the actual command
that should be scheduled and run.

The last step is to make `launchd` aware of this configuration. To do that, `launchctl`
must be used to load the plist. This can be done by running
`launchctl load ~/Library/LaunchAgents/me.jcomo.brewd.plist`. If I no longer
want a service to be running, I can simply unload it.

Now my brew formulas will be updated approximately every Monday morning at 10am.
The emphasis is on approximately; this would be a bad replacement for cron for
a task that needed to run on a strict schedule.

There is a lot more depth to `launchd` and it fits more use cases than the one
presented in this post. For more information, I'd encourage exploring the man
pages for both `launchd.plist` and `launchctl` as it was hard to find specific
help online.

Disclaimer: there are likely other solutions to this problem. While `launchd`
and `launchctl` certainly have their quirks, I find them to be a great solution
for running daemon processes on my laptop.


Using launchctl as an anacron replacement

I am currently maintaining a service at my day job that takes arbitrary events
from mobile games and subsequently validates and processes them so that the
customers who are sending this data to us can then view it with pretty graphs
and other metrics via our dashboard. Our customers can then use this data to
optimize their advertising campaigns and target certain publisher or user groups
that they believe gives them a better return on investment.

Right now, the service is written in Python (using [Flask](http://flask.pocoo.org/))
and has been functioning just fine, but we are rapidly expanding this service
to be able to handle more and more events. Some of those events can require
heavier processing that blocks the request thread, which is not good. I have
been experimenting with Scala and [Finagle](https://twitter.github.io/finagle/)
(a high concurrency RPC framework) as we already use this stack for some of our
more heavily trafficked services.

As I was exploring Finagle, I was running little toy services to make sure that
it would be relatively easy to port everything over if we decided to go down
that route. One of the events that we handle is receipt validation for in app
purchases. For receipts from certain app stores, we need to make an RPC call
to their servers where the response will have the result of the validation.

While trying to implement a basic HTTP service in Finagle, I stumbled into some
difficulty when sending requests over HTTPS. Although Finagle is pretty well
documented in general, I had trouble finding the answer to this issue. Hopefully,
the following explanation will help others get unstuck with this issue in the
future.

## Validating Amazon Receipts

For this example, we're going to take the scenario of validating receipts with
Amazon's [receipt validation service](https://developer.amazon.com/public/apis/earn/in-app-purchasing/docs-v2/verifying-receipts-in-iap-2.0).
All of the specs and seemingly random strings shown in the code examples come
straight from their documentation.

Our API for this service should allow a user to wrap an Amazon receipt inside of
a request and feed it to our client. The service should look something like this.

``` scala
class AmazonReceiptValidationClient(dest: String) extends Service[AmazonReceiptRequest, Response] {
  private val service = Http.newService(dest)

  override def apply(req: AmazonReceiptRequest): Future[Response] = {
    service(req) map { resp =>
      if (resp.getStatus == HttpResponseStatus.OK) resp
      else throw new Exception("Could not validate Amazon receipt")
    }
  }
}

val client = new AmazonReceiptValidationClient("appstore-sdk.amazon.com:443")
```

This is where I started out and originally ran into some trouble. When calling
the service, I would get an error that the connection was reset by peer. In other
words, Amazon had had enough of my HTTP-with-no-TLS BS and refused to
service my request.

The correct implementation for this is actually quite straightforward after
seeing it, but the documentation is not very clear. The private definition
for `service` would change to the following

``` scala
private val service = Http.client.withTls(hostname).newService(s"$hostname:443")
```

And would then be instantiated as

``` scala
val client = new AmazonReceiptValidationClient("appstore-sdk.amazon.com")
```

The `withTls` method returns a client (or service factory, as Finagle calls them)
that will use TLS with requests. You can then create a service using this new
client.

For the sake of completeness, I've included the definition of `AmazonReceipt`
and `AmazonReceiptRequest` below.

``` scala
case class AmazonReceipt(id: String, developerSecret: String, userId: String)

case class AmazonReceiptRequest(receipt: AmazonReceipt) extends RequestProxy {
  override lazy val request = Request(new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, servicePath))

  def servicePath =
    s"/version/1.0/verifyReceiptId/developer/${receipt.developerSecret}/user/${receipt.userId}/receiptId/${receipt.id}"
}
```


Making HTTPS requests with Finagle

I have found python to be one of the easiest languages to pick up, and I think there are many people out there who could corroborate that. That alone is a big reason that it is used for teaching. However, under the hood there are a lot of features in the language where the implementation details can be downright scary to the newcomer and even to the moderately experienced pythonista: metaclasses, dunder methods (eg. `__init__`), and perhaps even decorators depending on the complexity of the function wrapper.

At the moment I put myself into that moderately experienced bucket. I'm pretty comfortable overall but I haven't had a chance to explore the more advanced areas of the language. Metaclasses are the first topic I wanted to dive into.

## Meta what?
By definition, metaclasses are classes that describe and instantiate other classes. Cue the head exploding gif. It sounds pretty convoluted but in reality it's not. In python everything is an object. When you write a class definition, you are essentially defining the properties that each instance of the class will have. Similarly, with metaclasses you are defining the properties that each *class* will have.

## Metaclass uses
Metaclasses are commonly used as class factories and as ways to add functionality to a class itself. Frameworks make use of this (eg. Django) and some libraries also use them. One example of a library is Mongokit.

In Django, metaclasses are used to add information to the user created classes such as permissions and the corresponding database table name. Mongokit uses metaclasses to allow for validation on custom types.

## How do they work?
The definition of a class will typically be something like this:

``` python
class Container(object):
    name = 'default'
```

where `object` is the superclass. We can create basic objects using the `object()` callable, so can we do something to dynamically create basic classes? You bet. This is where the `type` function comes into play. It either takes one argument or three arguments, but we care about the three argument version (docs [here](https://docs.python.org/2/library/functions.html#type)).

We can create the same Container class dynamically with a call to the `type` builtin.

``` python
Container = type('Container', (object,), dict(name='default'))
```

The first argument is the name of the class as a string. The second is a tuple containing all base classes (evaluated left to right). Finally, the third argument is dictionary of the properties to set on the class. In this example, the sole item in the dictionary is a static string but by passing in functions, you can provide methods to each instance of the class that will be created. After all, calling `obj.method()` is nearly equivalent to `obj['method']()` because of the way python implements it objects, so passing a dictionary makes sense.

Since everything is an object, by subclassing `type` we've created a metaclass. To use a metaclass, you can either specify it using the `__metaclass__` property of a class

``` python
class Container(object):
    __metaclass__ = ContainerMeta

    name = 'default'
```

*or* you can use it to create classes dynamically (and override the initializers in the metaclass in the same way you would for an object).

``` python
Container = ContainerMeta('Container', (object,), dict(name='default'))
```

## Creating an immutable container library
The easiest way I learn is by example, and with something like metaclasses, it really makes sense to write code that uses them. I wasn't getting much just by reading about them.

I settled on writing an immutable container library using metaclasses. I will walk through a watered-down version to demonstrate the use of metaclasses but the full library can be found on [Github](https://github.com/jcomo/immut).

We're going for a simple API that looks like this `ImmutableContainer(container_name, *attributes)` where `container_name` is a string that would be the class name and each `attribute` is a string that will end up being a read-only property. The function call will return a **class** where the specified attributes can be instantiated but never modified.

``` python
RequestModel = ImmutableContainer('RequestModel', 'version', 'user')

u = User('Jonathan')
request = RequestModel(user=u, version=2)

request.version  # returns 2
request.version = 3  # we want this to raise an AttributeError
```

This looks like a job for metaclasses! We can define an `ImmutableContainerType` that will create container classes and replace the `setattr` method with one that will raise exceptions for the attributes that should be immutable.

``` python
class ImmutableContainerType(type):
    def __init__(cls, name, bases, dct):
        super(ImmutableContainerType, cls).__init__(name, bases, dct)
        attributes = dct.get('attributes', [])
        cls.__setattr__ = cls.__class__.get_setattr(attributes)

    @classmethod
    def get_setattr(mcs, attributes):
        def setter(self, name, value):
            if name in attributes:
                raise AttributeError("Property {} is immutable".format(name))
            super(self.__class__, self).__setattr__(name, value)
        return setter
```

Since this metaclass subclasses `type` we can create an instance of it (a class) and the `__init__` method will be called.

``` python
RequestModel = ImmutableContainerType('RequestModel', (object,),
                                       dict(attributes=['version', 'user']))
```

Well, that works. But it doesn't quite fit the API we're looking for. A little wrapper will fix that.

``` python
def make_container(container_name, *attributes):
    return ImmutableContainerType(container_name, (object,),
                                  dict(attributes=attributes))

ImmutableContainer = make_container
```

Much better.


## Concluding thoughts
Could I have made the immutable container library without using metaclasses? Absolutely. The API may have ended up looking a bit different but it could definitely be done without using metaclasses.

For most things, using metaclasses will be overkill. While they are extremely powerful, they can make the code harder to reason about. I've found that the best place to use them are in frameworks but they also tend to pop up in some libraries.


Immutable Containers Using Python Metaclasses

EDIT: This may still work with Sublime Text, but I highly recommend using a capable IDE to write Scala code. [IntelliJ](https://www.jetbrains.com/idea/) is my personal choice.

This post details steps to writing and compiling Scala code from within Sublime Text 2. It includes a walkthrough on how to configure the build system to support Scala and also provides some additional information about configuring keybindings to further ease the workflow.

By the end, you should be able to build Scala code using `⌘B` for "build" - the same way other scripts can be run from within the editor.

### Scala Build Support

Here is how you would run Scala code and see the output without having to leave the editor. We create a new build system to specify Scala configuration.

1. Navigate to Tools &rarr; Build System &rarr; New Build System...
1. In the editor, enter the following and save it as `Scala.sublime-build`

``` javascript
{
  "cmd": ["scala", "$file"],
  "path": "$PATH:/usr/local/bin",
  "file_regex": "^(.+):(\\d+): .+: (.+)",
  "selector": "source.scala"
}
```

`cmd` is specified as an array representing the command you would type into a shell. In this case, it would be something like `scala path/to/file`. The `$file` variable takes care of expanding the current file's path.

`path` specifies the path of the Scala installation. Mine is at `/usr/local/bin`. Note that is concatentated with the path instead of overwriting it (prepending or appending to the existing path is use-case specific).

`selector` tells the editor which files should use this configuration. *Note that this option only works if your build system is set to Automatic*.

`file_regex` allows you to capture messages from stdout of the Scala executable and display them within the editor. This particular regex will recognize compilation errors, allowing you to cycle through them easily.

You can read more about the build options in the Sublime docs [here](http://docs.sublimetext.info/en/latest/reference/build_systems.html).

### Keybindings

Optionally, create keybindings for cycling through messages from the plugin when a results window is displayed in the editor.

1. Navigate to Sublime Text 2 &rarr; Preferences &rarr; Key Bindings - User
2. Enter these keybindings in. You can specify any keys here; these are the ones I use.

``` javascript
[
  ...
  { "keys": ["ctrl+r"], "command": "next_result" },
  { "keys": ["shift+ctrl+r"], "command": "prev_result" },
  ...
]
```

### Hello, world!

Let's create a new file and write a simple hello world program to test out our new simple IDE.

``` scala
object TestBuild {
  def main(args: Array[String]) {
    println("Hello, world!")
  }
}
```

Run this with `⌘B` and you should see the output. Now how about errors? Modify the code slightly to include a compilation error and then build again to see errors in the console. `ctrl+R` and `shift+ctrl+R` (the keybindings configured in the previous section) can be used to cycle through errors.


November 28, 2015

May 10, 2015

January 3, 2015

June 21, 2014