YOW! West wrap-up

I had a great time at YOW! West over the last few days – highlights for me were the talks from Jeff Patton, Dave Thomas and the unsung Stew Gleadow.

Slides for my talk are available for download here. I’ve also made the source code for the demo ‘Wishlist’ app available on bitbucket.

During Dave’s talk, I made a list of as many controversial one-liners he dropped as I could. Here they all are, without further comment:
“CPU architects have spent decades trying to make crappy serial code execute in parallel”
“KLOCs kill”
“Dependencies strangle”
“Abstractions bloat”
“A framework is a component on the way to being built that never got finished”
“Refactoring is a joke”
“Computer Science courses teach bad Java and UML diagrams”
“Don’t use text formats like CSV, XML, JSON”
“Batch programming is better”
“Code-Deploy-Monitor (Testing considered harmful)”
“Exception handling should be removed from all programming languages”
“Web services = marketechture”
“Hell of software is dependencies”
“All code will be written in an improved version of SQL by 2020”
“IDEs are needles for addicts”

Replacing the PC

Dustin Curtis came out recently with this piece, subsequently picked up by Marco Arment. The gist of it is that the tablet market will eventually be replaced by smartphones, backed up by demographic figures from Pew Research:

Younger people, on the contrary, simply do their computing entirely on their mobile phones. That’s what they’ve grown up using. Older adults use laptops and tablets to access the internet. Teens and young adults use their phones.

I have to disagree with this statement. It may be true for tweens & teens, but in my experience, primary & early childhood kids pretty much exclusively use iPads (NB: not tablets, iPads specifically). If it’s true that each generation will attach themselves to the specific form factor they first used, the smartphone’s day in the sun is going to be brief.

However, things aren’t so simplistic, and I have some problems with the entire hypothesis. In particular, we need to consider what it is that differentiates a smartphone & a tablet, especially as the hardware is almost identical nowadays. Is it screen size? Not exactly, we already have enough problems drawing an invisible line between the categories that we’ve invented an entirely new one. It can’t be cellular connectivity. Could it be the label applied by the manufacturer? If Apple’s 4.7″ iPhone is branded the ‘iPad Nano’, does that change the category?

No, in my mind, the key defining attribute that makes a smartphone a smartphone, is whether or not it’s capable of traditional (carrier) voice calls*. By that definition, I’m quite confident in predicting that the smartphone market will be completely replaced by ‘tablets’ (of various sizes) within the next 5-10 years.

 

* Thought exercise: if Apple announced a 4G iPod touch (with no Phone app installed), would it be categorised as a smartphone?

 Yes, I’m aware we’re likely to end up calling some of these devices ‘phones’ anyway.

Update – I asked the expert himself on Twitter and got smacked down:

YOW! West 2014

Just an update to let everyone know I’ll be presenting at the YOW! West conference that’s on 13th – 14th of May. I’ve wanted to attend the conference on the east coast for a number of years, but haven’t made it to one yet – hopefully our WA flavoured version will be as good (or better).

My presentation is entitled “Beyond SQLite & Core Data – NoSQL for Mobile” – it will focus on iOS implementations, but the concepts are just as applicable for Android & Windows Phone devs. I’m working on some interesting demos, so I’m hoping people will enjoy it.

Bird Nerd 1.0

So, I built a game.

Bird Nerd IconFollowing the yanking of Flappy Bird from the App Store (and the subsequent proliferation of indie apps), a colleague said: “We should build an app called ‘Flappy Word’, where instead of flying through pipes, you collect letters and make words”. This sounded like an absolute winner of an idea, so I went home and coded up the basic game in SpriteKit that evening. We spent the next week and a bit refining gameplay, designing artwork,  gathering feedback from users, changing the name to something that doesn’t ‘leverage a popular app’, and submitted it to the App Store.

I have to confess to not being much of a gamer (and I haven’t built a game before), so I’m not well equipped to assess whether the game is any good, or predict how many will download it. However, the process of building it was certainly enjoyable, and as I learnt a lot doing it, I’ll go through some of the key design decisions here.

SpriteKit

I haven’t done much work with game engines, but for a game newbie, SpriteKit is a very well-designed framework (if you’re okay with iOS 7+ only). Its allegedly heavily inspired by the popular Cocos2D, but beyond that, it’s a first class Apple framework with expected levels of integration and consistency with the rest of UIKit & CoreFoundation. I’d picked up a copy of Dmitry Volevodz’ ‘iOS 7 Game Development’, which uses an endless runner game as an example, and was able to use this to ramp up on the framework pretty quickly.

SpriteKit has a relatively simple and understandable model, which revolves around SKScenes, SKNodes and SKActions. On an 8-bit style game it required liberal use of node.texture.filteringMode = SKTextureFilteringNearest to prevent antialiasing when scaling up low-res artwork.

Some of the other tips/techniques I discovered were:

  • OpenGL really works the simulator – it always spins up the MacBook fans regardless of whether it’s doing much.
  • Prefer using SKActions for behaviour rather than dumping everything in the -update: method.
  • Subclassing SKSpriteNode for each node type is a good idea, in order to keep the code well separated.
  • Implementing UI elements like buttons in SceneKit is pretty clunky. It would be nice if it was easier to mix UIKit controls into the scene.
  • Texture atlas support is nice, but they aren’t generated during command-line builds – this drove me crazy for a while trying to work out why the TestFlight builds kept crashing.
  • OpenGL really, really does not like running while the app’s in the background. Pause the scene in -applicationWillResignActive: and -applicationDidEnterBackground:

Word lists & game logic

The interesting problems that needed to be solved here were:

  1. spawning new letters in a random, yet playable order
  2. detecting when a valid (or invalid) word is formed

Item 2 requires a word list. As a writer of a non-US flavour of English it was important to me that there be a choice of wordlists (thanks to 12dicts). Despite the Apple documentation and some older references to the contrary, there is definitely a British English language preference in iOS, so it was relatively transparent to load the correct list according the the user’s language settings. The word lists required some pre-processing in a simple ruby script to remove words with punctuation and words shorter than 3 characters. Potentially, I could also strip out words that had a valid word as a prefix (e.g. ‘doggerel’ is unplayable as it’s prefixed with ‘dog’), but I’ve kept them in the list for the moment for potential game enhancements.

The word lists contain up to 75,000 words, which is workable as in-memory data, but should really use an efficient lookup mechanism rather than scanning the entire array each time a letter is hit. Because I know the lists are sorted, I can use a binary search – Cocoa provides one with the -indexOfObject:inSortedRange:options:usingComparator: method of NSArray, which I implemented with a prefix check in the comparator. Each collision, the game can quickly check whether the current letter combination is a valid word, is the start of a valid word, or is invalid.

For item 1, letters are spawned randomly using a very simple frequency weighting (i.e. vowels are more likely). Further localisation of the app would need the frequencies adjusted (and accented letters included, depending on language).  However, during play we noticed it got tedious waiting for a random spawn of the one letter you need, so I included a probabilistic component that includes valid next letters more frequently. This results in the game hinting fairly explicitly in some instances (i.e. if there’s only one valid letter).

Graphic design

It’s probably obvious neither of us are designers! We wanted to go for a retro style, both to make it more Flappy Bird-esque, and because we’re both ancient enough to have played C64/Amiga/classic Mac era games. It has an Australian bush theme, just for something different (see if you can spot the Western Grey Kangaroo).

As an example to try to illustrate the workflow, the bird sprite went through the following evolution:

Bird Nerd Sprite v1 S: “I slightly modified the Flappy Bird sprite to make it look more like a magpie lark”

H: “That’s meant to be a magpie lark?”

Bird Nerd Sprite v2H: “Fek me drawing is hard!”

Bird Nerd Sprite v3S: “I gave him a bigger eye.”

Bird Nerd Sprite v4H: “Larger Bird Nerd sprite. I liked what you did with the glasses thing and have tried to enhance that further.”

Bird Nerd Sprite v5S: “I think his eye still needs to be bigger. Should he have more white on his belly?

Bird Nerd Sprite v6H: “I think a coloured beak and legs looks pretty good?”

Bird Nerd Sprite v7S: “Couldn’t help myself – I had to make his glasses bigger”

The coins were added late in development, after the sound effects were added, as they were quite reminiscent of the Mario-style coin collection and solved an issue with legibility and contrast in drawing the letters directly onto the background.

Game Physics

Bird Nerd Screenshot

The basic mechanics are close to that of Flappy Bird – tap to fly up. Flappy Bird appears to instantaneously set the upward velocity to a constant value rather than apply a set amount of force, i.e. it’s irrelevant how fast you were falling before the tap. I did wind back the gravity and upward velocity to make the game a bit more controllable – the early versions were incredibly difficult, and given the spelling component of the game is quite hard, it was a reasonable trade-off to make flying a little easier.

Initially we noticed players lurked at the top or bottom of the screen to wait for letters – we solved this by ‘landing’ and pausing the game if the bird got too low, and spawning letters all the way up to the top of screen (behind the score) to prevent staying high. Initially the letters were completely random with some hit-testing to prevent overlap, but once we went to letter coins it made more visual sense to lay them out on a grid.

The actual physics and collision code was remarkably simple thanks to SpriteKit; the only real issue I came across was a little bit of ‘drift’ in the player’s x position, presumably due to collisions (easily rectified by resetting in the SKScene -update). I shudder to think how much effort it would have been writing something like this in 6502 assembler on a C64.

Revenue Model

I went for iAd in-app advertising as a revenue model – even if I felt it was worth it, charging upfront for a game in such a crowded market is a really hard sell. In-App Purchase is where nearly all of the game revenue is, but it’s a fair bit of additional development work, and is generating wariness in some consumers thanks to an increasing number of slimy implementations. iAd isn’t regarded as a high earner, but I wanted to experiment with it (partly so I could have some stats for the next Perth iOS Meetup). Plus, I just can’t stand non-retina ads. And yes, I’m aware of the irony of complaining about pixelated ads in an intentionally pixelated game!

iAd implementation is quick & easy – the contract can be configured in about 10 minutes online, and integrating a banner view is only a couple of lines of code. There are a few gotchas – most of the Apple iAd documentation still refers to deprecated methods, and there are a handful of rejection-tempting faux pas you need to be aware of, such as displaying a blank banner view, or submitting screenshots showing test ads.

Final Thoughts

Working with someone else on an app was quite valuable, both in the sense of ‘two heads are better than one’, and to keep the motivation and development pace up. At this stage I have no idea how the app will be received, or how to go about marketing it, but I’ll add another post once the dust has settled and I’ve got some download and iAd stats. In the meantime, get your Bird Nerd on and post your score on Twitter!

Download Bird Nerd from the App Store

Basic Auth with a Web API 2 IAuthenticationFilter

MVC5/Web API 2 introduced a new IAuthenticationFilter (as opposed the the IAuthorizationFilter we needed to dual-purpose in the past), as well as a substantial overhaul of the user model with ASP.NET Identity. Unfortunately, the documentation is abysmal, and all the blog articles focus on the System.Web.Mvc.Filters.IAuthenticationFilter, not the System.Web.Http.Filters.IAuthenticationFilter, which is clearly something entirely different.

We had a project where we needed to support a Basic-over-SSL authentication scheme on the ApiControllers for a mobile client, as well as Forms auth for the MVC controllers running the admin interface. We were keen to leverage the new Identity model, mostly as it appears to be a much more coherent design than the legacy hodgepodge we’d used previously. This required a fair bit of decompilation and digging, but I eventually came up with something that worked.

Below is an excerpt of the relevant parts of our BasicAuthFilter class – it authenticates against a UserManager<T> (which could be the default EF version) and creates a (role-less) ClaimsPrincipal if successful.

public async Task AuthenticateAsync(HttpAuthenticationContext context, CancellationToken cancellationToken)
{
    var authHeader = context.Request.Headers.Authorization;
    if (authHeader == null || authHeader.Scheme != "Basic")
        context.ErrorResult = Unauthorized(context.Request);
    else
    {
        string[] credentials = ASCIIEncoding.ASCII.GetString(Convert.FromBase64String(authHeader.Parameter)).Split(':');

        if (credentials.Length == 2)
        {
            using (var userManager = CreateUserManager())
            {
                var user = await userManager.FindAsync(credentials[0], credentials[1]);
                if (user != null)
                {
                    var identity = await userManager.CreateIdentityAsync(user, "BasicAuth");
                    context.Principal = new ClaimsPrincipal(new ClaimsIdentity[] { identity });
                }
                else
                    context.ErrorResult = Unauthorized(context.Request);
            }
        }
        else
            context.ErrorResult = Unauthorized(context.Request);
    }
}

public Task ChallengeAsync(HttpAuthenticationChallengeContext context, CancellationToken cancellationToken)
{
    context.Result = new AddBasicChallengeResult(context.Result, realm);
    return Task.FromResult(0);
}

private class AddBasicChallengeResult : IHttpActionResult
{
    private IHttpActionResult innerResult;
    private string realm;

    public AddBasicChallengeResult(IHttpActionResult innerResult, string realm)
    {
        this.innerResult = innerResult;
        this.realm = realm;
    }

    public async Task<HttpResponseMessage> ExecuteAsync(CancellationToken cancellationToken)
    {
        var response = await innerResult.ExecuteAsync(cancellationToken);
        if (response.StatusCode == HttpStatusCode.Unauthorized)
            response.Headers.WwwAuthenticate.Add(new AuthenticationHeaderValue("Basic", String.Format("realm=\"{0}\"", realm)));
        return response;
    }
} 

Note that you’ll need to use config.SuppressDefaultHostAuthentication() in your WebApiConfig in order to prevent redirection from unauthorised API calls.

Book Review: Getting Started with LevelDB

Getting Started with LevelDB

I was asked to review the new LevelDB book from Packt (disclaimer: I know the author and was provided a free electronic copy). I’ve done a reasonable amount of experimental hacking using LevelDB on iOS in the past,so I was keen to see whether there was much I’d learn from a ‘Getting Started’ book.

Firstly, a bit about my background in this area. Thanks to my abiding hatred of all things Core Data (remind me to do a blog post on that later), I’d looked into LevelDB a little while ago as a possible alternative iOS persistence technology. I’ve used NoSQL (document) databases on web projects in the past, and I was looking for something that gives the performance, simplicity and syncability of something like CouchDB or RavenDB on iOS (before you comment, yes, I know about TouchDB and CouchBase Mobile). I found LevelDB to be a little too low-level to offer a complete persistence solution out of the box, and partially managed to avoid the temptation to go off and build one (another blog post). I knew Andy had a background in ISAM and embedded storage engines, not to mention that horrid C++ language, so I was hopeful he’d have some better approaches to real-work usage.

I also should mention there was a mild controversy over the book when it was released – the title implies generic LevelDB content, whereas the book focuses on practical implementation in OS X and iOS applications, which some people felt was misleading. I didn’t have a problem with it, but then I’m only really interested in using it in OS X & iOS applications.

The ‘building and installing’ chapters are quite detailed, and explicitly show things going wrong, provide an explanation why, and show how to rectify it. I found them a little bit tedious, but people inexperienced with building open-source C/C++ projects would probably need the extra hand-holding (to be honest, I ignored the makefile & built LevelDB in Xcode – kudos to the developers for not making the build too exotic).

Andy then goes on to examine the C++ API and explain the query semantics and some of the core idioms and public data structures. This lays important conceptual foundations, but given I was already familiar with these I skimmed over most of it. It’s worth noting the code in these chapters is all defiantly C++ (e.g. cout rather than NSLog etc) – I can understand why this was done, given the disappointing and inconsistent state of Objective-C API options (more on that in a moment), but it doesn’t make it any less jarring.

Finally we hit the Objective-C code and some of the OS X/iOS samples. I had a bit of a chuckle at the line:

Some people also have a strong aversion to C++ and will avoid anything that lacks an Objective-C interface.

(I suspect he’s probably talking about me). Complicating this task is the fact there’s no official or standard Objective-C API for LevelDB, so the book spends a bit of time covering the three most popular open source wrapper interfaces, and duplicates some of the sample code across the three. Eventually Andy settles on a customised version of APLevelDB for the remainder of the book (though I’m not sold on exposing the raw C++ DB reference), however ideally we’d instead have available a popular, stable, complete, well-maintained, idiomatic Cocoa LevelDB wrapper. This could have made the book much simpler and better, and allowed it to spend more time and energy discussing other topics. Hopefully someone who’s not me will write this someday.

Next we receive a walkthrough implementing a basic sample app – the book describes an OS X application, but it’s worth noting all the downloadable code samples include the iOS equivalent. There are also descriptions of some debugging tools, including dump, lev, and implementing a REPL on an iOS device via an embedded web server (quite a clever trick, though I’m not sure if I’ll ever use it). The sample app is then extended with more advanced functionality, and this is where (for me) things really started to get interesting. This included secondary indexes, key design considerations, custom comparators, record-splitting, and ‘schema support’ extensions to assist in maintaining keys. The last was quite a nifty idea – I can see the potential for a declarative (e.g. json file in the app bundle) index definition mechanism. Less code is always better™.

Chapter 9 was titled “A Document Database”, and I hoped it was something closer to what I’ve used in CouchDB. It wasn’t really – the sample was more geared towards a metadata database of external files. However, it does include a basic text indexing implementation, which is an area I’ve battled with before, so it’s given me enough to take this idea further. The links to test indexing algorithms and open source implementations are also invaluable – additional resources are extensively referenced throughout the book; it’s one of the things that’s been done very well.

Chapter 10, “Tuning and Key Policies” is gold, and probably worth the sticker price of the book on its own if you’re serious about implementing a LevelDB solution. It starts out describing LevelDB under the covers – memtables, SSTs, the eponymous levels, snapshots and Bloom filters. The various performance settings are discussed and recommendations made, then Andy gets into a discussion of how to structure data and keys to optimise performance, much of which revolves around understanding when & how often your data is read and updated. He also discusses some optimisations made by Basho in the Riak codebase.

Lastly, the appendix covers using LevelDB from three scripting languages (Ruby, Python & JavaScript (node)), which was worthy of inclusion given I’ve already come across the need to script data in & out of a database.

In summary, I think the book would be invaluable to anyone looking at using LevelDB from Objective-C, and most of it would still be very useful to developers on other platforms. It’s certainly given me a lot to mull over, and rekindled some of my excitement about LevelDB on mobile devices.

Build Server Traffic Lights

Traffic Light

I’ve wanted real build server traffic lights since I first implemented a Continuous Integration server in the mid 2000s. In those days, the trendy thing to do was to hook up red & green lava lamps to your build server, but CCTray’s red/green/yellow status indicators always seemed better suited to traffic lights. However, it was something that always got put in the ‘someday’ pile. More recently, I’d become interested in hardware automation platforms like Arduino, and it seemed like an ideal first project, so I dusted off the concept.

Obtaining the traffic light unit itself was relatively straightforward – in WA, the old style incandescent units are being progressively replaced with LEDs, so the reasoning was there’d be some that are superfluous to requirements. A few phone calls later, I managed to track down the contractor handling the replacement and do a beverage-related deal for a second-hand traffic light. The hardest part was actually explaining what I intended to do with it!

These traffic light units don’t contain any switching logic or complex electronics at all – they have a 240VAC feed for each light, with industrial grade internal transformers stepping down to 10V and driving 20W high-pressure bulbs. I’d seen reports that the standard bulbs were too bright for indoor use, but a test run showed it was probably just okay, and it was certainly much simpler to keep the lighting as-is while I got the rest of the hardware working.

Traffic Light 3The intention was to run the lights as a networked device (rather than a USB one, requiring an active host computer), as this would enable more flexibility in installation. I ordered an Arduino Ethernet and relay shield from Little Bird Electronics, and set about coding the controller software.

Traffic Light UI

The code is available online here – it’s adapted from a similar project by Dirk Engels. The Arduino runs a web server that serves a page displaying the current status of the light, as well as buttons to control the light and RESTful control URLs to provide build server integration. My main changes to the design were:

  • Integration of a DHCP library, to remove the hard-coded IP address and make it possible to move the light between networks without reprogramming.
  • Bonjour support, to advertise the light at ‘traffic-light.local’ and remove any requirement for DNS entries/DHCP reservations on the network.
  • A failover mode that flashes amber if the light has not heard from the build server in over 5 minutes. This mimics real world behaviour and seemed more appropriate than turning off or displaying the last known state indefinitely.

Traffic Light 2

Wiring in the controller was pretty simple – the 240V mains feed powers the 9V DC power supply for the Arduino, as well as the 10V transformers for the lights via the relay shield. Initially these were switched on the high-voltage side, but the inrush current appeared to play havoc with small switch-mode power supplies (i.e. phone chargers) on the same circuit, so I rewired to switch on the low-voltage side. This also allowed me to remove two of the transformers and freed up some internal space; I ended up being able to neatly mount the controller on one of the unused transformer brackets.

Traffic Light 4Obviously the light needed a pole; I constructed one using galvanised fence post and some sub-par oxy welding. I would have liked to run the wiring down inside the pole, but unfortunately the size of the mains plug was going to make this difficult (given I wanted the light to stay easily removable). A few coats of suitable yellow paint and it was good to go.

After installing the light in the office, we developed a small powershell script to query the build server and update the light. It’s had a significant benefit in putting the build status unavoidably in front of the developers, and the builds have become noticeably more ‘green’ than they have been for some time.

There are a few areas I’d design differently if I did it again:

  • Use a hardware flasher circuit for the failover mode (via the fourth relay) – the software flasher works okay, but there’s a noticeable stutter in the flashes if the controller is doing something else (like responding to a web request). I’m not enough of a hardware whiz to build one of these though.
  • Install bulkhead RJ45 & 3-pin PC power connections on the traffic light housing, so that the cables are detachable – this would permit variable cable lengths and potentially allow routing inside the pole.
  • Use low-wattage bulbs rather than the specialised 20W high pressure bulbs – the traffic light is a bit bright straight-on. Unfortunately the existing bulb holders have a unique bayonet mount and they’d need to be replaced with something else (e.g. automotive BA15S).