Clear Annoying Vim Macros

Recording a macro in vim involves pressing q followed by a letter in visual mode. Next time you press the letter it will run the recording. Handy, but not so great when you accidentally store something in w and then press it when you really meant to save.

The quickest way around this is to press qwq in visual mode. This just records an empty macro for register w.

You can see what’s been recorded by running the :reg command. It’s often possible to work out what it is that you’re accidentally running by looking at the listing.

Macros can be wiped in a more manual way by running :let @w=''. (Replace @w with the register that you corrupted).

Various registers are used by vim to implement the likes of yank so it’s dangerous to arbitrarily clear all the macros, however a simple vimscript function could clear all the alphabetical registers which should be safe.

Clear Annoying Vim Macros

Beginning Unit Tests

The most common thing I hear from people who don’t unit test but would like to is that they don’t know where to start. I’ve documented here the basics of how to write an initial unit test, given some knowledge of Zend Framework and phpunit. I’ve also tried to cover the main points of conflict that developers who have not previously automated their tests tend to run into.

Testing goes from “want” to “have”. So start with a “want”, e.g. “title must be html escaped”. Presumably this “want” will be a description of a new feature or solution to a bug in your software.

Document your want in a test case:

// A PHPUnit subclass with some fixtures to initialise your application
// and make it runnable.
class ProductPageTestCase 
  extends \Zend\Test\PHPUnit\Controller\AbstractHttpControllerTestCase 

  public function setUp()
     // ...

  public function testTitleMustBeHtmlEscaped() 

Work out how to express your “want” in terms of an assertion. Start at the assertion and work back:

$this->assertContains("<title>" . htmlspecialchars("<") . "</title>", $content);

(I’m using htmlspecialchars because wordpress really loves to ruin escaping in code samples, normally you would write the escaped less-than in the most simple way).

Next work out how to run the page. In the case of a Zend controller test you run:

$content = $this->getResponse()->getContent();

All that’s happening here is that Zend is simulating the dispatch process that would happen when it receives a real HTTP request.

Work out how you can make the content contain your use case:

$this->database->insert('product', array('product_name' => "<"));

This is the "fixture, run, test" trifecta which you will see in all tests.

Note that what I've written in the example is really an "integration" or "component" test which is testing the full application stack rather than one completely isolated unit. This is usually what you must start with if your application does not currently have any test automation and will give you the most testing value in the fastest way. A unit test of your application classes would need to avoid dependencies as much as possible in order to avoid failing every time a dependency changes. The method of writing the tests is still essentially the same.

The fixture is a point where many get confused. Surely you have to make massive changes to your app to make your database editable from within a test, right? The answer is yes — and that's expected. You are forced into a design by your tests which requires that you make dependencies injectable and mockable. This requires that your application is modular and decoupled which is generally a good thing so the tests are error-checking your design too.

Tools like Zend's service locator are intended to make this kind of design easy, for example a service factory can return an in-memory sqlite for your database and mock transports for your apis when in a test environment. If you're using a modern framework like Zend you may find that even if you're not automating your testing now you are already using the tools you need to do it (but you will need to change the specifics of your implementation to take advantage of that).

In Zend, a test setup fixture might look something like this:

public function setUp() 
   // I prefer to use a config which uses my autoloads but skips production
   // credentials and overrides everything to mocked values.  This gives a 
   // short reusable test setup but means your service factories have to do 
   // more work.
   $this->setApplicationConfig(require APP_ROOT . "/config/environments/unit-tests.php");
   $services = $this->getApplicationServiceLocator();
   // You can alter the services manually too:
   $db = new \PDO("sqlite::memory:");
   $db->exec("CREATE TABLE product ( ... )");
   $services->setService("MyAppDb", $db);

There are several choices for implementing fixtures. My example simply manipulates the persistence layer directly, which is usually the only choice for existing untested software (at least to begin with). Fixtures can be more readable and further encourage good design if you involve your application's real models (or mocks thereof) rather than bypassing them and using the persistence layer directly.

Generally when you're modifying an existing test case, it should be very obvious how to create a new test method since new methods should look a lot like the existing ones. Structural duplication is expected in a unit test since simplicity — especially avoiding deeply layered software — is more valuable than duplication in test maintenance.

It is also important to be aware that the test will make it hard for you to change the behaviour of that use case or of its fixture. This will make it necessary for you to maintain backwards compatibility features for the sake of your tests. This can be awkward and seems to be a nasty surprise for new developers who are used to breaking compatibility at will. Typically large applications worked on by multiple people need to maintain backwards compatibility anyway so this is another case of the unit tests forcing you into a particular design philosophy which is usually a good idea anyway. Nevertheless this seems to be a huge annoyance among new unit testers so it's worth keeping in mind how your software design needs to change.

Once your test is complete, watch it fail. It should fail in the right way; so in the example above you should expect to see an invalid HTML title produced. This is important because it verifies that your tests are detecting the condition that you intended.

Next implement the feature. Do this in the fastest possible way you can. Don't worry about what sins you have to commit to make the thing happen. Repeatedly run the test until it passes. This is a great way to break out of design worry deadlock and over-engineering — it does not matter how it works so long as it works.

If you find yourself writing lots of print statements at the implementation stage then your test may be too broad. Try abstracting and writing unit tests of the specific part that you are working with. Once again the unit testing is giving you a signal to change the design of your software. In general your tests should be replacing debugging, not adding additional work. Avoiding the debugging process is probably the most important factor for making TDD a fast development process. This can be a hard habit to break and needs some practice. Try implementing a new assertion every time you would have added a print statement.

Once the test passes you can work on cleaning up the code. Remove the sins you committed, clear up and make things pretty. You should not generally need to make major architectural changes at this point; your main job is to remove anything that could be interpreted as duplication, as well as getting rid of future risks like global variables.

If you do this right, you can develop features very quickly without putting any major hurdles into the product for the next time that you have to modify it. Quality trends upwards and your existing code has less impact on the cost of new features. The more you keep the product supple, the more your simple changes become trivial, and your complex changes become achievable.

There is more to testing than this, of course, but the general mantras are:

  • always work from "want" to "have"
  • keep your tests as simple as possible
  • work in small increments
  • don't duplicate code
  • remember the cleanup stage
  • allow the unit tests to steer your design decisions
Beginning Unit Tests

Simulating a High-Latency Network Traffic

Latency is extremely important for web performance since every HTTP connection involves a round trip to the server and there is limited concurrency to avoid this. It can be very useful to see what a high-latency connection will look like and can highlight some problems you were not expecting. Especially useful, it can make the differences between two different implementations more pronounced.

To simulate latency use the tc (traffic control) software from the Linux command line. Adjust the delay as you see fit.

[root@box] # tc qdisc add dev eth0 root netem delay 42ms

This is using the netem module which has several features to configure packet loss, duplication, and latency. (You can find these documented in your systems manual pages of course).

To remove the latency either reboot or run the following:

[root@box] # tc qdisc del dev eth0 root
Simulating a High-Latency Network Traffic

Programming FCB1010 UNO Chip Stomp Boxes for BIAS FX

I really like BIAS FX for headphone playing. It has some limitations and idiosyncrasies but the sound is excellent and the pricing is pretty competitive for what you get.

I decided I would like to control BIAS using a footswitch so I bought a Behringer FCB1010. This worked OK for the expression pedals and I could configure the toggles but the LEDs did not match the toggle state. So I upgraded the FCB1010 with the UNO “unofficial” firmware in order to get its stompbox mode.

Unfortunately people aren’t kidding when they say this is really really obfuscated to configure.

My goal is to configure stomp boxes to work with BIAS FX’s midi learn (which they do not do by default). In this case we need to program a CC to send for each stompbox and pedal and the patch selects are more or less irrelevant.

Global Configuration

I’m assuming the UNO is installed and the memory is initialised. You need to do this in order to enable stompbox.

  • hold “down” while turning on the box

  • turn on LED “9” (this is stompbox mode)

  • turn on LED “8” if you want the stompboxes to be on the upper row. I do not and will assume you keep them on the lower row too, it is relevant in some parts.

  • you might need to turn on LED “5” (disable repeating PCs, I didn’t but I’m using CCs not PCs so I don’t think it applies.)

  • everything else should be turned off by default

  • hold “down” to save and exit (we’re skipping a bunch of screens but that’s fine)

Patch Configuration

To configure stomp box 1.

Configure Patch Stompbox Resets

  • hold down stomp box 1 and the “down” button. It does not seem to matter if you have a patch selected or not but this might just be a display thing.
  • the lights enabled on this screen will tell you which stompboxes will be set/reset/left alone when you switch to the patch. If you set the upper row to be stompboxes then this will be buttons 6-10; otherwise 1-5.

  • hold down each stompbox button to set them to be unchanged.

  • select “up” to move to the next screen

There’s a bunch more stuff you can do here like change the CC of the pedals depending on your stomp and possibly use the set/reset behaviour to make patches out of your stomboxes. You’ll need to experiment with these settings. I haven’t worked out what it does in BIAS yet. I suspect it will not work well as BIAS does not seem to respond to on/off toggles with usually is implemented with with same CC but a 0 or 127 velocity. BIAS appears to merely change toggle state every time a CC arrives, a bit of a shame since the UI does have separate switches for on and off.

Configure Toggle Mode

  • toggle switch “8” on (toggle mode for CC1)
  • toggle switch “9” off (toggle mode for CC2)

  • toggle switch “10” on if you want a momentary switch; off if you want a stompbox.

  • press “up” to continue to the next screen

The toggle modes mean that the switch will send a primary and secondary values separately when enabled. It probably won’t matter for BIAS.

A momentary switch is something that will only stay on while you are holding it down. For some reason this was enabled for all switches by default. There is a way of turning this off globally.

Configure the Stomp Box Value

This should be pretty familiar if you’ve already used the FCB1010 with the standard firmware.

  • everything should be off except “6” which means “CC1”. Hold down switches to disable them. This says what messages will be sent. There is a pre-defined order for sending messages which is in the manual somewhere.
  • press button “6”, it will flash

  • press “up” to configure the controller number

  • use the footswitches or expression pedals to set whatever value you like (I set it to 26)

  • press “up” to configure the primary velocity

  • set it to “127”

  • press “up” to configure the secondary velocity

  • set it to “0”

  • hold “down” to save the changes

Again BIAS does not appear care about the velocities but the way I’ve done this you do need to set them so you may as well set them the standard way. Because of this you will need to manually drag the FXes into the right places to make sure the state matches the LEDs. This is the most messy limitation with this system and means we can’t really be equivalent with a real stomp box.


It may be that my programming can be optimised somewhat. I think the “set/reset” functionality could be promising if only BIAS had a dedicated on/off sensor.

The stomp box (or patches) should be able to be used to change the values of the expression pedals which would make it possible to have a contextual expression pedal for even more flexibility. To be honest single expression pedals are very cheap so perhaps this is overkill.

It would be nice to have some more tactile controls just to change FX plugin values but you still run into the problem where the controls are not in the right state to start with unless there is two way communication between these devices.

It’s sort of annoying that the stompbox signals can’t (as far as I know) be different on a different bank because you’re limited to a mere 5 pedals now.

After all this I would recommend that UNO’s companion editor app is used since everything I’ve done here should be perfectly easy to change using that. Only reason I didn’t use is is that there is a delay in obtaining a license key and I am impatient.


User guide:

Demo of the same process for a G Major 2 (with less explanation about what the stages mean):

Programming FCB1010 UNO Chip Stomp Boxes for BIAS FX

Working Around Merge Conflicts In a Git Rebase

Sometimes you get merge conflicts when rebasing complicated repositories. It doesn’t always happen but it seems to be when existing conflicted merges are resolved. If you’re sure that your rebase doesn’t really conflict with anything in your repository, then you can tell the merge strategy to pick the new change using the following command:

git rebase -s recursive -X theirs master

This just picks the state in the new patch; in other words the diff will reflect the new changes plus any merge resolutions.

It’s still possible to see conflicts when files are deleted and modified in a merge conflict, however you can resolve this with git checkout --theirs FILE ; git add FILE.

I’m not completely sure on why this situation occurs. I think git is simply being safe about how it treats the patches in the history. What we’re doing here is pretending the merge solution never happened and that all commits played well with each other which is not the case in reality. Therefore this command is destroying the history of patches as they actually happened. I’m merging extremely old repositories that have very messy histories anyway so it’s difficult for me to assess the impact of this other than to say that the log looks OK and the files are as they should be.

If all you’re looking for is a nice output of git blame and a bisectable history then this method should be OK for you.

Working Around Merge Conflicts In a Git Rebase

Splitting Subsets of a Git Repository

Sometimes I need to split one or two useful files from a predominantly useless repository. This can be done with subtree merges and subdirectory filters, but if the files are not organised very well then this can prove difficult.

The following quick filter will get you just a few files, providing they don’t move around too much:

git filter-branch --index-filter \ 
  'git rm --cached -qr -- . && \ 
  git reset -q $GIT_COMMIT -- \ 

Note that I’m using index-filter rather than tree-filter.

We’re just deleting everything and then resetting the files we want all in one go.

As I wrote in my other post “merging a git repository”, we can move all those files to a subdirectory easily:

git filter-branch --prune-empty \
  --tree-filter 'mkdir -p new-subdir; \
  git ls-tree --name-only $GIT_COMMIT | \
  xargs -I FILES mv FILES new-subdir'

This means we can merge just a tiny part of a repository in very easily.

cd target-repo
git pull --allow-unrelated-histories filtered-repo

Modern versions of git want the --allow-unrelated-histories argument. You’ll need to rebase as well or those commits won’t have any of the stuff your existing repositories had which can prove pretty annoying.

Make sure it’s to your liking by checking out one of the revisions you just imported.

Splitting Subsets of a Git Repository

Beware of Dumb Objects

In some projects I often see systems which look like this:

class InvoiceFactory
    public function __construct($service);
    public function createObjects($param);

class Invoice
    public function __construct($array);
    public function createObjects();

class InvoiceProducts
    public function __construct($object);

class WarehouseInvoice
    public function __construct($invoice, $products);

// Classes are used like this:
$invoices = new InvoiceFactory($database);
// Load database rows by id.
$array = $invoices->fetchInvoices($someId);
// Create useable objects from those rows
$invoice = new Invoice($array);
// List of products
$productList = $invoice->getProducts();
// Usable list products.
$products = InvoiceProducts($productList);
// Convert those rows into a usable form.
$objectsDto = $products->fetchProducts();
// Convert one useable form into the one that we actually need.
$products = new WarehouseProducts($invoice, $products);
// Convert list of products to one we can use in a template
$yetAnotherDto = $products->listProducts();
// Actually use the object.

Here we’ve gone from a factory to an object to a list of objects and back to a single object again several times. This can go on and on through several iterations of tiny one-function stateless objects that just convert data.

In this example, the code never actually does anything until the objects are converted into primitive data because all the models only accept primitive data. It seems attractive at first that all the objects would be without dependencies and behave as simple services, however there are several problems with this:

  • usage code is long
  • all objects involved need to be understood in terms of how they accept and produce data in order to know what’s going on
  • difficult to include data from other services without performing more data wrangling
  • difficult to change the behaviour of primitive data since it’s all pre-computed
  • no common API for access to data derived from the service, every client has to build their own conversion
  • difficult to unit test correct usage because it all relies on integration

There seems to be a natural tendency to obsessively break objects down to their smallest possible form, especially around a single database table or similar technical object.

I think the example above would have been just as good with a single object of functions to perform these conversions.

When you find yourself implementing this kind of pattern, consider using just one root service model and smarter objects to access the data.

// Persistence and external service access.
class ObjectService
    public function __construct($service);
    public function fetchObjectById($id);
    public function fetchExtraObjectData($id);

// Containing value object.
class ObjectList
    public function __construct($service, $id);
    public function getActualObjects();

// Element value object.  This can still use the ObjectService.
class ActualObject 
    public function __construct($objectList, $id);
    public function getActualProperty();

// Declare dependencies.
$service = new ObjectService($database);
$objects = new ObjectList($service, $id);
// Load usable data.

This gives you a more flexible implementation where your data access concerns are dealt with by the smart value objects and the service concerns are dealt with the root object service. The maintainer does not need to know anything about the interactions between these objects, they only need to know the dependencies (which should be defined by the constructor interface).

The conclusions I draw are:

  1. don’t be scared of high-level value objects
  2. dependencies are fine if they mean client code needs to know less
  3. create new abstractions only when they remove work from client code
Beware of Dumb Objects

W3af Quickstart

Sometimes a blog post is better than memory.

mkdir -p src/w3af
cd src/w3af
git clone --depth 1 .
virtualenv venv
# It does seem to be necessary to do this rather than just using the bins.
# This is generated by the consolescript
. /tmp/

Next we’ll create a usable profile to spider a website and try to xss it.

./venv/bin/python ./w3af_console
w3af>>> plugins
w3af/plugins>>> list crawl
w3af/plugins>>> crawl web_spider
w3af/plugins>>> audit xss
w3af>>> profiles
w3af/profiles>>> save_as spider-xss
Profile saved.

(To disable a plugin later use type !plugin, e.g. audit !xss.)

When you next use w3af you can run profile use spider-xss.

Now exploit some things:

w3af>>> target set target
The configuration has been saved.
w3af>>> start
New URL found by web_spider plugin: ""

Pressing enter will give you a quick status on what it’s doing.

Note that in my version the console will hang if you haven’t yet set a target which will mean you have to reconfigure everything so take care to avoid this.

Wait a very long time and you’ll get a report. Any found exploits will be printed out in real time.

It’s a good idea to test your profile like this:

w3af>>> plugins audit xss
w3af>>> plugins crawl web_spider
w3af>>> target set target http://localhost/inject-me.php
w3af>>> plugins crawl config web_spider 
w3af/plugins/crawl/config:web_spider>>> set only_forward True 
w3af/plugins/crawl/config:web_spider>>> back
The configuration has been saved.
w3af>>> start

Here’s the exploitable script:


<?php echo @$_GET['text']; ?>

<form method="GET">
<input type="text" name="text">
<input type="submit" value="go">

The result:

w3af>>> start
New URL found by web_spider plugin: "http://localhost/inject-me.php"
A Cross Site Scripting vulnerability was found at: "http://localhost/inject-me.php", using HTTP method GET. The sent data was: "text=" The modified parameter was "text". This vulnerability was found in the request with id 37.
Found 1 URLs and 2 different injections points.
The URL list is:
- http://localhost/inject-me.php
The list of fuzzable requests is:
- Method: GET | http://localhost/inject-me.php/inject-me.php
- Method: GET | http://localhost/inject-me.php/inject-me.php | URL encoded form: (text)
Scan finished in 1 second.
Stopping the core...

The output plugins are also useful.

  output text_file
  output config text_file
    set output_file output-w3af.txt
    set verbose True

The above will produce more greppable output. Useful if you’re spidering a whole site with thousands of pages.

$ grep ' vulnerability\]' output-w3af.txt
[Wed Aug 10 14:09:36 2016 - vulnerability] A Cross Site Scripting vulnerability was found at: "http://localhost/inject-me.php", using HTTP method GET. The sent data was: "text=" The modified parameter was "text". This vulnerability was found in the request with id 37.

Not sure why it tells you that the sent data was “text=”. It looks from the request log like it actually sent something else entirely. I think the problem may be that it’s looking for the string =" in the document.

GET /inject-me.php?text=shlj2%3C%2F-%3Eshlj2%2F%2Ashlj2%22shlj2shlj2%27shlj2shlj2%60shlj2shlj2%20%3D

So over all I’m not too impressed with the xss detection. There are far to many false positives, the UI is rather buggy, and it’s difficult to understand exactly why it reports empty strings as injections. Hopefully someone will find this braindump more useful though.

W3af Quickstart

Event Based Actions to Remove Boilerplate Code

In my application I tend to have a large number of actions that fall into a set of very similar boilerplate structures. They are never close enough to use a standardised model framework and there are too many different structures to use fancy controller configuration. A useful pattern to deal with this is to use events to remove the boilerplate.

public function createSomeObjectAction() 
    $form = new SomeForm();
    $editor = new SomeEditor();
    $action = FormAction::create($this, $form);

    $action->onValid(function($context) use($editor) {
       $id = $editor->createSomeObject($context->getValidData());
       return $this->redirect()->toRoute('EditObject', array('id' => $id));

    $action->onOther(function($context) {
       $view = new \Zend\ViewModel\ViewModel();
       $view->setVariable('form', $context->getForm());
       return $view;

    return $action->run();

It’s useful to see if you can write controller actions this way without using any if statements. This makes it easy to establish integration tests which just trigger each of the events. You also don’t need to do any deep zend specific configuration like you would with rendering strategies or event listeners — it’s “just php”.

The alternative here is to implement some interface which has its own idea of the onValid and onOther functions and passing this to some strategy.

public function createSomeObjectAction() 
    $editor = new SomeEditor();
    $model = new SomeObjectActionModel();
    return ActionModelRunner::create($model)->run();

While this makes the action a lot shorter, it means that the maintainer now has to understand more APIs: the controller action, the action model interface, the action model, and the action model runner. With the event based system you only really need to understand the controller action and the FormAction. The functionality is right in front of you.

I find in general that reducing the number of APIs that the maintainer has to deal with is usually a good idea up to the point where the implementation complexity is too hard to understand in one go. These actions are basically just chaining function calls together — no if statements — so abstracting things does not have any benefit.

Another use case is to deal with errors as a structure:

public function createSomeObjectAction() 
    return $this->handleErrors(function() {
        $model = new SomeModel();
        return ViewModel();

private function handleErrors($action) 
    $handler = new ApiErorrHandlerAction($action);
    $handler->on('SpecificException', function($handler, $ex) {
        return $handler->respondWithTemporaryError($ex->getMessage());
    return $handler->run();

This is very useful when dealing with APIs which can throw errors at any point and reduces your need for the same boilerplate try-catch in every single action. Instead your intention is shown by the structure of the action.

The drawback is that your generic action needs to be somewhat understood by the reader. This is not too bad in the case of the FormAction example, but it’s more of a problem when you write something like a PaginatedListingAction or a CreateOrEditAction which will abstract dealing with your model code. It’s important to avoid making things so abstract and deeply layered that the maintainer can’t understand how the alter the behaviour of the system.

That said, if you have a lot of actions with the same structure, this technique of declaring behaviour as events can be more understandable and more flexible than a highly engineered framework based on model interfaces.

Event Based Actions to Remove Boilerplate Code