Therefore I started looking for possible solutions to this problem. After some brief research purchasing a dog whistle (usually used for dog training) seemed the logical choice, since according to Wikipeida they produce a frequency between 16000 Hz and 22000 Hz (the ultrasonic range from 20000 Hz upwards being outside that audible to the human ear) - perfect since the goal is not to add an annoying whistling sound to an existing dog barking sound.
A dog whistle however is a rather low tech approach, therefore I decided to make an electronic whistle. The first step was purchasing some speakers that can operate in the ultrasonic range. It turns out that general purpose tweeters (Maplin: WF09K) can do 4 kHz to 40000 Hz, I found a set of 4 going cheap on eBay. The problem then becomes generating the ultrasonic sound to power the speakers, I was considering purchasing a cheap second hand car audio amplifier from eBay and using the computer to generate the sound but it seemed like more work than it should be. After checking the Maplin site again I found out they sell an "Ultrasonic Pest Repeller" which produces a frequency adjustable between 12 kHz and 38 kHz.
I was not sure if connecting the Pest Repeller straight to 4 (150Wmax / 75Wrms) tweeters would work however it actually works too well therefore making the next step adding a volume control and perhaps purchasing some hearing protection before trying it out.
Also worth noting that some commercial products seem to use tweeters that look very similar to the ones I chose.