Thursday, November 5, 2009

Optimizing a custom Trim() function in C#

I needed to write a Trim() function that removed anything that wasn't a digit from the beginning and end of a string. What usually happens when I write a blog entry like this is that someone posts a comment a few days later saying "hey, you didn't need to write that, it's already in the .NET library." If it is, I haven't been able to find it.

Here is my TrimNonDigits() function first attempt:

protected string TrimNonDigits_v1(string text)
{
    while (text.Length > 0 && !Char.IsDigit(text, 0))
    {
        text = text.Substring(1);
    }
    while (text.Length > 0 && !Char.IsDigit(text, text.Length - 1))
    {
        text = text.Substring(0, text.Length - 1);
    }

    return text;
}

Now this function works and for what I needed it for (trivially short strings) it would do the job. But it is my intention to add it to a string extension library and I had a fear that someone would pass in an enormous string and with all the Substring() operations creating new strings this may become extremely inefficient.

To see how inefficient this function was I wrote another unit test to specifically measure the speed at which the code ran. Here is that unit test:

[TestMethod]
public void trim_non_digits_big_text()
{
    int repeat = 10000;
    UnitTestDerived utd = new UnitTestDerived();
    string lotsOfCharacters = String.Join("", Enumerable.Repeat("abcdef", repeat).ToArray());
    string text = lotsOfCharacters + "1" + lotsOfCharacters;
    string expected = "1";
    string actual = utd.TrimNotDigits(text);
    Assert.AreEqual(expected, actual);
}

The unit test passes but takes 21 seconds to run. Although extremely unlikely that this amount of non-digit-text on either side of a number might appear it's still a valid use case so the code needed reworking so that the Substring() function was only called once. This is the resulting change to the code:

protected string TrimNonDigits(string text)
{
    int start = 0, end = text.Length - 1;
    while (text.Length > start && !Char.IsDigit(text, start))
    {
        start++;
    }
    if (start != end)
    {
        while (end > start && !Char.IsDigit(text, end))
        {
            end--;
        }
    }
    return text.Substring(start, 1 + end - start);
}

The unit test runs against this new function in under 1 second. That's a dramatic difference and good demonstration for the case of not using Substring() repeatedly in a loop if you can help it.

A side note. I also had several other unit tests that checked the validity of the results of the algorithm before I did the optimization. These included the testing of edge cases and extremes. This made the optimization refactor a cinch because I was fairly confident at the end that the algorithm will work in all situations.

Tuesday, November 3, 2009

MultiThread testing for speed

I recently wrote this little test program to validate that multi-threading was working the way I expected it to.

class Program
{
    private delegate void funcs();

    static void Main(string[] args)
    {
        funcs[] functions = new funcs[]
        {
            new funcs(MultiThreadTest),
            new funcs(SingleThreadTest)
        };
        foreach (funcs function in functions)
        {
            DateTime startTime = DateTime.Now;
            for (int i = 0; i < 10; i++)
            {
                function();
            }
            while (ThreadWorker.ThreadCount > 0)
            {
                Thread.Sleep(1);
            }
            Console.WriteLine("{0} time taken: {1}",
                function.Method.Name, DateTime.Now - startTime);
        }
        Console.ReadLine();
    }

    private static void MultiThreadTest()
    {
        ThreadWorker threadWorker = new ThreadWorker();
        ThreadStart threadDelegate =
            new ThreadStart(threadWorker.Runner);
        Thread newThread = new Thread(threadDelegate);
        newThread.Start();
    }
    private static void SingleThreadTest()
    {
        ThreadWorker threadWorker = new ThreadWorker();
        threadWorker.Runner();
    }
}

class ThreadWorker
{
    public static int ThreadCount = 0;

    public ThreadWorker()
    {
        ThreadCount++;
    }
    public void Runner()
    {
        IEnumerable<string> strings =
            Enumerable.Repeat(
            @"the not so quick brown fox was
caught by the hen and eaten with eggs", 50);
        string text = String.Join(" ",
            strings.SelectMany(a => a.Split()).ToArray());
        for (int i = 0; i < 1000; i++)
        {
            // Do some random LINQ stuff to occupy the processor
            string[] list = text.Split(' ');
            string[] l2 = list.Distinct().ToArray();
            l2 = list.OrderBy(a => a).ToArray();
            l2 = list.OrderByDescending(a => a).ToArray();
        }
        ThreadCount--;
    }
}

I tested this bit of code by running it four times on each of two machines.

The first was a dual-core and the single-threaded-test took an average of 33.4 seconds and the multi-threaded-test 17.8 seconds. This makes the multi-threaded part 1.9 times faster which is about what you'd expect if you allow two processors to work on the problems in parallel.

The second machine was a quad core with hyper-threading so essentially eight cores. This took an average of 29.3 seconds for the single-threaded-test and 4.6 seconds for the multi-threaded-test. An improvement factor of 6.4, not as close to 8 as I was expecting but not that far off. If anybody knows why the eight cores do not come as close to an eight-times factor as the two cores came to a two-times factor I'd love to hear from you in the comments.

The two processors were:

  • Intel Core 2 CPU 6400 @ 2.13 GHz
  • Intel Xeon CPU L5410 @ 2.33GHz

Something that I found interesting was that the slower processor ran 1.65 times faster than the fast processor when taking advantage of multi-threading. This has important implications for the software that you write. The single-threaded test ran 1.14 times (14%) faster on the faster processor. However, the multi-threaded code on the slower processor runs 65% faster than the single-threaded code on the faster processor.

If you're looking for a performance boost there may be more performance in multi-threaded code than in a faster processor. In fact, processor speed is probably not what you're looking for. The best combination would be multi-threaded code on multi-core boxes.

 

Monday, November 2, 2009

Using LINQ to join two string lists without repeats

I have a list of members that have to play a game against each other and I want to generate a complete list of all the members against every other member without repeating any games. My list looks like this:

string[] members = {"Alphie", "Jerome", "Silky", "Buzz" };

My first attempt at generating a list using LINQ was this:

IEnumerable<string> q = from one in members
                        from two in members
                        select one + " plays " + two;

which resulted in:

Alphie plays Alphie
Alphie plays Jerome
Alphie plays Silky
Alphie plays Buzz
Jerome plays Alphie
Jerome plays Jerome
Jerome plays Silky
Jerome plays Buzz
Silky plays Alphie
Silky plays Jerome
Silky plays Silky
Silky plays Buzz
Buzz plays Alphie
Buzz plays Jerome
Buzz plays Silky
Buzz plays Buzz

This is not exactly what I was looking for. I wonder who the winner would have been in Buzz versus Buzz?

The secret is to put a where clause before the select statement:

where one.CompareTo(two) < 0

This will eliminate duplicates when CompareTo(two) == 0 and also alphabetically sort the two players eliminating them playing against each other a second time. This is the complete code snippet:

IEnumerable<string> q = from one in members
                        from two in members
                        where one.CompareTo(two) < 0
                        select one + " plays " + two;

and here are the revised results:

Alphie plays Jerome
Alphie plays Silky
Alphie plays Buzz
Jerome plays Silky
Buzz plays Jerome
Buzz plays Silky
 
 

Thursday, October 29, 2009

Unit testing and code conversion

Python

I've just done a little exercise in code conversion from Python to C# and the icing on the cake were two unit tests written in the Python code that confirmed that my code had been converted correctly.

I know nothing about Python so it was lucky that this code was about 10 functions and only a few pages long. For the bits of code (mostly syntax) that weren't obvious I found an online quick reference to Python and used that to search for the unusual keywords and work out what they did.

The code conversion I did inline by copy pasting the Python code into a C# class in a Visual Studio project and then converting each line into C# leaving the variable names and code structure intact as much as possible. I had the original Python file open in Notepad++ on a second monitor as a reference.

At the bottom of the Python file were a couple of unit tests with expected results and input parameters. I rewrote those unit tests in Visual Studio's Unit Tester and used the same inputs and expected outputs and they ran successfully. As a result of those unit tests that was probably the most successful code conversion that I have ever done and a very productive one as well.

Quicken Deluxe 2010 spams your desktop

Quicken

I just bought and installed Intuit Quicken Deluxe 2010 from their site for $59. The first thing to disappoint me was the fact that there was nowhere during the purchasing procedure to enter my coupon code and get a discount. I was in too much of a hurry to bother with hunting around for how to do this so they got to keep my $10 discount.

The second thing that really annoyed me was that the installation procedure dumped three spam links on my desktop for other products that they sell.

I haven't even run their software yet and they've had two strikes that annoyed me sufficiently to blog negatively about them. It might be great software but so far it looks like a shady company using back-alley tactics.

Edit on 11/30/2009: That link above (Intuit Quicken Deluxe 2010) has the software priced at $54.99 so you save a through dollars through NewEgg. I didn't know about it at the time so paid the full $59 to Intuit.

Tuesday, October 27, 2009

Network Saturation Finally




I have finally achieved my goal of network saturation on my home network.
As with most people, I have a small off-the-shelf router that does the standard 100 Mbs. Most of the computers in my house are hardwired because the wireless signal is slower and weaker in the far corners of the house and also because we bought a spec home that had all the rooms pre-wired with Ethernet.
In the past, when I've been copying files from one computer to another and I've looked at the transfer rate over the network I've been disappointed that only 40% to 60% of the available bandwidth was being utilized. The hardware supports 100 MBS so why isn't it transferring data at that rate dammit?
The reason is because of the slowest component in the chain which has always been the hard disk speed. Well not anymore. I've just bought myself a new computer and with this I got a Patriot Torqx PFZ128GS25SSDR 2.5" Internal Solid state disk (SSD) which promises 260 MBS read and 160 MBS write.
The computer that I was copying from had the data sitting on a Seagate Barracuda LP 1.5TB 3.5" SATA 3.0Gb/s Hard Drive -Bare Drive. I don't know what the read speed of that is (yet - I'll come back and update this later) but I'm guessing that it's over 100 MBS or I wouldn't have achieved network saturation.
Is it going to be worth getting a faster router? Not yet I think. The times I'll be copying between computers with fast hard drives is probably going to be rare. I'll wait until my internet connection exceeds 100 MBS. My prediction is that will happen in about 7 years time.

DVDBurn.exe in Server 2003

I've just been battling for the last 30 minutes to try and burn an ISO as an image to a DVD. I have a version of Nero Burn and tried to line up the planets with this bit of not-so-great software and ended up creating a data DVD with the ISO as the single file on this DVD. I took a second attempt with Nero but just couldn't find the option to create it as an image.

Did I mention that I was burning this DVD image from Windows Server 2003?

I stumbled across this little gem of a utility that I must have installed on this server with the Resource Kit Tools. It's called dvdburn.exe and has a younger brother called cdburn.exe.

DVDBurn.exe takes 3 parameters:

dvdburn <drive> <image> [/Erase]

Worked like a charm:

C:\Software>dvdburn d: en_windows_7_ultimate_x64_dvd.iso
Media type: DVD-R
Preparing media...
Error setting timestamp; this error will be ignored, some drives can work without this
- 100.0% done
Finished Writing
Waiting for drive to finalize disc (this may take up to 30 minutes).............

Success: Finalizing media took 10 seconds
Burn successful!