Decimal.MinValue costs more than you expect

Recently during profiling session one method caught my eye, which in profiler's decompiled version looked like this:

public static double dec2f(Decimal value)
{
    if (value == new Decimal(-1, -1, -1, true, (byte) 0))
        return double.MinValue;
    try
    {
      return (double) value;
    }
    catch
    {
      return double.MinValue;
    }
}

This is part of the legacy code written years ago and according to the profiler (which was in sampling mode) there were too much time wasted in this method. In my opinion this is because try-catch block preventing inlining and I spent some time to refresh my memories about decimal to double conversion tricks. After ensuring this conversion can't throw, I removed try-catch block.

But when I looked again at simplified version of source code, I wondered why decompiled version shows strange Decimal constuctor while it is simple Decimal.MinValue usage:

public static double dec2f(decimal value)
{
    if (value == DecimalMinValue)
    {
        return Double.MinValue;
    }

    return (double)value;
}

First, I thought it is a decompiler's bug, but fortunately it also shows IL version of method:

    public static double dec2f(Decimal value)
    {
      if (value == new Decimal(-1, -1, -1, true, (byte) 0))
.maxstack 6
.locals init (
[0] float64 V_0
)
IL_0000: ldarg.0      // 'value'
IL_0001: ldc.i4.m1    
IL_0002: ldc.i4.m1    
IL_0003: ldc.i4.m1    
IL_0004: ldc.i4.1     
IL_0005: ldc.i4.0     
IL_0006: newobj       instance void [mscorlib]System.Decimal::.ctor(int32, int32, int32, bool, unsigned int8)
IL_000b: call         bool [mscorlib]System.Decimal::op_Equality(valuetype [mscorlib]System.Decimal, valuetype [mscorlib]System.Decimal)
IL_0010: brfalse.s    IL_001c
        return double.MinValue;

Well, looks like it is true that the Decimal constructor is involved EVERY time you use constant like Decimal.MinValue! So next question arises - what is inside this constructor and is there any difference in using Decimal.MinValue and defining local field like:

static readonly decimal DecimalMinValue = Decimal.MinValue;

Well, the answer is - there is a difference if every penny counts in your case:

                     Method |        Mean |    StdDev |
--------------------------- |------------ |---------- |
 CompareWithDecimalMinValue | 178.4235 ns | 0.4395 ns |
   CompareWithLocalMinValue |  98.0991 ns | 2.2803 ns |

And the reason of this behavior is DecimalConstantAttribute which specified how to create a decimal every time you use one of the decimal's constants.

Pitfall of TaskCreationOptions.LongRunning

Recently I had conversation with colleague about using TaskCreationOptions.LongRunning for task which pings another service every N seconds till the end of application lifetime. Fortunately just a few days before I glanced over default task scheduler implementation in Reflector:

[SecurityCritical]
protected internal override void QueueTask(Task task)
{
    if ((task.Options & TaskCreationOptions.LongRunning) != TaskCreationOptions.None)
    {
        new Thread(s_longRunningThreadWork) { IsBackground = true }.Start(task);
    }
    else
    {
        bool forceGlobal = (task.Options & TaskCreationOptions.PreferFairness) != TaskCreationOptions.None;
        ThreadPool.UnsafeQueueCustomWorkItem(task, forceGlobal);
    }
}

The thing that catch my attention was the way dedicated thread is created. So I built small demo and it immediately confirmed that so called long-running task on default thread-pool scheduler uses dedicated thread only until it encounters first await. Afterwards it releases ‘dedicated’ thread and uses thread-pool threads to execute it’s continuation block, same thread-pool threads which are used by other tasks.

Declarative IoC container configuration

Nowadays it's hard to develop modern application without using IoC container. Absence of container leads to coupled codebase, expensive and hard to maintain unit tests (and they're not "unit" at all) and, as result, slow development speed and low quality software.

When using IoC container it’s important to ease the process of component and dependency registration and provide enough information to reader (for example during code reviews). While I’m big fan of Convention over Configuration, it doesn't always fit my needs. In applications I develop, I split components into at least 2 categories: components with lifetime bound to lifetime of application and the majority of components with lifetime per operation, such as HTTP request or service bus message. Cache, message bus are examples of former and command handlers, repositories, ApiControllers represent the latter.

Having scope per operation brings the following benefits:

• Atomicity and isolation. I usually treat this scope as unit of work and try to develop persistence so that result of operation is visible to outer world only after successful completion of operation.
• Performance and scalability. Each concurrently executing request will have it’s own copy of state, handlers, repositories and so on, avoiding excessive contention. Watch and read Pat Helland’s “Immutability Changes Everything”.
• Forget about exception safety. Writing exception safe code is hard! It is much easier to throw away scoped container with all it's instances than writing exception safe code.

So it is very important to understand lifetime of the component when reading its code. It helps find mistakes like usage of non thread-safe structures inside long-living class:

[ComponentRegistration(Scope = Lifetime.Application)]
internal sealed class MagicWand : IMagicWand
{
    private HashSet<string> _state = new HashSet<string>();

    // ...
}

As you may understand from this example, I use attribute to specify lifetime of the component. Unfortunately, there are some pitfalls when it comes to building container during application startup. When using Assembly.GetTypes() (or even GetExportedTypes()) on few specific (for example, by file mask) assemblies to discover components with registration attribute, you may occasionally load lot of assemblies which will significantly slow application startup. It may be fine for service running 24/7 that starts once in a while, but desktop application that consists of 3-5 assemblies of “business logic” and 30-50 of UI controls (like Infragistics) will suffer a lot.

This happens, for example, when assembly you’re scanning contains type which derives from type defined in another assembly. The problem can be minimized by moving components not related to DI to separate assembly, but I was looking for better solution. What if generate IoC specific component registrations at compile time?! Roslyn looks like the solution for this and many other problems I have, but I’ve got old proven hammer, which is Text Template Transformation Toolkit or T4.

Check out how I did it for StructureMap, meanwhile I’m going to do the same for my lovely Autofac.

Service API: Batching

What if you need to transfer large number of items to remote service? The goal is to transfer information as fast as we can. What would be the most efficient solution?

Well, the first naive approach obviously doesn't work:

AccountChange[] accountChanges = ...

foreach (var change in accountChanges)
{
    client.SendChange(change);
}

We accumulating network latencies and the overall timing is awful:

Naive
======================
Elapsed: 00:00:30.4708671
Rate: 3281.82324683501

Next popular solution is to pack all items into single batch:

AccountChange[] accountChanges = ...

client.SendChange(accountChanges);

Which gives much better timing:

OneBatch
======================
Elapsed: 00:00:02.4503693
Rate: 40810.1750213733

Could be even better? Let's analyze what happens behind the scene.
First, we need to serialize _all_ items, then we transfer bits to our remote service, then service should deserialize _all_ items, before it can start processing them.
All these steps run on single thread no matter how many idle cores you have. So using single batch we removed all network round-trips, but we added latency and it doesn't scale out.

Now the winning strategy is clear: pack items into multiple batches. Smaller batch sizes lower latency, but increase overall time. Choose carefully depending on your case.

MultiBatch
======================
Elapsed: 00:00:01.0337523
Rate: 96734.9721978853

The sample is available here

But what if last approach doesn't work for us? If, for example, sender is limited in number of connections it can use or it should send items in single atomic batch? Could we still perform better then one single batch approach?

To be continued...

Service API: Throughput and Latency

There are at least two things you should consider when designing scalable Service APIs - throughput and latency. Throughput is the number of request service can process per, for example, second. Latency is time to process single request. It is obvious, thank you, Cap! But how these two are related to scalability?

Well, throughput can be theoretically improved by scaling out your service. Add more instances and your throughput grows. I put "theoretically" because often your multiple instances end up waiting each other on some shared resource, database for example. That's why it is shouted on every corner that scalable architecture should be put into product from the beginning.

Latency is another story. You can't reduce latency by scaling out. Of course if your system is overloaded then latency goes high, you add instances and latency drops down. But you can't put it below some limit no matter how many instances you add. That's what I'm talking about!

So morale of the post: do not add latency "by design" - you can't mitigate this later on by scaling out.

But what about Service APIs? How can we add latency "by design" in Service APIs?
To be continued...

Setting WCF MaxBufferPoolSize quota can cause memory leak

If you use WCF and TransferMode.Buffered be careful with MaxBufferPoolSize setting. If non-zero value is used, then buffer manager won't release allocated memory. Which means if you set high value, let's say Int32.MaxValue, and you send or receive large message then this memory won't be reclaimed by GC.

Using 0 as MaxBufferPoolSize switches to GCBufferManager which simply allocates and releases memory every time buffer requested.

For more details look at System.ServiceModel.Channels.BufferManager internals.

IQueryable is root of many problems

It's often suggested to avoid unnecessary abstractions in your projects. Some respectful community members like to repeat it over and over again. But how to distinguish between required and unnecessary abstraction? What values can abstraction bring in?

Let's take a look at sample code which uses NHibernate's session to access data.

 internal sealed class TaskCoordinator
 {
  private readonly ISession _session;
 
  public TaskCoordinator(ISession session)
  {
   _session = session;
  }
 
  public void DoSomething()
  {
   /* ... */
 
   var tasks = GetRunningTasks(1);
 
   /* ... */
 
   foreach (var task in tasks)
   {
    var previousTask = GetPreviousTask(task);
 
    /* ... */
   }
  }
 
  private Task[] GetRunningTasks(int workflowId)
  {
   var utcNow = DateTime.UtcNow;
   return _session.Query<Task>()
    .Where(t => t.IsActive)
    .Where(t => t.WorkflowId == workflowId)
    .Where(t => t.StartsAt <= utcNow)
    .Where(t => t.ExpiresAt > utcNow)
    .ToArray();
  }
 
  private Task GetPreviousTask(Task task)
  {
   return _session.Query<Task>()
    .Where(t => t.WorkflowId == task.WorkflowId)
    .Where(t => t.StartsAt < task.StartsAt)
    .Where(t => t.ExpiresAt <= task.StartsAt)
    .FirstOrDefault();
  }
 }

Problem is: should we use session directly or should be hide it under another level of abstraction.
But before making any decision let me ask you a few questions:

1. Except type safety, what is the difference between these LINQ-queries and SQL statements used directly in code?
2. Without studying whole codebase, can you tell me what are use cases and what indices should be created in database to cover these use cases?
3. Can you estimate L2 cache hit ratio for your queries?
4. Can you easily (easily means isolated, without building two-page test setup) test buggy method GetPreviousTask?
5. Can you write unit (means without involving database) tests for TaskCoordinator?

These questions and your answers help you make informed decision and choose between previous implementation and introducing another level of abstraction:

 public interface ITaskRepository
 {
  Task[] GetRunningTasks(int workflowId);
  
  Task GetPreviousTask(Task task);
 }
 
 internal sealed class TaskCoordinator
 {
  private readonly ITaskRepository _repository;
 
  public TaskCoordinator(ITaskRepository repository)
  {
   _repository = repository;
  }
 
  public void DoSomething()
  {
   /* ... */
 
   var tasks = _repository.GetRunningTasks(1);
 
   /* ... */
 
   foreach (var task in tasks)
   {
    var previousTask = _repository.GetPreviousTask(task);
 
    /* ... */
   }
  }
 }

Actually title is provoking, it's not the IQueryable problem, it's a problem of any wide unbounded contract.

MongoDb vs. MS SQL: how to write to journal without additional seeks

In my previous post I figured out why single-threaded MongoDb benchmark is limited to 1000 inserts per second. But actually only SSD can reach this limit: 980 inserts per second on SSD and only 700 on HDD. Modified multi-threaded (8 threads) benchmark gives 7900 inserts on SSD and 5200 on HDD. Why there is such a big difference if journal is append-only storage. HDD should perform quite well in sequential write scenario. Can we close the gap?

If you read first article in the series you remember I was surprised that MS SQL does near the same number of writes as number of test iterations. But MongoDb doubles this numbers accessing not only journal file, but also NTFS metadata file.

Look how journal file is created:

The purpose of FILE_FLAG_NO_BUFFERING is obvious:

In these situations, caching can be turned off. This is done at the time the file is opened by passing FILE_FLAG_NO_BUFFERING as a value for the dwFlagsAndAttributes parameter of CreateFile. When caching is disabled, all read and write operations directly access the physical disk. However, the file metadata may still be cached. 

And what does FILE_FLAG_WRITE_THROUGH?

A write-through request via FILE_FLAG_WRITE_THROUGH also causes NTFS to flush any metadata changes, such as a time stamp update or a rename operation, that result from processing the request.

Windows provides this ability through write-through caching. A process enables write-through caching for a specific I/O operation by passing the FILE_FLAG_WRITE_THROUGH flag into its call to CreateFile. With write-through caching enabled, data is still written into the cache, but the cache manager writes the data immediately to disk rather than incurring a delay by using the lazy writer.  

In other words this flag a) flushes NTFS metadata b) keeps written data available in cache manager memory.
Do we really need to update file attributes on every write operation paying additional HDD seek?! Do we really need this data to be cached? I expect data is read from journal only during recovery.

I removed FILE_FLAG_WRITE_THROUGH and here are numbers: 7900 inserts per second on SSD and 7800 on HDD! Gap is closed!

I expect pre-allocating journal file may give additional improvement, but now our 1 ms introduced delay is real bottleneck and should be addressed first. May be after journal throughput is improved there is no need to keep default value for journalCommitInterval at 100 ms?

Let me submit issue to JIRA accompanied by pull request and see what 10gen folks will say about it.

UPDATED: BTW, after modification disk load drops from 55% to 7-10% during benchmark. As I already said this guy is our current bottleneck:

sleepmillis(oneThird);

UPDATED: [proof] after modification mongod.exe doesn't double number of writes:

Stay tuned!

MongoDb vs. MS SQL: journalCommitInterval problem

  In my previous article I got awful results for MongoDb and some "proofs" led me to wrong away. Actually not all of them were wrong, but before let's deal with simplier problem.

  My benchmark results were 29 inserts per second, 34 ms per insert. I thought it was due to HDD seeks, because MongoDb updates two files per write. But when I ran same test on SSD I got same results. Something blocks my requests! And here it is:

  I was surprised that write to journal is also delayed, looks like guys from 10gen try to group some disk operations together. The problem is that every write is delayed, unconditionally! I ran MongoDb without specifying journalCommitInterval, so in my case delay is oneThird = (100 / 3) + 1 = 34!!! My every insert operation is delayed by 34 ms! Yes, sir, my benchmark confirms it!

  After reading documentation I've found out that smallest valid journalCommitInterval is 2 which gives oneThird = (2 / 3) + 1 = 1 ms. I ran MongoDb with this param value expecting to get something around 1K inserts per second. But I got only 700 inserts per second. Not that much... Learned from my previous mistakes I ran same benchmark at my SSD and got around 980. Much closer to that I expected keeping in mind 1 ms penalty delay.

  There is something fishy why MS SQL doesn't update 'last write time' but MongoDb does...

UPDATED: how to write to journal without additional HDD seeks

MongoDb vs. MS SQL Server in 'durable insert' benchmark

  Recently I thought about append-only storage to store audit log. Seems like both MS SQL Server and MongoDb fit my needs, but I want to get some numbers. Here is my contest environment:

• Windows 7 Professional SP1 x64
• Intel Core i7-2600 @ 3.40 GHz
• SSD Corsair Force 3 (only for OS, all database files are on HDD)
• HDD Seagate ST320DM000 320GB @ 7200 rpm (rated as average access time 15.6 ms, 64 IOPS @ 4K block by HDD Tune)
• MS SQL 2008 R2 SP1
• MongoDb v2.4.3

The benchmark is very simple - insert small records as fast as possible. Code is written in C# and available here.

  First I ran my benchmark against MS SQL and got something around 2500 inserts/second. I do understand that it is just appending to transaction log file (.ldf) and I expected to get near same results for MongoDb. MongoDb was benchmarked in 'durable' mode which means with journal turned on (I'm not sure is it possible or not to turn off journal in recent versions). First results were surprising, at that moment I was sure I did something wrong - 29 inserts per second! Actually to get any results I had to reduce number of test iterations for MongoDb from 10000 to just 100, otherwise I couldn't wait till test completes.

Look at the times between writes during MS SQL benchmark:

And compare with MongoDb times:

Fractions of millisecond between MS SQL write operations and 34 ms between MongoDb writes! It took me a few hours to figure out what is going on, but I'm going to save your time. Did you notice I give detailed characteristics of my spin drive? 64 IOPS with 15.6 ms average access time and 29 inserts per second and 34 milliseconds between writes in benchmark...

  I used PerfView to verify my theory. Look at the disk activity while MS SQL performs 10K test iterations:

It does 10K+ writes (additional writes may be caused by test preparation phase) to a single file which is transaction log file. And here are MongoDb results for 100 iterations:

It does twice i/o write operations affecting two files! I supposed it updates some file metadata and I found "proof" quickly: MS SQL doesn't update last write time for InsertBenchmark_log.ldf file, but MongoDb does. That's why in latter scenario two files accessed. So I decided that MongoDb performance is penalized by HDD seek for every write operation.

  At the time of writing I understand I was totally blinded by my "proofs". Later I ran MongoDb benchmark on my SSD and got same results! Only 29 iterations per seconds! Obviously it is not related to HDD seek times...

  Let's summarize my findings before continue:

• MongoDb is very slow in my append-only durable storage benchmark
• It updates two files on the disk for every write operation
• But results are the same on SSD and HDD, that means problem is inside client bindings or mongo itself.

UPDATED: how journalCommitInterval affects benchmark

How to persist aggregate root. Part I

    Сохранение AR не такая уж и простая задача, как может показаться. На самом деле то как вы собираетесь хранить состояние (а может и не состояние) AR может сильно повлиять на всю архитектуру приложения.

    Рассмотрим первый, возможно наиболее часто используемый вариант - сохранение в реляционную базу данных. Можно сохранять сущности руками, а можно и с помощью ORM, в моем примере NHibernate, что мы и будем делать. А вот, кстати, и наши сущности:

    public class Order
    {
        public Order(int customerId)

        {
            CustomerId = customerId;

 
            OrderLines = new List<OrderLine>();
        }
 
        private Order()

        {

        }

 
        public int Id { get; private set; }

 
        public int Version { get; private set; }
 
        public int CustomerId { get; private set; }
 
        public double Total { get; private set; }
 
        public IList<OrderLine> OrderLines { get; private set; }
 
        public void BuyProduct(string productId, int quantity, double price)
        {

            OrderLines.Add(new OrderLine(this, productId, quantity, price));

 
            Total += quantity * price;
        }

    }
 
    public class OrderLine
    {
        public OrderLine(Order order, string productId, int quantity, double price)

        {

            Order = order;

            ProductId = productId;

            Quantity = quantity;

            Price = price;

        }

 
        private OrderLine()

        {

        }

 
        public int Id { get; private set; }

 
        public Order Order { get; private set; }

 
        public string ProductId { get; private set; }

 
        public int Quantity { get; private set; }

 
        public double Price { get; private set; }

    }

Весь код примера доступен здесь.

    Так как многие ORM поддерживают замечательную, на первый взгляд, возможность отложенной загрузки дочерней коллекции (lazy-load), то мы будем считать свойство Total на самом Order, например для случаев когда нам нужен для вывода только Total и не нужны данные с OrderItems. В случае "развесистых" AR с большим количеством вложенных коллекций на нескольких уровнях использование lazy-load с точки зрения производительности просто напрашивается.

    Тут то и спрятан подвох! Но прежде немного теории. AR это прежде всего consistency boundary. Именно исходя из соображений целостности нужно решать объединять ли 100500 сущностей под одним корнем или делать 100500 независимых AR или какое то промежуточное решение. Тема определения границ AR это отдельная сложная тема, достаточно хорошо изложена например здесь: Effective Aggregate Design. Таким образом мы должны и сохранять и загружать AR целостно, но так ли это в нашем случае? Нет, не так!

    Загрузим order, выведем в консоль значение поля Total, а затем пересчитаем это значение уже по коллекции объектов OrderLines. Но предположим что между этими двумя действиями кто то обновил order. Однако нас это затронуть не должно, у нас же consistency boundary, не так ли?

       using (var session = _sessionFactory.OpenSession())

       using (var transaction = session.BeginTransaction())
       {
           var order = session.Get<Order>(orderId);
 

           Console.WriteLine(order.Total);
 
           ConcurrentWriter(orderId);

 
           // Here collection is actually loaded

           Console.WriteLine(order.OrderLines.Sum(oi => oi.Quantity * oi.Price));

 
           transaction.Commit();

       }

       private void ConcurrentWriter(int orderId)
       {
           var thread = new Thread(() =>
           {

               using (var session = _sessionFactory.OpenSession())

               using (var transaction = session.BeginTransaction())

               {
                   var order = session.Get<Order>(orderId);
 
                   order.BuyProduct("Whiskey", 1, 10.12);

 
                   transaction.Commit();

               }

           });

 
           thread.Start();

 
           // give it chance to complete

           thread.Join(1000);

       }

    В результате мы видим в консоли два разных числа! При отложенной загрузке коллекции в мы загрузили не только уже имеющиеся OrderLines, но и еще то что добавил наш concurrent writer. А как же целостность данных?!

    Есть два известных мне способа "вернуть" целостность. Первый способ - открывать транзакцию с уровнем изоляции RepeatableRead. Но тогда нужно быть готовым к исключениям в точке обращения к коллекции OrderLines, потому что нашу транзакцию будут выбирать жертвой разрешения deadlock'а. Второй способ - отказаться от lazy и загружать весь AR целиком.

    Итог: использование реляционной базы данных для хранения AR это удобный в плане использования (ORM с их "плюшками", развитый инструментарий для самих СУБД) и быстроты реализации способ, имеющий "некоторые" проблемы с целостностью, если о них не задумываться. Для нагруженных решений и "развесистых" AR к недостаткам можно так же отнести необходимость нескольких дисковых операций для загрузки одного AR.

    В следующий раз (если он будет) попробуем хранить наш AR в документной базе.

Console.ReadKey blocks non-initialized output

Будьте осторожны, данный пример не будет работать, пока закомментирована строка (или пока не сойдутся тайминги J)

         class Program

         {

                 static void Main(string[] args)

                 {

                          ThreadPool.QueueUserWorkItem(_ =>

                          {

                                   Console.WriteLine("Hello, console!");

                          });

                          //Console.WriteLine("Press any key to exit...");

                          Console.ReadKey();

                 }

         }

Updated: исправлено в .NET 4.5 путем использования разных sync-object'ов в методах WriteLine и ReadKey