LINQ to Object: Skip() performance

Assume You have a List of 5000000 items. You want to do some paging so You need e.g. 100 items from offset 300000. Check these two implementations:

        public List<T> GetItems1<T>(List<T> source, int offset, int count)
        {
            List<T> ret = new List<T>();

            for (int i = offset; i < offset + count; i++)
            {
                ret.Add(source[i]);
            }

            return ret;
        }
        public List<T> GetItems2<T>(List<T> source, int offset, int count)
        {
            List<T> ret = source.Skip(offset).Take(count).ToList();

            return ret;
        }

What do You think, which performs better? You may say the second one of course. The first one indexes the source list and calls Add() method count times. The second simply enumerates once till offset then returns count items as a new List with the possibility of internal item addition optimalization. At least that were what I think.

But I was wrong!

The second implementation always slower. The magnitude depends on the offset value but it is always slower!

offset GetItems1 GetItems2
0 43 65
10000 59 729
100000 44 5162
1000000 42 52057
3000000 44 147608

The reason is inside the implemetation details of List and IEnumerable.Skip(). The first one knows where to find the nth item, the second one should enumerate to it. The conclusion as one of my colleagues pointed out: use as specialized tools as You can.

The code which I used for result above:

    [TestClass]
    public class Test
    {
        private int cycles = 100;
        private int offset = 0;
        private int count = 100;

        [TestMethod]
        public void perftest1()
        {
            var l = GetTestData();

            var sw = new Stopwatch();

            double r = 0;
            for (int i = 0; i < cycles; i++)
            {
                sw.Reset();
                sw.Start();
                var result = GetItems1(l, offset, count);
                sw.Stop();
                r += sw.ElapsedTicks;
            }

            Assert.Fail((r / cycles).ToString());
        }


        [TestMethod]
        public void perftest2()
        {
            var l = GetTestData();

            var sw = new Stopwatch();

            double r = 0;
            for (int i = 0; i < cycles; i++)
            {
                sw.Reset();
                sw.Start();
                var result = GetItems2(l, offset, count);
                sw.Stop();
                r += sw.ElapsedTicks;
            }

            Assert.Fail((r / cycles).ToString());
        }


        public List<T> GetItems1<T>(List<T> source, int offset, int count)
        {
            ...
        }


        public List<T> GetItems2<T>(List<T> source, int offset, int count)
        {
            ...
        }

        private List<int> GetTestData()
        {
            List<int> ret = new List<int>();

            for (int i = 0; i < 5000000; i++)
            {
                ret.Add(i);
            }

            return ret;
        }
    }

Redmine, awesome_nested_set, Issue.rebuild!, lock_version and ActiveRecord::StaleObjectError

Today I got a task to restructure about 400 issues in our Redmine instance. I should create a hierarchy and reparent existing issues under new ones. All the issues have a lot of related ones. But when I tried to modify parent issue field I got “Parent task is invalid” message after saving. I found pages about this problem and the only solution: remove relations, reparent the issue and recreate relations… 400 issues with a lot of relations… No.

I was updating the database instead: set the issues.parent_id to the appropriate new value. Naturally after that two things went wrong: some aggregated values e.g. estimated_hours and the hierarchy view, because of awesome_nested_set gem was used which stores hierarchy speedup values in rgt and lft fields of that table. First of all I fired out these fields in database (11840 was my main root issue):

UPDATE issues 
SET 
  rgt=null, 
  lft=null 
WHERE parent_id=11840

Than the following command needs to run in redmine root dir:

RAILS_ENV=production ruby script/rails runner Issue.rebuild!

After a long run, error occured:

ActiveRecord::StaleObjectError: ActiveRecord::StaleObjectError
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/locking/optimistic.rb:90:in `update'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/attribute_methods/dirty.rb:74:in `update'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/timestamp.rb:71:in `update'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/callbacks.rb:272:in `update'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:403:in `_run__1446014408__update__4__callbacks'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:405:in `send'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:405:in `__run_callback'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:385:in `_run_update_callbacks'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:81:in `send'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:81:in `run_callbacks'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/callbacks.rb:272:in `update'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/persistence.rb:348:in `create_or_update'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/callbacks.rb:264:in `create_or_update'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:590:in `_run__1446014408__save__4__callbacks'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:405:in `send'
from /var/lib/gems/1.8/gems/activesupport-3.2.19/lib/active_support/callbacks.rb:405:in `__run_callback'
... 18 levels...
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/relation/delegation.rb:6:in `__send__'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/relation/delegation.rb:6:in `each'
from /var/lib/gems/1.8/gems/awesome_nested_set-2.1.6/lib/awesome_nested_set/awesome_nested_set.rb:203:in `rebuild!'
from /var/lib/gems/1.8/gems/awesome_nested_set-2.1.6/lib/awesome_nested_set/awesome_nested_set.rb:213:in `call'
from /var/lib/gems/1.8/gems/awesome_nested_set-2.1.6/lib/awesome_nested_set/awesome_nested_set.rb:213:in `rebuild!'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/relation/delegation.rb:6:in `each'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/relation/delegation.rb:6:in `__send__'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/relation/delegation.rb:6:in `each'
from /var/lib/gems/1.8/gems/awesome_nested_set-2.1.6/lib/awesome_nested_set/awesome_nested_set.rb:210:in `rebuild!'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/scoping/default.rb:41:in `unscoped'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/relation.rb:241:in `scoping'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/scoping.rb:98:in `with_scope'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/relation.rb:241:in `scoping'
from /var/lib/gems/1.8/gems/activerecord-3.2.19/lib/active_record/scoping/default.rb:41:in `unscoped'
from /var/lib/gems/1.8/gems/awesome_nested_set-2.1.6/lib/awesome_nested_set/awesome_nested_set.rb:185:in `rebuild!'

After some googling I modified my command:

RAILS_ENV=production ruby script/rails console
irb(main):003:0> Issue.rebuild!

Same error, but instead of long nothinghappens block a lot of SQL statements were scrolling on my screen which indicate what is happening. Here are the interesting last ones:

   ...
   (0.2ms)  UPDATE `issues` SET `lock_version` = 53, `estimated_hours` = NULL, `updated_on` = '2014-11-25 12:29:00' WHERE (`issues`.`id` = 12751 AND `issues`.`lock_version` = 52)
   ...
   (0.1ms)  UPDATE `issues` SET `lock_version` = 53, `lft` = 388, `updated_on` = '2014-11-25 12:29:02', `rgt` = 419 WHERE (`issues`.`id` = 12751 AND `issues`.`lock_version` = 52)
   (0.1ms)  ROLLBACK

So it seems that the ActiveRecord contexts used by Redmine and nested set’s rebuilder differ: Redmine collects info about aggregated values and updates the record and the other one wants to update the hierarchy fields. Optimistic concurrency fails here.

My solution: stop Redmine and run this command in ruby console before Issure.rebuild!:

irb(main):003:0> ActiveRecord::Base.lock_optimistically = false

This will turn off concurrency locking in Your process so script will run. It seems the fields updated are not overlapped so running without locking is not a problem.

NuGet silently overwrites some files

On package update NuGet silently overwrites some files. Consider You have something like this in Your nuspec file:

    <file src="MyPackage\App.xaml.cs.pp" target="content\App.xaml.cs.pp" />
    <file src="MyPackage\App.xaml.pp" target="content\App.xaml.pp" />

Consider the user changed App.xaml.cs but didnt changed App.xaml. In this case when user updates Your package the App.xaml.cs file will be silently overwritten!

After a short search session on the net I found a same problem without any solutions. So here is mine install.ps1 with a workaround:

param($installPath, $toolsPath, $package, $project)
$appXamlFileName = $item.Properties.Item("FullPath").Value
Add-Content $appXamlFileName "<!-- this comment added by nuget package's install script to modify this file to workaround bug which overwrites App.xaml.cs file on next package update if this one isnt modified -->" 

On update NuGet will detect that App.xaml changed and will skip it AND App.xaml.cs file too!

Howto debug NuGet package’s install.ps1

While searching for solution around install.ps1 I quickly realised that the edit file -> create package -> publish -> update target cycle isnt very handly. There are step-by-step instructions on the net how to debug install.ps1, but when I tried to use them I run into new problems about incompatibility between my packages and nuget powershell cmdlets.

So I used a poor mans solution instead:

  1. Add some dummy content file to You package.
  2. Install Your package.
  3. Edit that dummy file (each next package install cycle will ask You about overwriting and that is what we need!)
  4. Open Package Manager Console (PMC) in VS
  5. Execute “Set-PsDebug -trace 2”. It helps You later.
  6. Open the install.ps1 in some editor (I used PowerShell ISE) from target project’s packages directory.
  7. In PMC run “update-package _yourpackagename_ -reinstall”. The installation will stop because file overwriting.
  8. In the PS editor edit and save install.ps1, but dont close it.
  9. Back to PMC and answer the overwriting question (with ‘L’ or ‘N’ to keep the modified file).
  10. See the results.
  11. Go to 7.

Not to smart, didnt uses tools and libraries but it works in acceptable cycle time.

Setting project item’s BuildAction from NuGet package

I created a package for internal use which has an App.xaml file in content. Naturally I would like to find it in target project with BuildAction set to “ApplicationDefinition”, but Visual Studio treats it as “Page”, because it is the xaml extension default.

I found a hopeful solution here promising a fast track. My first version of install.ps1 was this:

param($installPath, $toolsPath, $package, $project)

$item = $project.ProjectItems.Item("App.xaml")
$item.Properties.Item("BuildAction").Value = ???

The problem raised when I didnt found the “ApplicationDefinition” value in prjBuildAction enumeration on MSDN…

I found some similar examples on the net. Some of them have comment with a question: what to do if someone wants to set a value not in the enumeration? Neither of those comments has an answer which distressed me a bit.

Here I found a clue so I tried to enumerate those undefined prjBuildAction values with this code:

param($installPath, $toolsPath, $package, $project)

Add-Type -AssemblyName 'Microsoft.Build.Engine'
$msbuildproject = new-object Microsoft.Build.BuildEngine.Project
$msbuildproject.Load($project.FullName)

[System.Collections.ArrayList]$buildActions = "None", "Compile", "Content","EmbeddedResource"

$msbuildproject.ItemGroups | Where-Object { $_.Name -eq "AvailableItemName" } | Select-Object -Property "Include" | ForEach-Object {
  $act = $_
  $buildActions.Add($act)
}

$item = $project.ProjectItems.Item("App.xaml")
$item.Properties.Item("BuildAction").Value = [int]$buildActions.IndexOf("ApplicationDefinition")

Dont know why but the enumeration of the ItemGroups didnt worked. When I did a

Write-Host ($msbuildproject.ItemGroups | Format-List | Out-String)

it showed me a nice list of BuildItems, but when I run with a Where-Object against it I found nothing. The problem should be in PS syntax or the object instances. I dont maintain my PS knowledge which is based on my .Net and Linux scripting practice combined with looking for snippets from code examples around. I simply dont want to go more deeply into it because I feel PS is something “created” and not “born” if You understand what I mean.

I rewrite the script to get the available BuildAction values as follows:

...
$msbuildproject.ItemGroups | ForEach-Object {   
    $ig = $_
    @($ig.GetEnumerator()) | ForEach-Object { 
        $i = $_
        if ($i.Name -eq "AvailableItemName")
        {
            $buildActions.Add($i.Include);
        }
    }
...

And voilá, I got a nice list of values in $buildActions:

None
Compile
Content
EmbeddedResource
CodeAnalysisDictionary
ApplicationDefinition
Page
Resource
SplashScreen
DesignData
DesignDataWithDesignTimeCreatableTypes
EntityDeploy
XamlAppDef

My $idx becomes: 5, I checked value in Solution Explorer… and found “CodeAnalysisDictionary” there. No problem, it should be some 0/1 based indexing problem, lets set $idx+1, run, check, value okay! Lets try something other just for to be sure… again bad value! Unfortunately it seems the order of values collected with this algorithm isnt good as the link I mentioned above imply. Back to the start line.

While looking for solutions I somewhere found something else about ProjectItem’s “ItemType” property, so I tried to play with it. And suddently the Sun raised, the sky becomes blue, etc.:

param($installPath, $toolsPath, $package, $project)
$item = $project.ProjectItems.Item("App.xaml")
$item.Properties.Item("ItemType").Value = "ApplicationDefinition"

So simple and it works!

Error 286 The “BuildShadowTask” task failed unexpectedly. System.NullReferenceException: Object reference not set to an instance of an object.

One of my colleagues meets the message above during building solution which has a test project. Because the solution he found isnt trivial I decided to distribute it here too:

Refresh accessors in Test References or add if some of them are missing.

Using accessors is obsolete now, but if You have an ancent project with them which wont compile with this error I hope You find useful this info.

Assembly generation failed — Referenced assembly ‘…’ does not have a strong name

A lot of useful nuget packages can be found around. But some of them are useless when You try to reference them from an assembly which should be signed. During compilation you will get the error message:

Assembly generation failed — Referenced assembly ‘…’ does not have a strong name

How to add strong name to a third party assembly?

I used to download the original sources and recompile them with assembly signing. But last time I simply neither cannot find the original sources nor the original download location of the used version of binaries. I tried to disassemble the dll but without success: ilspy and reflector generated uncompilable code 🙁

But while googleing I found a simple 4 step solution here:

Step 1 : Run visual studio command prompt and go to directory where your DLL located.

For Example my DLL located in D:/hiren/Test.dll

Step 2 : Now create il file using below command.

D:/hiren> ildasm /all /out=Test.il Test.dll
(this command generate code library)

Step 3 : Generate new Key for sign your project.

D:/hiren> sn -k mykey.snk

Step 4 : Now sign your library using ilasm command.

D:/hiren> ilasm /dll /key=mykey.snk Test.il

Nice, eh?

Naturally You should repeat these steps each time when the referenced nugets are updated,
but this way is much easier than the download-actual-sources-and-recompile one…