Tuesday, April 28, 2015

Git gc out of memory and repository slowness - tip for solution

On our project we are using a git repository with current size around 9GB. At some point we started facing issues with slowing the access to repository down (many operations took several minutes to complete).

When trying to research the issue we tracked this to behavior of git gc being executed automatically in background without ever actually finishing the job. The error which appeared when git gc was executed manually was related to out of memory.

We temporarily disabled automatic git garbage collect which solved the slowness but it was far from ideal solution.

In meantime I was also playing with various memory allocation and pack size settings on my local clone but all those just made things worse.

At some point I realized that for historical reasons we have in the repository also bunch of binary files like zip archives, database backups, nuget packages etc. which raised a suspicion.

Researching it more I found following solution which helps exclude those huge binary files from packing.

Create or edit file .git/info/attributes and put following content to it (one line per excluded file type):

*.zip -delta
*.bak -delta

Hope this helps :-)

Monday, March 16, 2015

Introducing .NET C# Inversion Of Control and Microsoft Unity Hands-On Lab

Introduction

During past few months we introduced and heavily extended usage of Microsoft Unity IoC container in our code base as a part of the effort to make the code more loosely coupled.

As a result of those changes we now even more than before also rely on Inversion Of Control or more specifically Dependency Injection.

Thus both Microsoft Unity and IoC/DI are now crucial part of our toolbox. In order to bring everybody on our team up-to-speed as well as to have training material for newbies we decided to create a simple training material which should help us.

After brief discussion within the team we agreed that the best way how to handle it would be to:

  • Collect some solid resources describing the IoC/DI.
    • Martin Fowler is obviously first choice - though differences between IoC and DI are better explained in different resources :-).
  • Provide hands-on lab project which will cover all the specifics for:
    • Inversion Of Control/Dependency injection.
    • Microsoft Unity Container.
    • Will serve as a self-training material.
  • We will publish it on Github under MIT license.

Target audience

.NET Software developers/engineers and architects who:

  • Would and are willing to learn about IoC/DI.
  • Are familiar IoC/DI but would learn about Microsoft Unity IoC container.
  • Would learn about possible challenges which usage of the MS Unity can bring.

Training materials

With my colleagues we prepared set of projects which allows everybody to play with all the stuff on reasonably sized projects.

Brief introduction can be found here.

Github projects

If you are either familiar with Github or if you would use this as an opportunity to learn more about it you can just fork/clone repositories below.

Direct access

In the case that you do not like Git/Github you can use direct links below to get the latest version of training projects as well as sample solutions in form of ZIP packages:

Contributions

If you will find something which needs to be fixed or if you have some interesting sample task just send it as a Github pull request - we accept contributions under MIT license.

Thanks to everybody who already contributed with his time either in form of code or even advice :-).

Tuesday, March 10, 2015

Debugging T-SQL Stored Procedure Invoked From NUnit Tests In Visual Studio 2013 Debugger

Recently I had to write quite a few interesting stored procedures for MSSQL server which are covered by unit tests invoked as a part of continuous integration build in Team City.

Setting up the data and parameters for stored procedure takes some time and there are many scenarios thus I started looking for a ways:

  • How to debug stored procedures using the existing infrastructure without necessity to extract everything out and use separated debugger in the SQL Server Management Studio.
  • How to stub some of the data so the complex parts of queries can be easily verified.

In the end I got working debugging with following setup:

  • Stored procedures written in T-SQL for MSSQL.
  • Each stored procedure is covered by unit tests written in NUnit.
    • Thanks to tip of my colleague MSSQL guru Lubos I was able quickly setup SQL Server snapshots to be able revert the database quickly to its initial state.
    • Lubos also proposed very simple way on how to stub some data in procedures.
  • In order to be able quickly check what is going on inside the stored procedure use the Visual Studio 2013 debugger including the ability to step into the stored procedure.

Using stubs for data used inside stored procedures

  • Motivation here is that it is not always simple enough or even practical setup all the required data directly in the database.
    • Downside obviously is that since you are about to alter the stored procedure you have to be very careful.
  • My colleague proposed a very simple way for this purpose which seems to work:
    • Before running tests take a database snapshot so you can easily revert back.
    • Inside procedures use some markers which can be quickly identified and the content between them can be replaced - for example:
[Stub1]
SELECT * FROM [MyInvoices]
[#Stub1]
  • Next before you execute the stored procedure you fetch its source and replace the code
    between markers with select from data stub (for example temporary table):
[Stub1]
SELECT * FROM #MyInvoices
[#Stub1] 
  • Before you exercise the stored procedure you simply populate content of #MyInvoices temporary table and run it.

How to enable T-SQL debugging in Visual Studio 2013

This was the most tricky part of the whole procedure and it may be specific to my setup (MSSQL 2008 R2, VS2013).

  • As a prerequisite the Application debugging and SQL/CLR debugging must be enabled for the SQL Server in the SQL Server Object Explorer.
  • There are two ways how to get to SQL Server Object Explorer:
    • Directly open SQL Server Object Explorer via Visual Studio menu VIEW:
      Open SQL Server Object Explorer in VS 2013 menu
    • Alternatively use Server Explorer:
      • Firstly add a connection to your database
      • Then right click using mouse on registered database and select Browse in SQL Server Object Explorer:
        Open SQL Server Object Explorer from Server Explorer in VS2013
  • Once you get into the SQL Server Object Explorer enable both debugging options as visible on picture below:
    Enable debugging in SQL Server Object Explorer from VS2013

Running tests

  • Running tests just for verification purposes is very simple and basically any NUnit runner can be used.
  • In our case for the standard purpose serves very well Jetbrains Resharper.

Debugging tests

  • Unfortunately this didn’t work with built-in R# test runner.
  • Instead I use NUnit-x86.exe runner (I simply needed to force the process bitness to 32bits but I suppose that NUnit.exe will work as well):
    • Load test assembly into NUnit runner.
    • Attach Visual Studio 2013 debugger to running process.
    • Important part here is to have enabled both - Managed code and T-SQL code debugging prior to attaching to he NUnit-x86.exe process:
      Attaching to NUnit with enabled T-SQL debugging
    • Now set a .breakpoint in .NET code just around the code which is responsible for invocation of the stored procedure you are interested in, for example SqlDbCommand.Execute().
    • Run unit test from the NUnit runner and have it hit the breakpoint in Visual Studio.
    • Now from the _SQL Server Object Explorer open the body of stored procedure (just double-click on it),
    • Set a breakpoint inside the procedure.
    • And step thru the .NET code which is about to invoke the procedure.
    • If everything works well for you you are now inside the stored procedure and you can debug it.

Watching data inside the stored procedure

  • You can easily watch content of any variable inside the stored procedure.
  • I found very simple trick which can be used to watch also content of temporary tables and table variables.

    • At the place you would check the content add following statement (obviously adjusted for correct table/variable name):
    DECLARE @v XML = (SELECT * FROM #Parameters FOR XML AUTO, ROOT('MyRoot'))
    • Once you will hit the statement in the debugger you can easily watch the content of @v and visualize it for example via XML Visualizer.

Wednesday, January 7, 2015

How to find (and fix) files with missing CR (0xd) in Visual Studio

Motivation

On our project the primary source control is TFS. For larger features we have recently started using GIT repository with master branch automatically synchronized from TFS.
When resolving merge conflicts it happens from time to time that line ends get somehow corrupted (there are plenty of settings in GIT client and related diff/merge tools) which results into .cs files which have both/mixed - CR+LF and LF line endings.

How to find files with missing CR (0xd)

It is pretty simple to identify all the corrupted files using a regular expression in the Visual Studio ‘Find in Files’ dialog:

^(.*)[^\r]\n$

Dialog example:

visual_studio_search_for_files_containing_lf.png

How to fix files

Files can be easily fixed by reformatting:

  • Just press CTRL+A to select all the text inside the file
  • Then press CTRL+KF to make it formatted
  • After fixing the issues you can easily verify files using the above search pattern
    • It happens from time to time that the last empty line in the file is not corrected. The easiest way how to fix it is to remove it and perhaps add it again.

Thursday, December 18, 2014

Visual Studio 2012 debugger does not break after attaching to C#/.NET process

I had from time to time issue debug C#/.NET applications in Visual Studio 2012 after attaching Visual Studio 2012 debugger to a process.

Symptoms were that the debugger attached to the process but neither ‘Break All’ worked. The same applied for any preset breakpoint.

For some time I thought that Visual Studio installation for somehow corrupted on my system but since I was always able to workaround it via Debug.Assert() or Debugger.Break() calls put directly into code I had never motivation to really look for a solution nor reinstall the Visual Studio.

Today I really wanted to attach to a process to see what is going on inside and the issue happened again.

After a bit of playing I realized that in the case that debugger works after attaching correctly the ‘Attach to Process’ Visual Studio dialog looks like this (see ‘Attach to’ field):
visual_studio_debugger_does_recognize_process_type.png

For my process it didn’t work this time and ‘Attach to Process’ dialog looked like this (again see ‘Attach to’ field):
visual_studio_debugger_does_not_recognize_process_type.png

Apparently Visual Studio in some cases does not properly detect the type of the process and does not use correct debugger settings.

In order to solve my issue I finally found the ‘Select…’ button following ‘Attach to’ field where you can disable automatic detection of the process type and manually select a different one.

Zmrzka na hrázi

After selecting ‘Manager (v4.5, 4.0)’ and attaching debugger to process again everything worked well.