LINQ to Objects in Depth (1) Local Sequential Query

[LINQ via C# series]#

[LINQ to Objects in Depth series]#

Latest version: https://weblogs.asp.net/dixin/linq-to-objects-local-sequential-query#

LINQ to Objects queries sequences of .NET objects in local memory of current .NET application or service. Its data source and the queries are represented by IEnumerable.

Iteration pattern and foreach statement#

C#/.NET follows iterator pattern to define sequence of values, and implement sequential access to the values in sequence in a unified approach. Iteration pattern consists of a sequence (also called container of items, or aggregate of elements) and an iterator:

1
internal abstract class Sequence
2
{
3
    public abstract Iterator GetEnumerator(); // Must be public.
4
}
5

6
internal abstract class Iterator
7
{
8
    public abstract bool MoveNext(); // Must be public.
9

10
    public abstract object Current { get; } // Must be public.
11
}

And their generic version is:

1
internal abstract class GenericSequence<T>
2
{
3
    public abstract GenericIterator<T> GetEnumerator(); // Must be public.
4
}
5

6
internal abstract class GenericIterator<T>
7
{
8
    public abstract bool MoveNext(); // Must be public.
9

10
    public abstract T Current { get; } // Must be public.
11
}

These types and members demonstrate the minimum requirements for iteration pattern:

The sequence is the container of sequential values, it has a GetEnumerator factory method returning an iterator
Iterator traverses all values in the sequence. Its MoveNext method returns a bool value to indicate whether there is still a next value that can be pulled. If true is returned, its Current property can be called to pull that value.

Then the values in above non-generic and generic sequences can be access with C# foreach statement:

1
internal static partial class IteratorPattern
2
{
3
    internal static void ForEach<T>(Sequence sequence, Action<T> processNext)
4
    {
5
        foreach (T value in sequence)
6
        {
7
            processNext(value);
8
        }
9
    }
10

11
    internal static void ForEach<T>(GenericSequence<T> sequence, Action<T> processNext)
12
    {
13
        foreach (T value in sequence)
14
        {
15
            processNext(value);
16
        }
17
    }
18
}

The above foreach loops are compiled to while loops:

1
internal static void CompiledForEach<T>(Sequence sequence, Action<T> processNext)
2
{
3
    Iterator iterator = sequence.GetEnumerator();
4
    try
5
    {
6
        while (iterator.MoveNext())
7
        {
8
            T value = (T)iterator.Current;
9
            processNext(value);
10
        }
11
    }
12
    finally
13
    {
14
        (iterator as IDisposable)?.Dispose();
15
    }
16
}
17

18
internal static void CompiledForEach<T>(GenericSequence<T> sequence, Action<T> processNext)
19
{
20
    GenericIterator<T> iterator = sequence.GetEnumerator();
21
    try
22
    {
23
        while (iterator.MoveNext())
24
        {
25
            T value = iterator.Current;
26
            processNext(value);
27
        }
28
    }
29
    finally
30
    {
31
        (iterator as IDisposable)?.Dispose();
32
    }
33
}

So the foreach loops is syntactic sugar to make above imperative control flow declarative. The generic version is always preferred, becuase the non-generic Iterator’s Current property returns object, it has to be explicitly casted to the expected type specified in the foreach statement, which could be a chance of failure.

To demonstrate the iterator pattern implementation, a sequence of values can be stored with a singly linked list, with one value in each node:

1
internal class SinglyLinkedListNode<T>
2
{
3
    internal SinglyLinkedListNode(T value, SinglyLinkedListNode<T> next = null)
4
    {
5
        this.Value = value;
6
        this.Next = next;
7
    }
8

9
    public T Value { get; }
10

11
    public SinglyLinkedListNode<T> Next { get; }
12
}

Then iterator can be implemented to traverse along the linked list nodes. Iterator pattern is imperative, and iterator can change its state during the iteration. When MoveNext is called and returns true, it have Current to return a different next value:

1
internal class LinkedListIterator<T> : GenericIterator<T>
2
{
3
    private SinglyLinkedListNode<T> node; // State.
4

5
    internal LinkedListIterator(SinglyLinkedListNode<T> head) =>
6
        this.node = new SinglyLinkedListNode<T>(default, head);
7

8
    public override bool MoveNext()
9
    {
10
        if (this.node.Next != null)
11
        {
12
            this.node = this.node.Next; // State change.
13
            return true;
14
        }
15
        return false;
16
    }
17

18
    public override T Current => this.node.Value;
19
}

And the sequence can be simply implemented as a iterator factory:

1
internal class LinkedListSequence<T> : GenericSequence<T>
2
{
3
    private readonly SinglyLinkedListNode<T> head;
4

5
    internal LinkedListSequence(SinglyLinkedListNode<T> head) => this.head = head;
6

7
    public override GenericIterator<T> GetEnumerator() => new LinkedListIterator<T>(this.head);
8
}

Now the values in the linked list sequence can be sequentially pulled with the foreach syntactic sugar:

1
internal static void ForEach(SinglyLinkedListNode<int> head)
2
{
3
    LinkedListSequence<int> sequence = new LinkedListSequence<int>(head);
4
    foreach (int value in sequence)
5
    {
6
        value.WriteLine();
7
    }
8
}

A general implementation of iterator pattern will be discussed later in this chapter.

IEnumerable and IEnumerator#

Initially, .NET Framework 1.0 provides IEnumerable and IEnumerator interfaces to represent iterator pattern:

1
namespace System.Collections
2
{
3
    public interface IEnumerable // Sequence.
4
    {
5
        IEnumerator GetEnumerator();
6
    }
7

8
    public interface IEnumerator // Iterator.
9
    {
10
        object Current { get; }
11

12
        bool MoveNext();
13

14
        void Reset(); // For COM interoperability.
15
    }
16
}

Many sequence and collection types implement IEnumerable so that they can be used with foreach, like ArrayList, Queue, Stack, etc. Then .NET Framework 2.0 supports generics, where the generic version, IEnumerable and IEnumerator, are provided:

1
namespace System.Collections.Generic
2
{
3
    public interface IEnumerable<T> : IEnumerable // Sequence.
4
    {
5
        IEnumerator<T> GetEnumerator();
6
    }
7

8
    public interface IEnumerator<T> : IDisposable, IEnumerator // Iterator.
9
    {
10
        T Current { get; }
11
    }
12
}

Since then the sequence and collection types are provided with IEnumerable implemented by default, like List, Queue, Stack, etc.

Later, .NET Framework 4.0 introduces covariance and contravariance for generic interface. As discussed in the Functional Programming chapter, T is covariant for both IEnumerable and IEnumerable. So their definitions are updated to:

1
namespace System.Collections.Generic
2
{
3
    public interface IEnumerable<out T> : IEnumerable // Sequence.
4
    {
5
        IEnumerator<T> GetEnumerator();
6
    }
7

8
    public interface IEnumerator<out T> : IDisposable, IEnumerator // Iterator.
9
    {
10
        T Current { get; }
11
    }
12
}

EnumerableAssert utility#

In Microsoft’s unit test framework MSTest, there are built-in assertion utility types provided:

Assert to check general conditions, providing methods like IsTrue, IsNotNull, AreEqual, etc.
StringAssert to check conditions for string, providing methods like Contains, StartsWith, EndsWith, etc.
CollectionAssert to check conditions for ICollection, providing methods like AllItemsAreInstancesOfType, AllItemsAreNotNull, IsSubsetOf, etc.

To demonstrate how to consume IEnumerator and IEnumerator with the iterator pattern, an EnumerableAssert utility type can be defined to check conditions for sequence. For example, the following assertion methods check whether the specified sequence is not null and is empty/is not null and is not empty/is null or is empty:

1
public static partial class EnumerableAssert
2
{
3
    public static void IsEmpty<T>(IEnumerable<T> actual, string message = null, params object[] parameters)
4
    {
5
        Assert.IsNotNull(actual, message, parameters);
6
        using (IEnumerator<T> iterator = actual.GetEnumerator())
7
        {
8
            Assert.IsFalse(iterator.MoveNext(), message, parameters);
9
        }
10
    }
11

12
    public static void Any<T>(IEnumerable<T> actual, string message = null, params object[] parameters)
13
    {
14
        Assert.IsNotNull(actual, message, parameters);
15
        using (IEnumerator<T> iterator = actual.GetEnumerator())
16
        {
17
            Assert.IsTrue(iterator.MoveNext(), message, parameters);
18
        }
19
    }
20

21
    public static void IsNullOrEmpty<T>(
22
        IEnumerable<T> actual, string message = null, params object[] parameters)
23
    {
24
        using (IEnumerator<T> iterator = actual?.GetEnumerator())
25
        {
26
            Assert.IsFalse(iterator?.MoveNext() ?? false, message, parameters);
27
        }
28
    }
29
}

The following methods check whether the specified sequence contains one single value/contains more then one values:

1
public static void Single<T>(IEnumerable<T> actual, string message = null, params object[] parameters)
2
{
3
    Assert.IsNotNull(actual, message, parameters);
4
    using (IEnumerator<T> iterator = actual.GetEnumerator())
5
    {
6
        Assert.IsTrue(iterator.MoveNext() && !iterator.MoveNext(), message, parameters);
7
    }
8
}
9

10
public static void Multiple<T>(IEnumerable<T> actual, string message = null, params object[] parameters)
11
{
12
    Assert.IsNotNull(actual, message, parameters);
13
    using (IEnumerator<T> iterator = actual.GetEnumerator())
14
    {
15
        Assert.IsTrue(iterator.MoveNext() && iterator.MoveNext(), message, parameters);
16
    }
17
}

The following methods check whether the specified sequence contains/does not contain the specified value:

1
public static void Contains<T>(
2
    T expected,
3
    IEnumerable<T> actual,
4
    IEqualityComparer<T> comparer = null,
5
    string message = null,
6
    params object[] parameters)
7
{
8
    Assert.IsNotNull(actual, message, parameters);
9
    comparer = comparer ?? EqualityComparer<T>.Default;
10
    foreach (T value in actual)
11
    {
12
        if (comparer.Equals(expected, value))
13
        {
14
            return;
15
        }
16
    }
17
    Assert.Fail(message, parameters);
18
}
19

20
public static void DoesNotContain<T>(
21
    T expected, IEnumerable<T> actual,
22
    IEqualityComparer<T> comparer = null,
23
    string message = null,
24
    params object[] parameters)
25
{
26
    Assert.IsNotNull(actual, message, parameters);
27
    comparer = comparer ?? EqualityComparer<T>.Default;
28
    foreach (T value in actual)
29
    {
30
        if (comparer.Equals(expected, value))
31
        {
32
            Assert.Fail(message, parameters);
33
        }
34
    }
35
}

The following AreSequentialEqual method checks whether 2 sequences’ values are sequentially equal:

1
public static void AreSequentialEqual<T>(
2
    IEnumerable<T> expected,
3
    IEnumerable<T> actual,
4
    IEqualityComparer<T> comparer = null,
5
    string message = null,
6
    params object[] parameters)
7
{
8
    Assert.IsNotNull(expected, message ?? $"Expected sequence is null.", parameters);
9
    Assert.IsNotNull(actual, message ?? $"Actual sequence is null.", parameters);
10

11
    comparer = comparer ?? EqualityComparer<T>.Default;
12
    using (IEnumerator<T> expectedItorator = expected.GetEnumerator())
13
    using (IEnumerator<T> actualIterator = actual.GetEnumerator())
14
    {
15
        int expectedIndex = 0;
16
        for (; expectedItorator.MoveNext(); expectedIndex++)
17
        {
18
            Assert.IsTrue(
19
                actualIterator.MoveNext(),
20
                message ?? $"Expected sequence has more than {expectedIndex} value(s), actual sequence has {expectedIndex} value(s).",
21
                parameters);
22
            T expectedValue = expectedItorator.Current;
23
            T actualValue = actualIterator.Current;
24
            Assert.IsTrue(
25
                comparer.Equals(expectedValue, actualValue),
26
                message ?? $"Expected and actual sequences' values are not equal at index {expectedIndex}. Expected value is {expectedValue}, actual value is {actualValue}.",
27
                parameters);
28
        }
29
        Assert.IsFalse(
30
            actualIterator.MoveNext(),
31
            message ?? $"Expected sequence has {expectedIndex} value(s), actual sequence has more than {expectedIndex} value(s).",
32
            parameters);
33
    }
34
}

foreach loop vs. for loop#

Array is a special type. A concrete array T[] inherits System.Array type, which does not implement IEnumerable but IEnumerable:

1
namespace System
2
{
3
    public abstract class Array : ICollection, IEnumerable, IList, IStructuralComparable, IStructuralEquatable
4
    {
5
    }
6
}

Instead, T[] directly implements IEnumerable, ICollection, and IList, as long as T[] is single dimensional, and zero–lower bound. So array T[] can be used with foreach loop:

1
internal static void ForEach<T>(T[] array, Action<T> action)
2
{
3
    foreach (T value in array)
4
    {
5
        action(value);
6
    }
7
}

For better performance, it is compiled into a for loop, accessing each value with index. For array, this is cheaper than calling MoveNext method and Current getter:

1
internal static void CompiledForEach<T>(T[] array, Action<T> action)
2
{
3
    for (int index = 0; index < array.Length; index++)
4
    {
5
        T value = array[index];
6
        action(value);
7
    }
8
}

And so is string. Since string is a sequence of characters, it implements IEnumerable. When string is used with foreach loop, it is also compiled to for loop for better performance:

1
internal static void ForEach(string @string, Action<char> action)
2
{
3
    foreach (char value in @string)
4
    {
5
        action(value);
6
    }
7
}
8

9
internal static void CompiledForEach(string @string, Action<char> action)
10
{
11
    for (int index = 0; index < @string.Length; index++)
12
    {
13
        char value = @string[index];
14
        action(value);
15
    }
16
}

LINQ to Objects queryable types#

Most of pull-based .NET sequence and collection types implements IEnumerable, like T[], List, Dictionary<TKey, TValue>, HashSet, Collection, Stack, Queue, etc. Here is a detailed list of .NET types implemented IEnumerable:

System.Collections.Generic.IEnumerable
Microsoft.Collections.Immutable.IImmutableQueue
- Microsoft.Collections.Immutable.ImmutableQueue
Microsoft.Collections.Immutable.IImmutableStack
- Microsoft.Collections.Immutable.ImmutableStack
Microsoft.Collections.Immutable.IOrderedCollection
- Microsoft.Collections.Immutable.ImmutableList
System.Collections.Concurrent.IProducerConsumerCollection
- System.Collections.Concurrent.ConcurrentBag
- System.Collections.Concurrent.ConcurrentQueue
- System.Collections.Concurrent.ConcurrentStack
System.Collections.Concurrent.BlockingCollection
System.Collections.Generic.ICollection
- System.Collections.Generic.IDictionary<TKey, TValue>
  - System.Collections.Concurrent.ConcurrentDictionary<TKey, TValue>
  - System.Collections.Generic.Dictionary<TKey, TValue>
  - System.Collections.ObjectModel.ReadOnlyDictionary<TKey, TValue>
  - System.Dynamic.ExpandoObject
- System.Collections.Generic.IList
- - System.ArraySegment
  - System.Collections.Generic.List
  - System.Collections.ObjectModel.Collection
  - - System.Collections.ObjectModel.ObservableCollection
    - System.Collections.ObjectModel.KeyedCollection<TKey, TItem>
  - System.Collections.ObjectModel.ReadOnlyCollection
- System.Collections.Generic.ISet
  - System.Collections.Generic.HashSet
  - System.Collections.Generic.SortedSet
System.Collections.Generic.IReadOnlyCollection
- System.Collections.Generic.IReadOnlyDictionary<TKey, TValue>
  - System.Collections.Generic.Dictionary<TKey, TValue>
  - System.Collections.ObjectModel.ReadOnlyDictionary<TKey, TValue>
  - Microsoft.Collections.Immutable.IImmutableDictionary<TKey, TValue>
  - - Microsoft.Collections.Immutable.ImmutableDictionary<TKey, TValue>
    - Microsoft.Collections.Immutable.ImmutableSortedDictionary<TKey, TValue>
  - System.Collections.Generic.Dictionary<TKey, TValue>
  - System.Collections.ObjectModel.ReadOnlyDictionary<TKey, TValue>
- System.Collections.Generic.IReadOnlyList
  - Microsoft.Collections.Immutable.IImmutableList
    Microsoft.Collections.Immutable.ImmutableList
  - System.Collections.Generic.List
  - System.Collections.ObjectModel.Collection
  - System.Collections.ObjectModel.ReadOnlyCollection
- Microsoft.Collections.Immutable.IImmutableSet
  - Microsoft.Collections.Immutable.IImmutableHashSet
    Microsoft.Collections.Immutable.ImmutableHashSet
  - Microsoft.Collections.Immutable.IImmutableSortedSet
    Microsoft.Collections.Immutable.ImmutableSortedSet
System.Collections.Generic.LinkedList
System.Collections.Generic.Queue
System.Collections.Generic.SortedList<TKey, TValue>
System.Collections.Generic.Stack
System.Linq.IGrouping<TKey, TElement>
System.Linq.ILookup<TKey, TElement>
- System.Linq.Lookup<TKey, TElement>
System.Linq.IOrderedEnumerable
System.Linq.ParallelQuery*
- System.Linq.OrderedParallelQuery
System.Linq.IQueryable*
- System.Linq.IOrderedQueryable
  - System.Linq.EnumerableQuery
  - System.Data.Objects.ObjectQuery
  - System.Data.Entity.Core.Objects.ObjectQuery
    System.Data.Entity.Core.Objects.ObjectSet
  - System.Data.Entity.Infrastructure.DbQuery
  - - System.Data.Entity.DbSet
  - Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable
- System.Data.Linq.ITable
- - System.Data.Linq.Table
- Microsoft.EntityFrameworkCore.DbSet
T[] (not System.Array)

So the LINQ to Objects query methods and query expression are available for to all the above types. Please notice ParallelQuery represents local sequence where values can be pulled in parallel. It implements IEnumerable, so it also supports pulling values sequentially. IQueryable represents remote sequence of values. It also implements IEnumerable, which its values can be loaded to local memory of current .NET application or service, and be queried locally and sequentially. This chapter covers LINQ to Objects queries for IEnumerable. ParallelQuery is covered in the Parallel LINQ chapter, and IQueryable is covered in the LINQ to Entities chapter.

Non-generic sequence#

For historical reason, there are a number of .NET early built-in types only implement IEnumerable. The following example queries these types from the core library:

1
internal static void NonGenericSequences()
2
{
3
    Type nonGenericEnumerable = typeof(IEnumerable);
4
    Type genericEnumerable = typeof(IEnumerable<>);
5
    IEnumerable<Type> nonGenericSequences = typeof(object).Assembly // Core library.
6
        .GetExportedTypes()
7
        .Where(type =>
8
        {
9
            if (type == nonGenericEnumerable || type == genericEnumerable)
10
            {
11
                return false;
12
            }
13
            Type[] interfaces = type.GetInterfaces();
14
            return interfaces.Any(@interface => @interface == nonGenericEnumerable)
15
                && !interfaces.Any(@interface =>
16
                    @interface.IsGenericType
17
                    && @interface.GetGenericTypeDefinition() == genericEnumerable);
18
        })
19
        .OrderBy(type => type.FullName); // Define query.
20
    foreach (Type nonGenericSequence in nonGenericSequences) // Execute query.
21
    {
22
        nonGenericSequence.FullName.WriteLine();
23
    }
24
#if NETFX
25
    // System.Array
26
    // System.Collections.ArrayList
27
    // System.Collections.BitArray
28
    // System.Collections.CollectionBase
29
    // System.Collections.DictionaryBase
30
    // System.Collections.Hashtable
31
    // System.Collections.ICollection
32
    // System.Collections.IDictionary
33
    // System.Collections.IList
34
    // System.Collections.Queue
35
    // System.Collections.ReadOnlyCollectionBase
36
    // System.Collections.SortedList
37
    // System.Collections.Stack
38
    // System.Resources.IResourceReader
39
    // System.Resources.ResourceReader
40
    // System.Resources.ResourceSet
41
    // System.Runtime.Remoting.Channels.BaseChannelObjectWithProperties
42
    // System.Runtime.Remoting.Channels.BaseChannelSinkWithProperties
43
    // System.Runtime.Remoting.Channels.BaseChannelWithProperties
44
    // System.Security.AccessControl.AuthorizationRuleCollection
45
    // System.Security.AccessControl.CommonAcl
46
    // System.Security.AccessControl.DiscretionaryAcl
47
    // System.Security.AccessControl.GenericAcl
48
    // System.Security.AccessControl.RawAcl
49
    // System.Security.AccessControl.SystemAcl
50
    // System.Security.NamedPermissionSet
51
    // System.Security.Permissions.KeyContainerPermissionAccessEntryCollection
52
    // System.Security.PermissionSet
53
    // System.Security.Policy.ApplicationTrustCollection
54
    // System.Security.Policy.Evidence
55
    // System.Security.ReadOnlyPermissionSet
56
#else
57
    // System.Array
58
    // System.Collections.BitArray
59
    // System.Collections.CollectionBase
60
    // System.Collections.ICollection
61
    // System.Collections.IDictionary
62
    // System.Collections.IList
63
    // System.Resources.IResourceReader
64
    // System.Resources.ResourceSet
65
#endif
66
}

.NET Core’s core library has less types, because many types are moved to separate NuGet packages. For example, in .NET Core, ArrayList, DictionaryBase, Hashtable, Queue, ReadOnlyCollectionBase, SortedList, Stack are moved to NuGet package System.Collections.NonGeneric. A Cast query method is provided to cast non-generic sequence can be casted to generic sequence for further LINQ to Objects query, which will be discussed later.