Why Microsoft needs to do something about Azure Website and Https certificate

Websites are going Https as a normal protocol now. It’s secure and faster than it was. Even Google gives a push in their ranking algorithm to websites that are https. With Azure, the free and easy solution is to use LetsEncrypt. It’s a company that give for free certificate with a 3 months live before it expires. Afterward, it can be renewed. However, it’s a cumbersome task. Microsoft Azure had a top rated UserVoice asking to bring Https on Azure and since LetsEncrypt provided something convenient and that someone on the web created an Azure extension to setup it up, Microsoft closed this need as “done”. It’s not a surprise to see that now, the top ranked request in Azure UserVoice is to have something that doesn’t rely on a third party.

While LetsEncrypt is doing a great job, the problem is the free Azure Extension. It’s maintained by someone outside Microsoft and it’s not as ready, follow or is compatible with other Azure features. While the best solution would be to just have to use a checkbox to activate Https, at least a solution that is not broken should exist.

Some may have problem during setup — I did not. It went all smooth and took less than 1 hour. The problem started few weeks after the installed. First, it doesn’t support Azure Slot. It means that every time you switch between slot, something break. For example, switching slot break the jobs that are responsible to renew the certificate in 3 months. I learned that the hard way by having an email from LetsEncrypt few days before the certification expired.

Second, how to debug what this extension is doing is not clear and documented. It’s pretty black-boxed. You can end up having multiple certificated installed for the same DNS. That is what happen to me when I reinstalled the third-party Azure Extension. And, even if I had many new certificated installed, I was still receiving from LetsEncrypt notice about that my certificate was about to expire.

At the end, if you are using Azure Deployment Slot and the Azure Extension for LetsEncrypt, you will have to manually to install again the certificate. It’s not a big deal, but it’s not working “as advertised”. I do not understand how come Microsoft simply closed this first UserVoice request without evaluating that implications. The current proposed solution is broken with how Azure’s features and not maintained by anyone active (nothing has changed in the last 3 months).

Application Insight to get Popular Pages even if Multiple ids in URL

Application Insight is very powerful system that let you collect telemetry on your website. It’s a service on Azure, which is currently free with the limitation to keep data for 7 days and aggregated data for 90 days.

One interesting thing you can do, without having to configure anything else than using Application Insight with your website, is to query the pageViews table. It allows you to get information about each page requested by user. However, if you are using an application you may have url with ID. For example http://yourwebsite.com/user/12345/data/group/567/. The problem is that you will get result with noise. The same page has variables in the url. A simple fix is to replace any integer. Here is a code you can use on your own data which replace integer with a ‘x’ character.

 pageViews
 | extend replaced_operation_ParentName = replace(@'(\d+)', @'x', operation_Name)
 | where timestamp >= ago(7d) 
 | summarize c = count(replaced_operation_ParentName) by replaced_operation_ParentName
 | where c > 1
 | order by c desc 

The result look like this:

urlapplicationinsight

This is pretty fast, with less than 1 sec I got all the result. If you are not using integer in your url, but Guid, you can change the Regex to replace the pattern of a Guid.

How to boost caching performance to cache Entity Framework object

Entity Framework objects are dangerous for caching because of their nature to keep references to object. If you have an object that contains a list of object that can contain back the initial object, you come in a scenario where you have a infinite deepness of reference. While this is not a problem in memory, since it’s just pointer. However, if you serialize, it can be problematic. Json.Net provides some way to serialize reference which will serialize once and then refer to the object by a $ref id. However, this can still be expensive because the framework needs to navigate through the objects tree to determine if or not it needs more serialization. Another way to optimize the serialization with Json.Net is to have a custom ContractResolver where you can evaluate the level of deepness you are and stop serializing. The reference + custom ContractResolver looks like this:

public static class Serialization
{
    public static string Serialize<T>(T objectToSerialize, int maxDepth = 5)
    {
        using (var performanceLog = new GlimpseCodeSection("Serialize"))
        {
            using (var strWriter = new StringWriter())
            {
                using (var jsonWriter = new CustomJsonTextWriter(strWriter))
                {
                    Func<bool> include = () => jsonWriter != null && jsonWriter.CurrentDepth <= maxDepth;
                    var resolver = new DepthContractResolver(include);
                    var serializer = new JsonSerializer();
                    serializer.Formatting = Formatting.Indented;
                    serializer.ContractResolver = resolver;
                    serializer.ReferenceLoopHandling = ReferenceLoopHandling.Serialize;
                    serializer.PreserveReferencesHandling = PreserveReferencesHandling.Objects;
                    serializer.TypeNameHandling = TypeNameHandling.All;
                    serializer.ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor;
                    serializer.NullValueHandling = NullValueHandling.Include;
                    serializer.Serialize(jsonWriter, objectToSerialize);
                }
                return strWriter.ToString();
            }

        }
    }

    public static T Deserialize<T>(string objectSerialized)
    {
        using (var performanceLog = new GlimpseCodeSection("Deserialize"))
        {
            var contractResolver = new PrivateResolver();
            var obj = JsonConvert.DeserializeObject<T>(objectSerialized
                , new JsonSerializerSettings
                {
                    ReferenceLoopHandling = ReferenceLoopHandling.Serialize,
                    PreserveReferencesHandling = PreserveReferencesHandling.Objects,
                    TypeNameHandling = TypeNameHandling.All,
                    ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor,
                    ContractResolver = contractResolver,
                    NullValueHandling = NullValueHandling.Include
                });
            return obj;
        }
    }

    /// <summary>
    /// Allow to have private method to be written in the serialization
    /// </summary>
    public class PrivateResolver : DefaultContractResolver
    {
        protected override JsonProperty CreateProperty(MemberInfo member, MemberSerialization memberSerialization)
        {
            var prop = base.CreateProperty(member, memberSerialization);

            if (!prop.Writable)
            {
                var property = member as PropertyInfo;
                if (property != null)
                {
                    var hasPrivateSetter = property.GetSetMethod(true) != null;
                    prop.Writable = hasPrivateSetter;
                }
            }

            return prop;
        }
    }

    public class DepthContractResolver : DefaultContractResolver
    {
        private readonly Func<bool> includeProperty;

        public DepthContractResolver(Func<bool> includeProperty)
        {
            this.includeProperty = includeProperty;
        }

        protected override JsonProperty CreateProperty(MemberInfo member, MemberSerialization memberSerialization)
        {
            var property = base.CreateProperty(member, memberSerialization);
            //See if we should serialize with the depth
            var shouldSerialize = property.ShouldSerialize;
            property.ShouldSerialize = obj => this.includeProperty() 
                                                && (shouldSerialize == null || shouldSerialize(obj));

            //Setter if private is okay to serialize
            if (!property.Writable)
            {
                var propertyInfo = member as PropertyInfo;
                if (propertyInfo != null)
                {
                    var hasPrivateSetter = propertyInfo.GetSetMethod(true) != null;
                    property.Writable = hasPrivateSetter;
                }
            }


            return property;
        }

        protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
        {
            IList<JsonProperty> props = base.CreateProperties(type, memberSerialization);
            var propertyToSerialize = new List<JsonProperty>();
            foreach (var property in props)
            {
                if (property.Writable)
                {
                    propertyToSerialize.Add(property);
                }
                else
                {
                    var propertyInfo = type.GetProperty(property.PropertyName);
                    if (propertyInfo != null)
                    {
                        var hasPrivateSetter = propertyInfo.GetSetMethod(true) != null;
                        if (hasPrivateSetter)
                        {
                            propertyToSerialize.Add(property);
                        }
                    }
                }
            }
            return propertyToSerialize;
        }

    }

    

    public class CustomJsonTextWriter : JsonTextWriter
    {
        public int CurrentDepth { get; private set; } = 0;
        public CustomJsonTextWriter(TextWriter textWriter) : base(textWriter)
        {
        }

        public override void WriteStartObject()
        {
            this.CurrentDepth++;
            base.WriteStartObject();
        }

        public override void WriteEndObject()
        {
            this.CurrentDepth--;
            base.WriteEndObject();
        }
    }
}

The problem is that even with those optimizations, the time can be long. One common pattern is that you have a big Entity Framework object that you want to serialize. Before sending the object to serialize, you want to cut some branches by setting to null properties. For example, if you have the main entity that has many collections, you may want to null the collection and just setting the object with less sub-objects into Redis. The problem is that if you null a property, the main object will have some missing data and your object is in a bad state. So, the pattern is to serialize the object once and deserialize it right away. Null some properties on that deserialized object, which is a total clone. Any changes doesn’t affect the real object. From that clone, you can serialize this one and set it to Redis. The problem is that it takes 2 serializations operation and 1 deserialization while the best case scenario would be 1 serialization.

The pattern remains good, but the way to achieve it is wrong. A better approach would be to clone the object in C#. The benefit is the speed, the disadvantage is that you need to have cloning method on all your classes which can be time consuming. It’s also difficult to know how to clone each object. Often you will need a shallow clone and a deep clone. Depending of the situation and the class, you need to call the right cloning method. The speed is varying a lot but on big cloning object where the graph is huge I saw result going from 500ms to 4ms. Very good for a first clone operation. After, cutting some properties and serializing again, the same object can take about 20ms to serialize.

Here is an example :

public Contest ShallowCloneManual()
{
	var contest = (Contest)this.MemberwiseClone();
	contest.RegistrationRules = this.registrationRules.DeepCloneManual();
	contest.AllowedMarkets = this.AllowedMarkets?.ShallowCloneManual();
	contest.ContestOrderType = this.contestOrderType?.DeepCloneManual();
	contest.Creator = this.Creator?.ShallowCloneManual();
	contest.DailyStatistics = this.DailyStatistics?.ShallowCloneManual();
	contest.InitialCapital = this.InitialCapital.DeepCloneManual();
	contest.Moderators = this.Moderators?.ShallowCloneManual();
	contest.Name = this.Name.DeepCloneManual();
	contest.TransactionRules = this.TransactionRules.DeepCloneManual();
	contest.StockRules = this.StockRules?.DeepCloneManual();
	contest.ShortRules = this.ShortRules?.DeepCloneManual();
	contest.OptionRules = this.OptionRules?.DeepCloneManual();
	contest.Portefolios = this.Portefolios?.ShallowCloneManual();
	return contest;
}

public Contest DeepCloneManual()
{
	var contest = (Contest)this.MemberwiseClone();
	contest.RegistrationRules = this.registrationRules.DeepCloneManual();
	contest.AllowedMarkets = this.AllowedMarkets?.ShallowCloneManual();
	contest.ContestOrderType = this.contestOrderType?.DeepCloneManual();
	contest.Creator = this.Creator?.ShallowCloneManual();
	contest.DailyStatistics = this.DailyStatistics?.ShallowCloneManual();
	contest.InitialCapital = this.InitialCapital.DeepCloneManual();
	contest.Moderators = this.Moderators?.DeepCloneManual();
	contest.Name = this.Name.DeepCloneManual();
	contest.TransactionRules = this.TransactionRules.DeepCloneManual();
	contest.StockRules = this.StockRules?.DeepCloneManual();
	contest.ShortRules = this.ShortRules?.DeepCloneManual();
	contest.OptionRules = this.OptionRules?.DeepCloneManual();
	contest.Portefolios = this.Portefolios?.DeepCloneManual();
	return contest;
}

Some improvements could be done to be more generic. For example, DeepCloneManual could take a static option object which track the deep level and stop cloning. The impact of doing the cloning in C# was significant on Azure Webjob where thousand of objects needed to be reduced and send to Azure Redis. You can see by yourself the drop in the following graph where the 75th percentile get down from 16 minutes to less than 4 minutes and the 95th percentile from +20 minutes to 4 minutes.
CustomCSharpClone

To conclude, cloning by serializing and deserializing an Entity Framework object is expensive in term of processing but fast to use. It should be used with parsimony.

How to Write a IF Statement that determine a value

Pretty much a basic case if you have done some programming. How to write a IF statement is an agnostic problem when it’s to assign one or multiple variables to be used. There is two patterns that I often see. The first one assign the variable or property directly.

if (/*what ever*/) {
    this.icon = "icon1";
}
else {
    this.icon = "icon2";
}

The second approach set the value into a temporary, scoped, variable and at the end of the IF assign the value to the field/property.

var iconType = "";
if (/*what ever*/) {
    iconType  = "icon1";
}
else {
    iconType  = "icon2";
}
this.icon = iconType ;

These two examples could be that instead of assigning to this.icon would be that we call this.featureMethod(icon). Like the two examples above, in the first approach, you would see the method twice, while on the second approach you would assign the value into a variable and have the method call once at the end. The first approach is appealing because you do not have to assign a temporary variable. However, we have code duplication that doesn’t seem to bother most people. The real problem is in code maintenance. If the method that needs to be invoked change it’s signature, you have two places to change instead of 1. If the IF become with more condition (else if) you will have to call the method (or assign field/property) few more times instead of just keeping a single call. These two argumentation leans in favor of the second approach and there is more. The second approach is cleaner in term of figuring out what is going on. The first approach is taking a decision and executing at the same time. If you look at the method, you cannot have a clear view of what is happening. From top to bottom you have multiple sections that do a condition check + action. Thus, the second approach is cleaner. We could even break down the code into two distinct part: arrange and act. We could refactor the method into 2 sub-methods which one determines the values to be used and the second that set values or calls methods.

I am bringing that point because the first approach seems to be taken with the argument that it’s the same as the second one. The real justification is that the first one is taking 2 lines of code less, hence faster to type which make it an easy default choice. If you are using the first approach, I suggest that you try for few times the second approach. You will see the benefits slowly when working and modifying that code again in the future.

here is an example of 3 temporary variables

function getMyLink(threshold: number) {
    // Default
    let url: string = "http://aboveHundred.com";
    let className: string = "default";
    let padding: number = 0;
    // Logics
    if (threshold <= 0) {
        url = "http://underOrZero.com";
        className = "dark-theme";
        padding = 100;
    }
    else if (threshold > 0 && threshold < 100) {
        url = "http://betweenZeroAndHundred.com";
        className = "light-theme";
        padding = 200;
    }
    // Assignments
    this.url = url;
    this.className = className;
    this.padding = padding;
}

If the next iteration of changes in the code requires to change one of the assignment to other variable, we have a single place to change. If instead of assigning we need to return something, we also have a single place to change.

function getMyLink(threshold: number) {
    // Default
    let url: string = "http://aboveHundred.com";
    let className: string = "default";
    let padding: number = 0;
    // Logics
    if (threshold <= 0) {
        url = "http://underOrZero.com";
        className = "dark-theme";
        padding = 100;
    }
    else if (threshold > 0 && threshold < 100) {
        url = "http://betweenZeroAndHundred.com";
        className = "light-theme";
        padding = 200;
    }
    // Now we return
    return `<a href="${url}" class="${className}" style="padding:${padding}">Click Here</a>`;
}

In term of flexibility, you may have to define these variables but the code is structured to be well resistant to future changes. Also, when a function requires a lot of assignation, it is often a case that the method will be long. It means that it’s even harder to have an idea of what is going on if assignations are done all over the function. I strongly believe that while assigning a long list of variables can be cumbersome that assigning them directly to several places reduce the readability and introduce more error (like forgetting one assignment in a specific case which keep an old assignment).

There are pros and cons in both, but the one I illustrate has more pros than cons in my opinion.

To recap about the advantage of having to assign values and then calling or assigning:

  • Remove code duplication
  • Easier refactoring since only one signature to change
  • Clearer readability of what happen into a method
  • Allow faster refactoring into smaller methods