• Michael Lang
• | December 31, 2013
• | Azure
• | 1 Comments
• | azure, Windows Azure

Azure problems at deployment – resolved

When you create an application many times it works for you on your machine, but not when you deploy it elsewhere. This can happen on Azure websites or cloud services. But it isn’t the platforms fault. It would be the platforms fault if it didn’t supply a good way to determine the cause of the problem. However, as I recently found, Azure websites and cloud services have great diagnostic tools.

Remote Debugging

As of the Azure SDK 2.2 remote debugging is now available. This came out shortly after Visual Studio 2013 was released, and it also supports Visual Studio 2012. See Scott Guthrie’s post for the details.

I previously used all the normal logging and tracing options to find out what was happening. But sometimes custom logging code is not enough.

Web API, Server Error in Application

When I first deployed my mix of an MVC web application and a Web API service into a single Azure Cloud Service web role I received a standard “Server Error in ‘/’ Application”. But it worked fine locally in the emulator. This error can mean anything, so good luck finding the cause without more information.

I tried using the Windows Event log entries that appear when you turn on diagnostics and tracing in the Azure portal. No error appeared. I already had some logging with log4net that captured no errors. Luckily when Visual Studio 2013 was released they included remote debugging. So per Scott Guthrie’s instructions I used server explorer to connect a remote debugging session to the deployed web role, let it run, and the code threw an exception with the line in error highlighted in debug mode. It the exception helper I found this stack trace.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
   at System.Xml.XmlTextReaderImpl.OpenUrlDelegate(Object xmlResolver)
   at System.Threading.CompressedStack.runTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.CompressedStack.Run(CompressedStack compressedStack, ContextCallback callback, Object state)
   at System.Xml.XmlTextReaderImpl.OpenUrl()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.XPath.XPathDocument.LoadFromReader(XmlReader reader, XmlSpace space)
   at System.Xml.XPath.XPathDocument..ctor(String uri, XmlSpace space)
   at CompanyName.ProjectName.PublicMvcWeb.Areas.HelpPage.XmlDocumentationProvider..ctor(String documentPath) in 
   ..

The error was on the line to setup the autogenerated documentation for Web API. This file is autogenerated code created by NuGet package ‘Microsoft.AspNet.WebApi.HelpPage’

namespace CompanyName.ProjectName.PublicMvcWeb.Areas.HelpPage
{
    /// <summary>
    /// A custom <see cref="IDocumentationProvider"/> that reads the API documentation from an XML documentation file.
    /// </summary>
    public class XmlDocumentationProvider : IDocumentationProvider
    {
        private XPathNavigator _documentNavigator;
        private const string MethodExpression = "/doc/members/member[@name='M:{0}']";
        private const string ParameterExpression = "param[@name='{0}']";

        /// <summary>
        /// Initializes a new instance of the <see cref="XmlDocumentationProvider"/> class.
        /// </summary>
        /// <param name="documentPath">The physical path to XML document.</param>
        public XmlDocumentationProvider(string documentPath)
        {
            if (documentPath == null)
            {
                throw new ArgumentNullException("documentPath");
            }
            XPathDocument xpath = new XPathDocument(documentPath); //<- error here
            _documentNavigator = xpath.CreateNavigator();
        }
		...
	}
}

The problem was that the generated xml documentation for the web API / MVC site project was not in the folder noted in the code. The project settings were putting that generated xml document in the ~/App_Data/ folder per the Web API help provider documentation. But for some reason on publish to azure it was not included in the publish. All I had to do was change the output path and redeploy.

Diagnostics

I recently had a problem with a cloud service worker role project such that it kept restarting and hanging. I thought maybe it was just trying to do to much work for the role size. But then I looked at the diagnostics table. It is easy to turn on diagnostics for a worker role. It will log errors in a windows event viewer format. See references below for how to enable diagnostics data.

Here is the error from the WADWindowsEventLogsTable diagnostics table.

Application: WaWorkerHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException
Stack:
   at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<InitializeRole>b__0()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart()

A search engine result mentioned this was due to upgrading to the SDK 2.1 version. Since I upgraded to 2.2 a few months ago, I thought this may be a similar issue. I checked all my NuGet packages and all of the open source Candor NuGet packages I was referencing were using the same version of NuGet packages as the solution containing the worker role I was deploying to Azure. These were the same packages the working web site deployed to Azure was using.

  <package id="Microsoft.WindowsAzure.ConfigurationManager" version="2.0.1.0" targetFramework="net45" />
  <package id="WindowsAzure.Storage" version="2.1.0.3" targetFramework="net45" />

I’m still not quite sure what is trying to use the old version of the Azure SDK. However, it didn’t really matter. I decided to check the assembly redirects in the web and worker role. I did find one assembly redirect in my MVC web project that was missing in the worker role, Microsoft.WindowsAzure.ServiceRuntime; so I copied it into my worker role configuration. After deployment the worker role starting working. It seems to be a good precaution to add an assembly redirect for every azure assembly and anything they depend on.

  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.ServiceRuntime" publicKeyToken="31BF3856AD364E35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.2.0.0" newVersion="2.2.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Common.Logging" publicKeyToken="af08829b84f0328e" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.1.2.0" newVersion="2.1.2.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Data.OData" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-5.6.0.0" newVersion="5.6.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.Configuration" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.0.0.0" newVersion="2.0.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.WindowsAzure.Storage" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-2.1.0.3" newVersion="2.1.0.3" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="Microsoft.Data.Edm" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-5.6.0.0" newVersion="5.6.0.0" />
      </dependentAssembly>
      <dependentAssembly>
        <assemblyIdentity name="System.Spatial" publicKeyToken="31bf3856ad364e35" culture="neutral" />
        <bindingRedirect oldVersion="0.0.0.0-5.6.0.0" newVersion="5.6.0.0" />
      </dependentAssembly>
    </assemblyBinding>
  </runtime>

Platform as a Service

Windows Azure platform as a service options really give lots of helpful information when errors occur. I can deploy websites or cloud service web roles with confidence that I can diagnose the cause of any issue that I could similarly diagnose on a dedicated server. Platform as a service makes my life as a developer easier.

References

Scott Guthrie: Windows Azure: Announcing release of Windows Azure SDK 2.2 (with lots of goodies)
http://weblogs.asp.net/scottgu/archive/2013/10/22/windows-azure-announcing-release-of-windows-azure-sdk-2-2-with-lots-of-goodies.aspx

Initialize or Change Windows Azure Diagnostics Configuration
http://msdn.microsoft.com/en-us/library/windowsazure/hh411537.aspx

Technet forum: “WaWorkerHost.exe Exception Info: System.IO.FileNotFoundException”
http://social.technet.microsoft.com/Forums/en-US/6a715569-9f93-44b2-a165-677feb354710/waworkerhostexe-exception-info-systemiofilenotfoundexception

Candor open source libraries:
https://github.com/michael-lang/candor-common

Candor NuGet packages:
https://www.nuget.org/profiles/mlang/