Per test configuration files...

I have need to write multiple tests againts different configuration files. Is there anyway to specify at the test level which configuration file to use. If not how can I test different configurations

Thanks,
Paul

Answer this question

Per test configuration files...

Bodis
Hi Paul,

I assume you mean test run configuration files (.testrunconfig). No, we currently don't support specifying a different configuration file for each test. The only piece of configuration that can be different will be files to deploy. If you don't mind, can you tell us specifically which configurations you would like to set differently for each test

To use different configuration files, you will have to set each test run configuration as the "Active" run configuration, then run all tests that use that test run configuration. Repeat with another test run configuration and set of tests etc. To facilitate that, you may want to categorize your tests as test lists so you can easily group them together to run.

Thanks.
Winnie
Kevin A. G.
no I mean the app.config file for a .net application....

Consider an application with a custom configuration section. Tests need to be written to verify that the configuration code is properly working and that invalid configurations are dealt with properly by the application.

My narrow mind suggested that I write a 'good' configuration file and then write tests that use that configuration file to ensure the 'good' configuration scenarios. It also suggested that I write multiple 'bad' configuration files to test individual places where I'm looking for 'bad' configuration information.

I was hoping there was an attribute that I could apply to a test that would mark which .config file to use for the test, sadly this isn't the case. When you couple this oversight (I just don't see how you guys could develop a unit testing infrastructure that totally ignored .config files... totally mind blowing... absolutly makes me wonder if you guys ever talk to people who write .net code) with this overight (http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=134725&SiteID=1) you create a scenario where it is extremely complicated to make testing configuration files work on the local machine and against a build server.
Genovia
Hi Paul,

Sorry for misunderstanding your question.
No, we don't offer a solution for that in v1. While we do deploy the "App.Config" file, we don't deploy other config files in the project. I will make sure we track this requirement for v2.
The only workaround I can think of -- which I realize is very inconvenient -- is to specify, for each test, the specific .config file to use as a deployment item. In the body of the test, rename that .config on the fly so the app will use it.

Thanks for the feedback.
Winnie

Jim Duffy

I'm at the house now and watching a good football game, so I'm a little less on edge about having to beat my way through this mess today (and last Friday). Let me preface all of this again by expressing just how disappointed I am that configuration file scenarios have been overlooked for v1. It really is mindblowing when you consider that every single web developer (as least everone worth a salt) uses a minimum of 2 configuration files (one for development, one for deployment). This is such a problem that the asp.net team has released an extension for VS2005 that directly addresses multiple configuration files during deployment. I can see there becoming 3 common files, one for local development, one for deployment and one for the build server. Oh well, the tool is still better than the other stuff out there, so it's hard to complain; but... *glare*

So here is the solution that I worked out along with my suggestions to make this whole thing easier in v2. To recap here is the problem:

How do I test classes that extend System.Configuration.ConfigurationSection and ConfigurationElement

Here is a sample class that extends ConfigurationElement (ConfigurationSection would contain ConfigurationElement(s), but the scenario applies equally to both)

public sealed class TargetElement : ConfigurationElement
    {
        [SuppressMessage("Microsoft.Naming", "CA1705:LongAcronymsShouldBePascalCased", MessageId = "Member")]
        [ConfigurationProperty("oid", IsKey = true, IsRequired = true)]
        public String OID
        {
            get { return (String)base["oid"]; }
            set { base["oid"] = value; }
        }

        [ConfigurationProperty("interval", DefaultValue = "300", IsRequired = true)]
        [IntegerValidator(MinValue = 30, MaxValue = 43200)]
        public Int32 Interval
        {
            get { return (Int32)base["interval"]; }
            set { base["interval"] = value; }
        }

        [ConfigurationProperty("description", IsRequired = true)]
        public String Description
        {
            get { return (String)base["description"]; }
            set { base["description"] = value; }
        }

        [ConfigurationProperty("storageName", IsRequired = true)]
        public String StorageName
        {
            get { return (String)base["storageName"]; }
            set { base["storageName"] = value; }
        }
    }

1st Point: Be aware of this bug in the IntegerValidator.

The class above will ultimately become xml that looks like this in the .config file:

<addTarget oid="1.1.1.1.1.1.0" interval="30"
description="queries some oid for some very important billing data..."
storageName="Table000" />

My testing plan was to write one 'known good' configuration file that would be agressively validated on every point by individual tests that would load this configuration file. The second half of the plan was to write multiple configuration files with just 1 error using the ExpectedException attribute to capture expected execution.

2nd & 3rd Points: As mentioned in this thread, there is no attribute that can be used to delegate configuration file management to team test. I would recommend this (probably on the TestClass constructor) for v2. The base fixture could then use this in conjunction with some collection to hold all configuration files registered to address the MS point made in this post about why content files are not moved into the OUT directory. I would call that post a bug... I am totally flabergasted with the response that you have decided for me which of my files get moved during a test operation (at the very least a check box in the testconfig-->deployment form to let team test 'honor my content copy choices').

So I'm on a list with the Enterprise Application Block people and I know that they have a: extensive use of the new System.Configuration classes and b: have VSTS tests for all scenarios they support. I decided to fire open thier project to see what they did.

They have this helper method:

public static Configuration GetConfigurationForCustomFile(String fileName)
        {
            ExeConfigurationFileMap fileMap = new ExeConfigurationFileMap();

            fileMap.ExeConfigFilename = fileName;
            File.SetAttributes(fileMap.ExeConfigFilename, FileAttributes.Normal);

            return ConfigurationManager.OpenMappedExeConfiguration(fileMap, ConfigurationUserLevel.None);
        }

Basically this loads a configuration file by file name. Here is how I'm handling the ExpectedException test blocks.

TestMethod()]
        [ExpectedException(typeof(ConfigurationErrorsException))]
        public void DoesTheSnmpQueryTimeoutValidatorWork__MaxValue()
        {
            try
            {
                // get the bad configuration file
                Configuration config = GetConfigurationForCustomFile("_badHighSnmpQueryTimeout.config");

                // get the bad poll config section
                PollConfigSection pollConfigSection =
                    (PollConfigSection)config.GetSection(PollConfigSection.SectionName);

                // make sure the section exists
                Assert.IsNotNull(pollConfigSection);

                //   read the property to throw the exception

                Int32 queryTimeout = pollConfigSection.HostEntries[0].SnmpQueryTimeOut;
            }
            catch (ConfigurationErrorsException ex)
            {
                String message = ex.ToString();

                Assert.IsTrue(message.Contains("snmpQueryTimeOut") &&
                    message.Contains("The value must be inside the range 10-300."));

                throw;
            }
        }

Now because the ConfigurationErrorsException will be raised for almost every type of invalid configuration and we are testing for specific wrongs, I have to sniff the exception to see if the exception is the correct exception. This brings me to

Point 4: This message should contain the invalid value as well at the expected range. Here is the ToString() dump of the message:

System.Configuration.ConfigurationErrorsException: The value for the property 'snmpQueryTimeOut' is not valid. The error is: The value must be inside the range 10-300. (F:\My Projects\WebHosting.net\Fido\TestResults\paul.d.murphy_HOME-WORKSTATIO 2005-11-14 22_50_05\Out\_badHighSnmpQueryTimeout.config line 42)
   at System.Configuration.ConfigurationElement.get_Item(ConfigurationProperty prop)
   at System.Configuration.ConfigurationElement.get_Item(String propertyName)
   at Fido.SnmpPollerServer.Configuration.HostElement.get_SnmpQueryTimeOut() in F:\My Projects\WebHosting.net\Fido\Fido.CoreServers\SnmpPollerServer\Configuration\HostElement.cs:line 51
   at Fido.CoreServers.Test.SnmpPollerServer.ConfigurationTests.DoesTheSnmpQueryTimeoutValidatorWork__MaxValue() in F:\My Projects\WebHosting.net\Fido\Fido.CoreServers.Test\SnmpPollerServer\ConfigurationTests.cs:line 311

I hope this gives you a good starting point for examaning how team test can provide more robust application configuration scenario support.

Per test configuration files...

Answer this question