Optimization of test instance usage

[andreasjordan]

We originally used three different instances for our tests:

• instance1: A default instance
• instance2: A named instance on a fixed port
• instance3: A named instance on a fixed port with HADR enabled

We used different versions, but we only test every command on a specific version, so that does not make sense to me.

We are currently in a process to only use two instances to have a smaller setup on AppVeyor. But we use more aliases for those instances to be able to use complex test environments as well.

Current design:

• instanceSingle: For tests that only need a single instance and don't have spezial requirements. We could run those tests against different instances like default or named instance, FCI or AG instances, different versions and editions.
• instanceExpress: For tests that need an express edition. So we only run those tests if we have an express edition in the lab.
• instanceDefault: For all tests that need a default instance.
• instanceNamed: For all tests that need a named instance.
• instanceLinux: For all tests that need an instance on a Linux system.
• instanceMulti1 and instanceMulti2: For all the commands that should be tested against different instances but the instances could be on the same node. Version does not matter.
• instanceCopy1 and instanceCopy2: For all the copy command where the two instances should run on different nodes. InstanceCopy2 has to have at least the same or a higher SQL Server version than InstanceCopy1 so that a backup from InstanceCopy1 could be restored to InstanceCopy2.
• instanceHadr (later or in my lab: instanceHadr1 and instanceHard2): For all the HARD commands with AG and Mirroring where the two instances should run on different nodes and need HADR enabled. As we don't have two instances of the same version in AppVeyor, we currently use only SQL2017. For instanceHadr1 and instanceHard2: Both instances need to have the same SQL Server version.

I ran this code (current branch "more_tests_to_scenarios"):

$testPath = "$env:OneDrive\github.com\dataplat\dbatools\tests"

$testFiles = Get-ChildItem -Path $testPath -Filter *-Dba*.Tests.ps1

$result = foreach ($file in $testFiles) {
    $content = Get-Content -Path $file.FullName
    $instanceNumbers = foreach ($line in $content) {
        $code = $line -replace '#.*$',''
        [regex]::Matches($code, '\$TestConfig\.Instance(\w+)', [System.Text.RegularExpressions.RegexOptions]::IgnoreCase) | ForEach-Object { $_.Groups[1].Value }
    }
    $instanceNumbers = $instanceNumbers | Sort-Object -Unique

    [PSCustomObject]@{
        Command   = $file.Name -replace '.Tests.ps1$', ''
        Instances = $instanceNumbers -join ' '
    }
}

$resultGroups = $result | Group-Object -Property Instances | Sort-Object Name
$output = @(
    'Summary:'
    '========'
    ''
)
$output += foreach ($group in $resultGroups) {
    "$($group.Count) Tests using these instances: $($group.Name)"
}
$output += @(
    ''
    ''
    'Details:'
    '========'
    ''
)
$output += foreach ($group in $resultGroups) {
    ''
    "$($group.Count) Tests using these instances: $($group.Name)"
    foreach ($command in $group.Group.Command) {
        "* $command"
    }
}

$output | Set-Clipboard

And here is the output (current branch "more_tests_to_scenarios"):

Summary:

203 Tests using these instances:
123 Tests using these instances: 1
257 Tests using these instances: 2
1 Tests using these instances: 2 2_detailed
25 Tests using these instances: 2 3
4 Tests using these instances: 3
31 Tests using these instances: Copy1 Copy2
1 Tests using these instances: Copy1 Copy2 Linux
1 Tests using these instances: Express Single
21 Tests using these instances: Hadr
1 Tests using these instances: Linux Multi1 Multi2
23 Tests using these instances: Multi1 Multi2
8 Tests using these instances: Restart
2 Tests using these instances: s
3 Tests using these instances: Single

Details:

203 Tests using these instances:

• Add-DbaComputerCertificate
• Add-DbaReplArticle
• Backup-DbaComputerCertificate
• Clear-DbaConnectionPool
• Compare-DbaAgReplicaAgentJob
• Compare-DbaAgReplicaCredential
• Compare-DbaAgReplicaLogin
• Compare-DbaAgReplicaOperator
• Compare-DbaAgReplicaSync
• Compare-DbaAvailabilityGroup
• Convert-DbaLSN
• Convert-DbaMaskingValue
• ConvertTo-DbaDataTable
• ConvertTo-DbaTimeline
• Copy-DbaAgentServer
• Copy-DbaDataCollector
• Copy-DbaInstanceAudit
• Copy-DbaInstanceAuditSpecification
• Copy-DbaPolicyManagement
• Copy-DbaSsisCatalog
• Copy-DbaXESession
• Copy-DbaXESessionTemplate
• Disable-DbaReplDistributor
• Disable-DbaReplPublishing
• Enable-DbaReplDistributor
• Enable-DbaReplPublishing
• Export-DbaExecutionPlan
• Export-DbaLinkedServer
• Export-DbaReplServerSetting
• Export-DbaSpConfigure
• Export-DbatoolsConfig
• Find-DbaBackup
• Find-DbaCommand
• Find-DbaLoginInGroup
• Format-DbaBackupInformation
• Get-DbaAgBackupHistory
• Get-DbaAgentJobHistory
• Get-DbaAgentJobOutputFile
• Get-DbaClientAlias
• Get-DbaClientProtocol
• Get-DbaCmConnection
• Get-DbaCmObject
• Get-DbaComputerCertificate
• Get-DbaDbMirrorMonitor
• Get-DbaDbQueryStoreOption
• Get-DbaDiskSpace
• Get-DbaFirewallRule
• Get-DbaInstanceAuditSpecification
• Get-DbaKbUpdate
• Get-DbaLocaleSetting
• Get-DbaMaintenanceSolutionLog
• Get-DbaManagementObject
• Get-DbaMsdtc
• Get-DbaNetworkActivity
• Get-DbaNetworkCertificate
• Get-DbaPageFileSetting
• Get-DbaPbmCategorySubscription
• Get-DbaPbmObjectSet
• Get-DbaPbmStore
• Get-DbaPfDataCollector
• Get-DbaPfDataCollectorCounter
• Get-DbaPfDataCollectorCounterSample
• Get-DbaPfDataCollectorSet
• Get-DbaPfDataCollectorSetTemplate
• Get-DbaPrivilege
• Get-DbaQueryExecutionTime
• Get-DbaRandomizedDataset
• Get-DbaRandomizedDatasetTemplate
• Get-DbaRandomizedType
• Get-DbaRandomizedValue
• Get-DbaReplArticle
• Get-DbaReplArticleColumn
• Get-DbaReplPublication
• Get-DbaReplPublisher
• Get-DbaReplServer
• Get-DbaReplSubscription
• Get-DbaResourceGovernor
• Get-DbaRgWorkloadGroup
• Get-DbaRunningJob
• Get-DbaSpn
• Get-DbaSsisEnvironmentVariable
• Get-DbaSsisExecutionHistory
• Get-DbaTempdbUsage
• Get-DbatoolsChangeLog
• Get-DbatoolsConfig
• Get-DbatoolsConfigValue
• Get-DbatoolsError
• Get-DbatoolsLog
• Get-DbatoolsPath
• Get-DbaWsfcAvailableDisk
• Get-DbaWsfcCluster
• Get-DbaWsfcDisk
• Get-DbaWsfcNetwork
• Get-DbaWsfcNetworkInterface
• Get-DbaWsfcNode
• Get-DbaWsfcResource
• Get-DbaWsfcResourceGroup
• Get-DbaWsfcResourceType
• Get-DbaWsfcRole
• Get-DbaWsfcSharedVolume
• Get-DbaXEObject
• Get-DbaXESessionTemplate
• Get-DbaXEStore
• Import-DbaBinaryFile
• Import-DbaSpConfigure
• Import-DbatoolsConfig
• Import-DbaXESessionTemplate
• Install-DbaInstance
• Install-DbaSqlPackage
• Invoke-DbaAdvancedInstall
• Invoke-DbaAdvancedRestore
• Invoke-DbaAdvancedUpdate
• Invoke-DbaAgFailover
• Invoke-DbaDbAzSqlTip
• Invoke-DbaDbLogShipRecovery
• Invoke-DbaDbMirrorFailover
• Invoke-DbaDbUpgrade
• Invoke-DbaPfRelog
• Invoke-DbatoolsFormatter
• Invoke-DbatoolsRenameHelper
• Invoke-DbaXEReplay
• Join-DbaAvailabilityGroup
• Join-DbaPath
• Measure-DbatoolsImport
• New-DbaAzAccessToken
• New-DbaClientAlias
• New-DbaCmConnection
• New-DbaComputerCertificate
• New-DbaComputerCertificateSigningRequest
• New-DbaConnectionString
• New-DbaConnectionStringBuilder
• New-DbaDacPackage
• New-DbaDiagnosticAdsNotebook
• New-DbaDirectory
• New-DbaReplCreationScriptOptions
• New-DbaReplPublication
• New-DbaReplSubscription
• New-DbaScriptingOption
• New-DbaServiceMasterKey
• New-DbaSsisCatalog
• New-DbatoolsSupportPackage
• New-DbaXESession
• Read-DbaBackupHeader
• Read-DbaTransactionLog
• Register-DbatoolsConfig
• Remove-DbaAgentJobStep
• Remove-DbaAgReplica
• Remove-DbaBackup
• Remove-DbaClientAlias
• Remove-DbaCmConnection
• Remove-DbaComputerCertificate
• Remove-DbaDbBackupRestoreHistory
• Remove-DbaDbMasterKey
• Remove-DbaDbMirror
• Remove-DbaFirewallRule
• Remove-DbaNetworkCertificate
• Remove-DbaReplArticle
• Remove-DbaReplPublication
• Remove-DbaReplSubscription
• Remove-DbaSpn
• Repair-DbaDbMirror
• Repair-DbaInstanceName
• Reset-DbatoolsConfig
• Resolve-DbaNetworkName
• Resolve-DbaPath
• Save-DbaCommunitySoftware
• Save-DbaDiagnosticQueryScript
• Save-DbaKbUpdate
• Select-DbaBackupInformation
• Select-DbaObject
• Set-DbaAgentJob
• Set-DbaAgentJobOutputFile
• Set-DbaAgentJobOwner
• Set-DbaAgListener
• Set-DbaCmConnection
• Set-DbaDbMirror
• Set-DbaEndpoint
• Set-DbaNetworkCertificate
• Set-DbaPowerPlan
• Set-DbaPrivilege
• Set-DbaSpn
• Set-DbatoolsConfig
• Set-DbatoolsInsecureConnection
• Set-DbatoolsPath
• Show-DbaDbList
• Show-DbaInstanceFileSystem
• Start-DbaPfDataCollectorSet
• Stop-DbaPfDataCollectorSet
• Sync-DbaAvailabilityGroup
• Test-DbaAgSpn
• Test-DbaAvailabilityGroup
• Test-DbaBackupInformation
• Test-DbaCmConnection
• Test-DbaComputerCertificateExpiration
• Test-DbaDiskAlignment
• Test-DbaKerberos
• Test-DbaLsnChain
• Test-DbaMaxMemory
• Test-DbaReplLatency
• Unregister-DbatoolsConfig
• Update-DbaBuildReference
• Update-DbaMaintenanceSolution
• Update-Dbatools

123 Tests using these instances: 1

• Add-DbaPfDataCollectorCounter
• Add-DbaRegServerGroup
• Backup-DbaDbCertificate
• Backup-DbaDbMasterKey
• Backup-DbaServiceMasterKey
• Clear-DbaLatchStatistics
• Clear-DbaPlanCache
• Clear-DbaWaitStatistics
• Disable-DbaForceNetworkEncryption
• Disable-DbaHideInstance
• Disable-DbaTraceFlag
• Disconnect-DbaInstance
• Enable-DbaForceNetworkEncryption
• Enable-DbaHideInstance
• Expand-DbaDbLogFile
• Export-DbaCsv
• Export-DbaDacPackage
• Export-DbaDbTableData
• Export-DbaPfDataCollectorSetTemplate
• Export-DbaUser
• Find-DbaDbDisabledIndex
• Find-DbaDbDuplicateIndex
• Find-DbaDbGrowthEvent
• Find-DbaSimilarTable
• Get-DbaBackupInformation
• Get-DbaComputerSystem
• Get-DbaConnectedInstance
• Get-DbaConnection
• Get-DbaCustomError
• Get-DbaDatabase
• Get-DbaDbBackupHistory
• Get-DbaDbccSessionBuffer
• Get-DbaDbCertificate
• Get-DbaDbCompatibility
• Get-DbaDbDbccOpenTran
• Get-DbaDbEncryption
• Get-DbaDbFile
• Get-DbaDbFileMapping
• Get-DbaDbIdentity
• Get-DbaDbMasterKey
• Get-DbaDbOrphanUser
• Get-DbaDbTable
• Get-DbaDefaultPath
• Get-DbaDependency
• Get-DbaDeprecatedFeature
• Get-DbaDump
• Get-DbaErrorLog
• Get-DbaExtendedProtection
• Get-DbaExternalProcess
• Get-DbaFeature
• Get-DbaForceNetworkEncryption
• Get-DbaHideInstance
• Get-DbaInstalledPatch
• Get-DbaLogin
• Get-DbaMemoryCondition
• Get-DbaMemoryUsage
• Get-DbaModule
• Get-DbaOleDbProvider
• Get-DbaOpenTransaction
• Get-DbaOperatingSystem
• Get-DbaPermission
• Get-DbaPfAvailableCounter
• Get-DbaPlanCache
• Get-DbaProcess
• Get-DbaProductKey
• Get-DbaRegServer
• Get-DbaRegServerGroup
• Get-DbaReplDistributor
• Get-DbaSchemaChangeHistory
• Get-DbaSpConfigure
• Get-DbaUserPermission
• Get-DbaWaitResource
• Import-DbaPfDataCollectorSetTemplate
• Install-DbaWhoIsActive
• Invoke-DbaCycleErrorLog
• Invoke-DbaDbccDropCleanBuffer
• Invoke-DbaDbDbccCleanTable
• Invoke-DbaDbDbccUpdateUsage
• Invoke-DbaDbPiiScan
• Invoke-DbaWhoIsActive
• Mount-DbaDatabase
• Move-DbaRegServer
• Move-DbaRegServerGroup
• New-DbaDacOption
• New-DbaDacProfile
• New-DbaDbCertificate
• New-DbaDbDataGeneratorConfig
• New-DbaDbMaskingConfig
• New-DbaDbMasterKey
• Remove-DbaDatabase
• Remove-DbaDbCertificate
• Remove-DbaDbUser
• Remove-DbaLogin
• Remove-DbaPfDataCollectorCounter
• Remove-DbaPfDataCollectorSet
• Remove-DbaRegServer
• Remove-DbaRegServerGroup
• Remove-DbaTrace
• Rename-DbaLogin
• Repair-DbaDbOrphanUser
• Restore-DbaDbCertificate
• Set-DbaDbOwner
• Set-DbaDefaultPath
• Set-DbaExtendedProtection
• Set-DbaSpConfigure
• Set-DbaTempDbConfig
• Start-DbaTrace
• Stop-DbaExternalProcess
• Stop-DbaProcess
• Stop-DbaTrace
• Test-DbaConnection
• Test-DbaConnectionAuthScheme
• Test-DbaDbCollation
• Test-DbaDbDataGeneratorConfig
• Test-DbaDbDataMaskingConfig
• Test-DbaDbOwner
• Test-DbaIdentityUsage
• Test-DbaLastBackup
• Test-DbaLinkedServerConnection
• Test-DbaLoginPassword
• Test-DbaSpn
• Update-DbaInstance
• Write-DbaDbTableData

257 Tests using these instances: 2

• Add-DbaDbMirrorMonitor
• Add-DbaExtendedProperty
• Add-DbaServerRoleMember
• ConvertTo-DbaXESession
• Copy-DbaDbQueryStoreOption
• Copy-DbaSystemDbUserObject
• Disable-DbaDbEncryption
• Disable-DbaStartupProcedure
• Enable-DbaDbEncryption
• Enable-DbaStartupProcedure
• Enable-DbaTraceFlag
• Export-DbaBinaryFile
• Export-DbaCredential
• Export-DbaDbRole
• Export-DbaDiagnosticQuery
• Export-DbaInstance
• Export-DbaLogin
• Export-DbaRegServer
• Export-DbaScript
• Export-DbaServerRole
• Export-DbaSysDbUserObject
• Export-DbaXESession
• Export-DbaXESessionTemplate
• Find-DbaAgentJob
• Find-DbaDatabase
• Find-DbaDbUnusedIndex
• Find-DbaOrphanedFile
• Find-DbaStoredProcedure
• Find-DbaTrigger
• Find-DbaUserObject
• Find-DbaView
• Get-DbaAgentAlert
• Get-DbaAgentAlertCategory
• Get-DbaAgentJob
• Get-DbaAgentJobCategory
• Get-DbaAgentJobStep
• Get-DbaAgentLog
• Get-DbaAgentOperator
• Get-DbaAgentProxy
• Get-DbaAgentServer
• Get-DbaAvailableCollation
• Get-DbaBackupDevice
• Get-DbaBinaryFileTable
• Get-DbaCpuRingBuffer
• Get-DbaCpuUsage
• Get-DbaCredential
• Get-DbaDbAssembly
• Get-DbaDbAsymmetricKey
• Get-DbaDbccHelp
• Get-DbaDbccMemoryStatus
• Get-DbaDbccProcCache
• Get-DbaDbccStatistic
• Get-DbaDbccUserOption
• Get-DbaDbCheckConstraint
• Get-DbaDbCompression
• Get-DbaDbDetachedFileInfo
• Get-DbaDbEncryptionKey
• Get-DbaDbExtentDiff
• Get-DbaDbFeatureUsage
• Get-DbaDbFileGroup
• Get-DbaDbFileGrowth
• Get-DbaDbForeignKey
• Get-DbaDbLogShipError
• Get-DbaDbLogSpace
• Get-DbaDbMail
• Get-DbaDbMailAccount
• Get-DbaDbMailConfig
• Get-DbaDbMailHistory
• Get-DbaDbMailLog
• Get-DbaDbMailServer
• Get-DbaDbMemoryUsage
• Get-DbaDbObjectTrigger
• Get-DbaDbPageInfo
• Get-DbaDbPartitionFunction
• Get-DbaDbPartitionScheme
• Get-DbaDbRecoveryModel
• Get-DbaDbRestoreHistory
• Get-DbaDbRole
• Get-DbaDbRoleMember
• Get-DbaDbSequence
• Get-DbaDbServiceBrokerQueue
• Get-DbaDbServiceBrokerService
• Get-DbaDbSharePoint
• Get-DbaDbSnapshot
• Get-DbaDbSpace
• Get-DbaDbState
• Get-DbaDbStoredProcedure
• Get-DbaDbSynonym
• Get-DbaDbTrigger
• Get-DbaDbUdf
• Get-DbaDbUser
• Get-DbaDbUserDefinedTableType
• Get-DbaDbView
• Get-DbaDbVirtualLogFile
• Get-DbaEndpoint
• Get-DbaErrorLogConfig
• Get-DbaEstimatedCompletionTime
• Get-DbaExecutionPlan
• Get-DbaExtendedProperty
• Get-DbaFile
• Get-DbaFilestream
• Get-DbaHelpIndex
• Get-DbaInstanceAudit
• Get-DbaInstanceInstallDate
• Get-DbaInstanceProperty
• Get-DbaInstanceProtocol
• Get-DbaInstanceTrigger
• Get-DbaInstanceUserOption
• Get-DbaIoLatency
• Get-DbaLastBackup
• Get-DbaLatchStatistic
• Get-DbaNetworkConfiguration
• Get-DbaPbmCategory
• Get-DbaPbmCondition
• Get-DbaPbmPolicy
• Get-DbaPowerPlan
• Get-DbaRegistryRoot
• Get-DbaRegServerStore
• Get-DbaRgClassifierFunction
• Get-DbaRgResourcePool
• Get-DbaServerRole
• Get-DbaService
• Get-DbaSpinLockStatistic
• Get-DbaStartupParameter
• Get-DbaStartupProcedure
• Get-DbaSuspectPage
• Get-DbaTcpPort
• Get-DbaTrace
• Get-DbaTraceFlag
• Get-DbaWaitingTask
• Get-DbaWaitStatistic
• Get-DbaWindowsLog
• Get-DbaXESession
• Get-DbaXESessionTarget
• Get-DbaXESessionTargetFile
• Import-DbaRegServer
• Install-DbaSqlWatch
• Invoke-DbaBalanceDataFiles
• Invoke-DbaDbccFreeCache
• Invoke-DbaDbClone
• Invoke-DbaDbCorruption
• Invoke-DbaDbDataGenerator
• Invoke-DbaDbDataMasking
• Invoke-DbaDbDbccCheckConstraint
• Invoke-DbaDbLogShipping
• Invoke-DbaDbShrink
• Invoke-DbaDiagnosticQuery
• Measure-DbaBackupThroughput
• Measure-DbaDbVirtualLogFile
• Move-DbaDbFile
• New-DbaAgentAlertCategory
• New-DbaAgentJob
• New-DbaAgentJobCategory
• New-DbaAgentJobStep
• New-DbaAgentOperator
• New-DbaAgentProxy
• New-DbaAgentSchedule
• New-DbaCredential
• New-DbaDbAsymmetricKey
• New-DbaDbEncryptionKey
• New-DbaDbFileGroup
• New-DbaDbMailAccount
• New-DbaDbMailProfile
• New-DbaDbRole
• New-DbaDbSequence
• New-DbaDbSnapshot
• New-DbaDbSynonym
• New-DbaDbUser
• New-DbaFirewallRule
• New-DbaRgResourcePool
• New-DbaRgWorkloadGroup
• New-DbaServerRole
• New-DbaSqlParameter
• Read-DbaAuditFile
• Read-DbaTraceFile
• Read-DbaXEFile
• Remove-DbaAgentAlert
• Remove-DbaAgentAlertCategory
• Remove-DbaAgentJobCategory
• Remove-DbaAgentOperator
• Remove-DbaAgentProxy
• Remove-DbaAgentSchedule
• Remove-DbaCredential
• Remove-DbaDbAsymmetricKey
• Remove-DbaDbCheckConstraint
• Remove-DbaDbData
• Remove-DbaDbEncryptionKey
• Remove-DbaDbFileGroup
• Remove-DbaDbLogShipping
• Remove-DbaDbMailAccount
• Remove-DbaDbMailProfile
• Remove-DbaDbMirrorMonitor
• Remove-DbaDbOrphanUser
• Remove-DbaDbPartitionFunction
• Remove-DbaDbPartitionScheme
• Remove-DbaDbRole
• Remove-DbaDbRoleMember
• Remove-DbaDbSequence
• Remove-DbaDbSnapshot
• Remove-DbaDbSynonym
• Remove-DbaDbTable
• Remove-DbaDbUdf
• Remove-DbaDbView
• Remove-DbaEndpoint
• Remove-DbaExtendedProperty
• Remove-DbaRgResourcePool
• Remove-DbaRgWorkloadGroup
• Remove-DbaServerRole
• Remove-DbaServerRoleMember
• Remove-DbaXESession
• Rename-DbaDatabase
• Restore-DbaDbSnapshot
• Select-DbaDbSequenceNextValue
• Set-DbaAgentAlert
• Set-DbaAgentJobCategory
• Set-DbaAgentOperator
• Set-DbaAgentSchedule
• Set-DbaAgentServer
• Set-DbaDbCompression
• Set-DbaDbFileGroup
• Set-DbaDbFileGrowth
• Set-DbaDbIdentity
• Set-DbaDbQueryStoreOption
• Set-DbaDbRecoveryModel
• Set-DbaDbSequence
• Set-DbaDbState
• Set-DbaExtendedProperty
• Set-DbaLogin
• Set-DbaNetworkConfiguration
• Set-DbaResourceGovernor
• Set-DbaRgResourcePool
• Set-DbaRgWorkloadGroup
• Start-DbaDbEncryption
• Start-DbaEndpoint
• Start-DbaXESession
• Stop-DbaAgentJob
• Stop-DbaDbEncryption
• Stop-DbaEndpoint
• Stop-DbaXESession
• Test-DbaAgentJobOwner
• Test-DbaBackupEncrypted
• Test-DbaBuild
• Test-DbaDbCompatibility
• Test-DbaDbCompression
• Test-DbaDbQueryStore
• Test-DbaDbRecoveryModel
• Test-DbaDiskAllocation
• Test-DbaEndpoint
• Test-DbaInstanceName
• Test-DbaManagementObject
• Test-DbaMaxDop
• Test-DbaOptimizeForAdHoc
• Test-DbaPowerPlan
• Test-DbaTempDbConfig
• Test-DbaWindowsLogin
• Uninstall-DbaSqlWatch
• Watch-DbaXESession

1 Tests using these instances: 2 2_detailed

• Restore-DbaDatabase

25 Tests using these instances: 2 3

• Get-DbaAgentSchedule
• Get-DbaDbMailProfile
• Get-DbaDbMirror
• Get-DbaLinkedServer
• Get-DbaLinkedServerLogin
• Install-DbaAgentAdminAlert
• Install-DbaFirstResponderKit
• Install-DbaMaintenanceSolution
• Install-DbaMultiTool
• Invoke-DbaDbMirroring
• Invoke-DbaDbTransfer
• Invoke-DbaQuery
• New-DbaAgentAlert
• New-DbaDatabase
• New-DbaDbTable
• New-DbaEndpoint
• New-DbaLinkedServer
• New-DbaLinkedServerLogin
• Remove-DbaDbTableData
• Remove-DbaLinkedServer
• Remove-DbaLinkedServerLogin
• Set-DbaAgentJobStep
• Start-DbaAgentJob
• Sync-DbaLoginPassword
• Sync-DbaLoginPermission

4 Tests using these instances: 3

• Dismount-DbaDatabase
• Find-DbaInstance
• Install-DbaDarlingData
• Remove-DbaAgentJob

31 Tests using these instances: Copy1 Copy2

• Backup-DbaDatabase
• Copy-DbaAgentAlert
• Copy-DbaAgentJob
• Copy-DbaAgentJobCategory
• Copy-DbaAgentJobStep
• Copy-DbaAgentOperator
• Copy-DbaAgentProxy
• Copy-DbaAgentSchedule
• Copy-DbaBackupDevice
• Copy-DbaCredential
• Copy-DbaCustomError
• Copy-DbaDatabase
• Copy-DbaDbAssembly
• Copy-DbaDbCertificate
• Copy-DbaDbMail
• Copy-DbaDbTableData
• Copy-DbaDbViewData
• Copy-DbaEndpoint
• Copy-DbaExtendedStoredProcedure
• Copy-DbaInstanceTrigger
• Copy-DbaLinkedServer
• Copy-DbaRegServer
• Copy-DbaResourceGovernor
• Copy-DbaServerRole
• Copy-DbaSpConfigure
• Copy-DbaStartupProcedure
• Measure-DbaDiskSpaceRequirement
• Publish-DbaDacPackage
• Remove-DbaDatabaseSafely
• Start-DbaMigration
• Test-DbaMigrationConstraint

1 Tests using these instances: Copy1 Copy2 Linux

• Copy-DbaLogin

1 Tests using these instances: Express Single

• Test-DbaDbLogShipStatus

21 Tests using these instances: Hadr

• Add-DbaAgDatabase
• Add-DbaAgListener
• Add-DbaAgReplica
• Disable-DbaAgHadr
• Enable-DbaAgHadr
• Get-DbaAgDatabase
• Get-DbaAgDatabaseReplicaState
• Get-DbaAgHadr
• Get-DbaAgListener
• Get-DbaAgReplica
• Get-DbaAvailabilityGroup
• Grant-DbaAgPermission
• New-DbaAvailabilityGroup
• Remove-DbaAgDatabase
• Remove-DbaAgListener
• Remove-DbaAvailabilityGroup
• Resume-DbaAgDbDataMovement
• Revoke-DbaAgPermission
• Set-DbaAgReplica
• Set-DbaAvailabilityGroup
• Suspend-DbaAgDbDataMovement

1 Tests using these instances: Linux Multi1 Multi2

• Test-DbaDiskSpeed

23 Tests using these instances: Multi1 Multi2

• Connect-DbaInstance
• Get-DbaBuild
• Get-DbaDbSchema
• Get-DbaLastGoodCheckDb
• Get-DbaMaxMemory
• Get-DbaServerRoleMember
• Get-DbaUptime
• Import-DbaCsv
• Invoke-DbaDbDecryptObject
• New-DbaCustomError
• New-DbaDbSchema
• New-DbaDbTransfer
• New-DbaLogin
• Remove-DbaCustomError
• Remove-DbaDbSchema
• Set-DbaDbCompatibility
• Set-DbaDbSchema
• Set-DbaErrorLogConfig
• Set-DbaMaxDop
• Set-DbaMaxMemory
• Set-DbaStartupParameter
• Test-DbaPath
• Watch-DbaDbLogin

8 Tests using these instances: Restart

• Disable-DbaFilestream
• Enable-DbaFilestream
• Reset-DbaAdmin
• Restart-DbaService
• Set-DbaTcpPort
• Start-DbaService
• Stop-DbaService
• Update-DbaServiceAccount

2 Tests using these instances: s

• Get-DbaTopResourceUsage
• Test-DbaNetworkLatency

3 Tests using these instances: Single

• Add-DbaDbFile
• Add-DbaDbRoleMember
• Add-DbaRegServer

[potatoqualitee]

Sounds good 🙌🏼 The only thing is that we need a dedicated VM for the service commands because they break the other commands and need to be isolated.

[andreasjordan]

We don't run any tests in parallel. In AppVeyor we have currently 5 different VMs with their own instances. So this was actually never a problem.

[andreasjordan]

Based on the last AppVeyor run today:

scenario=2008R2, main_instance=localhost\SQL2008R2SP2, setup_scripts=\tests\appveyor.SQL2008R2SP2.ps1
130 tests in 13 min 40 sec

scenario=service_restarts, main_instance=localhost\SQL2017,localhost\SQL2016, setup_scripts=\tests\appveyor.SQL2017.ps1,\tests\appveyor.SQL2016.ps1
14 tests in 7 min 17 sec

scenario=2016, main_instance=localhost\SQL2016, setup_scripts=\tests\appveyor.SQL2016.ps1
257 tests in 25 min 48 sec

scenario=2016_2017, main_instance=localhost\SQL2017,localhost\SQL2016, setup_scripts=\tests\appveyor.SQL2017.ps1,\tests\appveyor.SQL2016.ps1
73 tests in 10 min 33 sec

scenario=default, main_instance=localhost\SQL2008R2SP2,localhost\SQL2016, setup_scripts=\tests\appveyor.SQL2008R2SP2.ps1,\tests\appveyor.SQL2016.ps1
256 tests in 15 min 44 sec

I would try to combine "scenario=service_restarts" and "scenario=2016_2017" as they use the same instances.

Then we have a free VM to start migrating tests to.

[andreasjordan]

I updated the original post so this does not get too long and is more of a view of the current status...