SOA Testing and xFIT
Perspective on SOA & Services Testing
Service testing - Scenario 1 - front-end perspective
Some issues which come as package with this kind of testing are:
- Front-end is not reliable testing platform - GUI keeps changing
- Duplicated testing - a service may be used in multiple places
- Does not test the Service Level Agreements
Example: If a service is guaranteed to return in less than 2sec, testing such a scenario from front end is not possible and moreover latency can be anywhere.
Governance Issues in Front end testing
Shared business services may be developed and maintained by an independent team - hence testing must happen independently of front-end applications.
Testing typically involves sending a request message and receiving a response message and then the test is declared Pass/Fail based on the validation of response message.
Technical Issues in Simplistic Approach
Business services often depend on legacy enterprise applications and even systems external to an enterprise. The simplistic approach to service testing does not measure the effect of service invocation on the applications a service depends on Service invocations often kicking off complex work flows. Simplistic approach falls short here as well.
Service Invocation can impact several other external systems. Hence in the holistic approach testing the system will measure the state of all the external systems to see the test passed.
In addition to all these scenarios there are additional challenges in services testing that have to be addressed by the automated testing tool like timing issues, data variation challenges, dependency problems and complexity of the test bed.
Perspective on BPM Testing
BPM testing - Scenario 1 - Typical Technical ChallengesTesting semi-automated workflows
Workflows are often a combination of manual, app-assisted and automated steps. Simulating all of them using one testing technology may not be easy. Front-end applications used in workflows may be disparate technically - Web applications, windows applications, arcane terminal interfaces and mobile applications.
Timing issues
Business Process transitions may be highly time dependent Example: If a task is pending for more than 3 days, escalate to manager. Time lines involved are usually long - so, clock manipulation will be necessary if tests are to be done in a limited amount of time.
Creating exception situations
Business processes often depend upon disparate services that are backed up by a number of independent applications. In the real-world, state is not always consistent across all applications. Simulating the same during testing can be too tedious. In fact, business analysts often miss these kinds of possibilities when preparing test plans.
Business Rules Coverage
Business rule engines are often used to make decisions in workflows. It is important to ensure good test coverage of rules that drive a workflow. Creating the contexts that trigger different rules requires good understanding of the domain and cannot be easily automated.
BPM testing - Scenario 2 - Typical Managerial Challenges
The biggest problem faced by managers, testers and developers alike is the breadth of skills required to complete testing of BPM platforms.
Typical Challenges
Rework necessitated by false positives and negatives - primarily due to the complexity of setting up and validating test beds like - Database is down or the machine is unreachable. Hard coding affects the quality of testing as the test bed and end points cannot be changed. And because of this it is very difficult to simulate end points/ services and applications.
Efficient Ways to testing SOA & BPM scenarios
SOA and BPM provide flexible architecture and solutions - hence regression testing is important. Key to success of automated SOA & BPM testing
- Reduce the round trip from development to testing
- Reduce the skills required to do testing
- Reduce the complexity of test bed
- Using the vendor tools to solve specific technical needs
- Create a platform that supports the end-to-end processes of testing
Test bed management - Imperatives in SOA & BPM testing
- Changing from one state to another so that tests can be run
- Examples: Cleaning the data bases, adding specific data to the database, Cleaning the queues, preparing the data
- Validating the test bed
- Is the application up? Can we add a row to the table in the database?
- Observing the test bed to judge the test results
- Did the address get added to the database? Did the message get added to the Queue?
- Did an event get generated after 10 minutes?
- Reusing the machines for tests
- Effective reuse of the machines for running the tests
- Abstract representation of test bed so that tests can be run anywhere
Test ware management - Imperatives in SOA & BPM testing
- Test data and test scripts organization
- Organization of test scripts by feature or functionality
- Organization of test scripts based on importance
- Test script dependency
- Example: Unless the basic "add customer order" passes, do not test "add customer with multiple addresses"
- Test data generation
- Parameterized data generation
- Test results organization
- Successive runs should not overwrite results or logs
- Test results should identify test scripts and test data
Run time support - Imperatives in SOA & BPM testing
- Ability to combine test scripts to run test scenarios
- Basic health tests should pick all the basic tests from various functionalities
- Performance test may pick a few tests randomly and test them for performance
- Libraries to validate test results
- Ability to observe the applications and capture the results
- Functions to compare the test outputs with golden outputs
- Comparison can be complex - may involve business logic
- Ability to mock the interfaces
- Not all systems will be available for testing
- Should provide interfaces so that tests can be run
Scripting support - Imperatives in SOA & BPM testing
- Ability to use vendor tools to drive part of the tests
- Any vendor tool to stimulate the system
- Sending a message, placing a file, causing an event
- Any vendor tool to validate the response
- Framework to implicitly create a valid test bed
- Framework to validate the response - on the complete test bed
- Ability to create templates so that a class of tests can be done
- Reuse of the logic
- Example: "This test is exactly like that other test, except the protocol is https"
- Ability to create libraries, shortcuts, and patterns for script writers to use
xFIT 2.0 - HCL Framework for SOA and BPM Testing
xFIT: Automated SOA Testing Framework
HCL’s automated agent based test harness for SOA and BPM testing is known as xFIT (CROSSFIT). xFIT is used to enable middleware product testing and development of adapters & connectors for different middleware products.
- Establishes common framework for SOA and Integration Test Automation
- Separation of test case flow from details allows test developers to:
- Focus on automation of test cases
- Delegate and Reuse the effort needed to figure out details specific to each endpoint
- Portability of test suites from testbed to testbed facilitates use by those who were not involved in test development (e.g., solution developers)
- Reinforces Best Practices
- Capturing of manual test results
- Data driven testing
- Tailored for SOA and Integration testing
Following are the new features that we are adding to xFIT 2.0 which is going to be launched soon.
Testcase Editor: Creating testcases in xFIT is a tedious task and only a skilled xFIT resource can do it effectively now. This GUI based tool will reduce the skills required to create a testcase.
Installer: Installation and configuration of xFIT requires an understanding of the configuration files that xFIT uses and modification of them. I windows we are providing an installer (GUI Based) which will install and configure xFIT in a few clicks. In Linux an RPM and configuration scripts will be provided to do this task.
UI Improvements: xFIT Web UI will be modified to improve the usability. Some of the bugs which are reported on the UI will be fixed along with this.
Integration with Third-party Automation Scripts: xFIT will be enabled to invoke scripts created the test automation tools like, HP QTP, HP WinRunner and Borland Silk Test by adding a few cartridges.
