top
Requirement/Problem description
Any systematic test development starts with the synthesis of a suitable
test architecture that has to be configured for the real test
campaigns. The TT-Medal test system infrastructure and tool platform
have to be flexible enough for test architectures in the different
application domains.
Solution
A Corba based distributed test system infrastructure with test
coordination has been described. Furthermore test logging facilities
(incl. XML-based log event reports) that support a distributed test
architecture are proposed.
Prototype and tools
The concepts and required tools became part of the TTCN-3 toolset used
in TT-Medal.
Demonstrator
The proposed approaches became part of the TTCN-3 test system toolset
that are applied in the case study demonstrations.
References
[D1.1.1] Test Infrastructure and Architecture.
[D1.1.2] Developing testing methodologies and integration of testing
tools. For more information please click here.
[D2.6.1] Demonstrator showing the test infrastructure and architecture. Please click here.
top
Requirement/Problem description
There exist several techniques for the development of test cases from
software models like UML-diagrams and for the derivation of test data.
These techniques are, for instance, based on the exploration of labeled
transition systems (LTSs) or the Classification Tree Method (CTM).
Applying these methods should be automatized for conformance testing,
so that valid and sound test cases in TTCN-3 are machine-generated. The
soundness of the test cases must be validated, too. It should be
possible to integrate the tool chain into a model-based test approach.
Solution
For both test generation approaches, exploration of LTSs and CTM,
solutions were provided. The solution working on LTSs is based on data
abstraction to overcome the problem of state space explosion. In a
first step, abstract test cases with only little reference to actual
data are generated. Doing so, additional traces in the test case are
generated, which are pruned by reintroducing data in a second step.
This second step is based on constraint-solving and serves both as a
validator of the generated test cases and as the data derivation
mechanism [D1.2.1, D1.2.2, CIP05]. The target language for test case
generation is
TTCN-3.
CTM is supported by the
Classification Tree Editor (CTE). Here, it is
possible to categorize potential test data regarding several aspects
and then generating actual test data either based on its type, on the
way, the SUT has to be stimulated (behavior-based data generation), or
based on data templates [D1.2.2].
To apply test generation techniques on UML-based software models, a
mapping from types of UML-diagrams to the supported test design
techniques has been made [D1.2.2]. Partial evaluation of UML state
charts is
the basis for the Conformiq Test Generator (CTG).
Prototype and tools
For the LTS-based test generation approach, a data abstractor, a test
sequence selector, a generator of TTCN-3 test cases out of abstract
test cases in an intermediate format as well as a generator for test
data constraints (test validation and test data service) have been
developed and are in a prototypical stadium now. The generation process
of abstract test cases is based on the existing tool TGV. The CTM-based
test generation is supported by the tool CTE, which has been integrated
with TTCN-3 by a plugin for the IDE-framework Eclipse.
CTG combines both test generation from a software model and test
execution in an “on-the-fly” test approach. The tool is based on
extended UML state-chart diagrams and has been extended by TTCN-3
support in this project [Whi].
References
[D1.2.1] Towards model based test generation and validation for TTCN-3.
[D1.2.2.1] Definition of methods for automated test generation for TTCN-3 based test systems.
[D1.2.2.2] Definition of methods for automated test generation for TTCN-3 based test systems.
[Whi] A Vision for Automated Testing, whitepaper.
[Wor] Presentations on the First Dutch Workshop on Formal Testing
Techniques, www.cwi.nl/~ustin/DWFTT.html
.
[CIP05] Jens R. Calamé, Natalia Ioustinova, Jaco van de Pol,
Natalia
Sidorova: Data Abstraction for Test Generation, 12th
Asia-Pacific Software Engineering Conference, accepted for 2005.
top
Requirement/Problem description
The validity of test cases is not guaranteed. This applies to manually
and even to automatically generated test cases. For this reason, we
must validate their soundness and correctness. This task has to be
supported by tools.
Solution
Test case validation is an integral part of the test generation by data
abstraction solution and its tool chain. With this approach, we can
validate abstract test cases. It will be automatically checked if the
abstract test case can be instantiated with data according to the
specification. This procedure is based on constraint-solving [Wor,
CIP05].
Another solution of the project directly validates TTCN-3 code.
Examples of error detected by this approach are deadlocks or paths
without setting a verdict. This approach is based on symbolic reasoning
and automated theorem proving.
Finally, the encoding and decoding in the communication between a
testing system (TTCN-3) and its SUT has to be validated, too, since
both operations can contain failures. Two approaches for codec
validation have been developed in the project, (1) encode a value and
compare it against an expected value or (2) encode and then immediately
decode a value and check, whether it is identical [D1.2.3, Jeb05].
Prototype and tools
The validation of overapproximating test cases is supported by
prototypical tools. The deadlock and verdict check of TTCN-3 code is
currently under construction, and will be demonstrated at the upcoming
ITEA symposium.
References
[Wor] Presentations on the First Dutch Workshop on Formal Testing
Techniques, www.cwi.nl/~ustin/DWFTT.html
.
[CIP05] Jens R. Calamé, Natalia Ioustinova, Jaco van de Pol,
Natalia
Sidorova: Data Abstraction for Test Generation, 12th
Asia-Pacific Software Engineering Conference, accepted for 2005.
[D1.2.3] Requirements and Concepts for Test Validation of TTCN-3 Tests.
[Jeb05] Dirk Jebing: Optimized Testing of Encoding and Decoding
Functions, MSc thesis, University of Dortmund, April 2005.
top
Problem description
Not considering reuse factors, testers are steered to redevelop test
assets for almost similar interfaces, components and features. Also,
the absence of a documented set of test design knowledge in test
development results in situations where solutions are invented all over
again. However, an unsystematic way of reusing test assets may cause
more harms than real benefits. Ad-hoc modifications easily lead us a
situation, where we have to maintain multiple instances of nearly the
same piece of code.
Solution
The asset describes a process for producing reusable test assets and
reusing them in a systematic way. The asset includes the definition of
requirements, guidelines, and test patterns to avoid duplicate test
assets on TTCN-3 language and TTCN-3 test system level, and defines
methodological aspect for test patterns to gather test design solutions
and experiences.
Prototype
and tools
Prototype of a tool to analyze and reuse TTCN-3 data types defined in
existing test suites and to generate test data for new test suites.
Demonstrator
Case study of TTCN-3 test patterns generation for operation-based
systems and its impact on test reuse.
Case study of reusable test asset development for TTCN-3 test systems
and test scripts that are used in testing of heterogeneous distributed
system.
References
[D1.3.1.1]General Requirements of Reusable TTCN-3 Tests.
[D1.3.1.2]Guidelines and Patterns for Reusable TTCN-3 Tests.
[Dei04] Thomas Deiß, TTCN-3 for Large Systems, 3rd Systems
Testing
and Validation Workshop 2004, Paris
[D1.3.2]A Process Model for Developing and Utilizing
Reusable Test Assets.
[D1.3.3]Requirements Specification of Test System Supporting
Reuse.
[MM04] A. Mäntyniemi, P. Mäki-Asiala. Improving
Efficiency of Testing with Test Reuse. 3rd Software Product Line
Conference (SPLC 2004) - Workshop on Quality Assurance in Reuse Contexts
[RK04] P. Ruuska, M. Kärki. TTCN-3 Language Characteristics
in
Producing Reusable Test Software. Eighth International Conference on
Software Reuse (ICSR-8).
[KA04] M. Karinsalo, P. Abrahamsson. Software Reuse and the Test
Development Process: A Combined Approach. Eighth International
Conference on Software Reuse (ICSR-8).
[M557] Pekka Mäki-Asiala. Reuse of TTCN-3 Code. VTT Publications
557.
[MMK05] A. Mäntyniemi, P. Mäki-Asiala, M.
Kärki.
A Process Model for Development and Utilization of Reusable Test
Assets. SERP'05 - The 2005 International Conference on Software
Engineering Research and Practice.
[Kar] Mikko Karinsalo. Improving Reusability in TTCN-3 Test Systems: A
framework-based solution. Master's Thesis.
[VS04] Alain Vouffo, Ina Schieferdecker. TTCN-3 Test Patterns. FATES
2004.
[KK05] Matti Kärki, Mikko Karinsalo. "Reuse in TTCN-3 Test
System",
TTCN-3 User Conference, June 2005. www.ttcn-3.org
.
[SM05] S. Schulz and S. Mueller, "Test Implementation using TTCN-3
Libraries", TTCN-3 User Conference, June 2005. www.ttcn-3.org .
[ETSI05] ETSI DTS/MTS-IPT-0001, "Methods for Testing and Specification
(MTS); Internet Protocol Testing (IPT); IPv6 Testing: Methodology and
Framework", Sophia-Antipolis, 2005.
top
Requirement/Problem description
A test profile is a specific approach to use a specification language
for test description. In TT-medal two kinds of profiles are addressed:
(1) the test profile for the modeling language UML (U2TP), and (2) the
domain specific test profiles for TTCN-3.
Solution
TTCN-3 is a general test notation applicable in various industrial
domains with different requirements on testing. A profiling concept is
proposed for each of the application domain.
Prototype
and tools
A tool supported approach for the graphical editing of U2TP documents
and the translation to TTCN-3 files have been described and developed.
Demonstrator
For each of the industrial case studies a specific approach for using
TTCN-3 has been proposed and implemented in the case studies.
References
[D1.4.1.1] Testing Profiles.
[D1.4.1.2] Testing Profiles.
[D2.4.2]Analysis of the methods and processes for UML test profiles.
