IoT InterOp-WARE
A Heck of a Challenge
Latif Ladid, IPv6 Forum President
 |
| Latif Ladid, IPv6 Forum President |
By 2020, the Internet of Things (IoT) is expected
to connect 50 to 100 Billion smart things and
objects1&2, paving the way to great economic
opportunities and challenges3. Forrester4 expects
the digital universe to grow by a factor of 32 by 2020
compared to 2012. According to Bell labs5, in 2013, ICT
consumed about 6% of global energy at about 108.4
gigawatts (or 109 nuclear plants, or 19 times the energy
consumption of New York City).
According to Cisco6, by
2018, over half of all IP traffic will originate with non-PC
devices and the machine-to-machine (M2M) traffic will
grow at an annual rate of 84%. According to the IERC
and the ITU, the largest barriers hindering the Internet of
Things market development is the lack of interoperability 7.
Internet of Things, M2M and oneM2M are major
emerging trends at the present and for the coming years of
the communication society. Several challenges have already
been identified in terms of governance, security/privacy and
convergence.
The current approach to IoT is obsolete as it still uses old
Internet models like gateways or NATed/Intranet or more
IntraNAT. We are simply making old mistakes of the Internet
in IoT today. This won’t move IoT to where it should go.
In an Industry Forum session at Globecom with Vint Cerf
and Geoff Mulligan, IPSO Chair and inventor of 6LoWPAN
and we came to the conclusion that IoT, M2M and oneM2M
are digging themselves into the same hole.
The Future Interop (F-Interop) project funded by the
European Commission has just started its research work
back in November 2015 for three years to research the
issues of the new IoT interop paradigm.
Current applications often follow the “one-networkone-
application” paradigm. We are shifting away from this
paradigm as IoT applications often require merging different
solutions from a vertical into a horizontal market. This
results in an increased need for interoperable solutions,
where “umbrella” networks are used to carry a large number
of independent traffic.
 |
| Figure 1. F-Interop addresses the standards valuecreation
process |
In order to be widely adopted, new technologies, products
and solutions go through the steps illustrated in Figure 1:
- Standardization: stakeholders discuss and align their
views on a common standard.
- Conformance: test and validate that an implementation
conforms to the standard.
- Optimization: in terms of Quality of Service, scalability,
energy consumption, etc.
- Market Launch: the solution is ready for initial roll-out
into the market.
Each phase requires extensive testing. Verifying
conformance to a standard, and verifying interoperability
with other vendors typically requires interoperability testing
events and interaction with third-party certification labs.
The traditional approach is to organize interoperability
events8, where different vendors meet face-to-face to test interoperability by going through an exhaustive
list of “interoperability tests”. Since this approach relies
on face-to-face meetings, it has a direct impact on the
speed of development of a standard, the cost of products
implementing such standards, and their time-to-market. This
is all the more problematic as it is commonplace for some
tests to fail for trivial reasons (packet formats, addressing
length, etc.). When this happens, vendors have to return
home, change their implementation, and wait until the next
event to re-test, usually months later.
The consequence is that:
- The current process is extremely labor-intensive, as
engineers travel across the globe only to find out they
need to make a minor fix;
- The cost associated with engineering time and travel
expenses is often too high for SMEs;
- Time-to-market is unnecessarily stretched, giving
vendors who want to adopt emerging standards a
disadvantage compared to vendors who come to
market with entirely proprietary solutions.
This process often scares vendors away from standardsbased
solutions. The end-user is therefore often and
unnecessarily locked into proprietary solutions, as
standards-based products haven’t hit the market when they
bought their first product.
| The F-Interop project proposes a ground-up redesign
of the way conformance, interoperability and performance
testing of protocols and standards is done. A novel
architecture centered on the cloud will enable the vast
majority of these tests to be done remotely. |
|
The outcome is that typically only one interop event
will be required for a company to develop standardsbased
interoperable products, cutting time-to-market by
6-12 months, and significantly lowering the engineering/
financial overhead.
Interoperability tests require an implementation to
interact with other devices running another implementation
of the same standard. Scalability tests require a large
number of such devices. It is impractical for an SME to buy a
large number of devices “just” for testing.
The European FIRE+ community provides several largescale
open testbeds, largely used by the academic research
community. We propose to position them at the center of
the cloud-based testing architecture. We are targeting the
testbeds below, covering the full spectrum of networked
devices. Partners of the consortium play the lead role in
each of the following testbeds.
- Fed4FIRE (http://www.fed4fire.eu/testbeds) is a
federation of (at the moment of writing) 24 FIRE+
testbeds spread over Europe, bringing together
technologies such as cloud, IoT/wireless/wireless mobile,
LTE, openflow, network emulation, all accessible through
the same toolset and account. All testbeds speak the
same API (Aggregate Manager API, XMLRPC based)
and are monitored continuously on their availability.
The total number of nodes (physical servers, physical
wireless devices, physical switches) is around 1000.
- OneLab (http://onelab.eu) is an experimental facility
made up of a federation of future Internet testbeds,
which together offer large-scale experimentation across
heterogeneous resources. The four platforms federated
under OneLab include:
- FIT IoT-Lab (http://www.iot-lab.info), a 2728-node
testbed of wireless devices deployed in France,
representative of tomorrow’s low-power wireless mesh
networks.
- FIT CorteXlab (http://www.cortexlab.fr), a cognitive
radio testbed with 80 wireless nodes deployed in Lyon,
France
- FIT NITOS-Lab (http://fit-nitos.fr), a wireless testbed
with over 100 nodes divided between sites in Volos
(Greece), Paris, Sophia-Antipolis, and Evry (France)
- PlanetLab Europe (PLE, http://planet-lab.eu) an
Internet overlay testbed with 300 servers located in
150 different sites across Europe.
The FIT IoT-Lab testbeds will primarily be used in
F-Interop; however, all of the testbeds federated under
OneLab will be accessible through the F-Interop platform
and will be available to users for experiment deployment.
This will enable those users who wish to extend their testing
beyong the IoT domain to so do.
- The European IoT lab project (http://www.iotlab.eu)
federates several IoT-related testbeds together with
crowd sourcing and crowd sensing capabilities. It aims at
researching the potential of crowdsourcing to extend IoT
testbed infrastructure for multidisciplinary experiments
with more end-user interactions. On the IoT testbed side,
in includes several testbed, including smart campuses
and smart buildings, as well as a smart office testbed.
The platform gathers a large heterogeneity of IoT devices
enabling F-Interop to expose all sorts of devices using
distinct communication protocols. On the user side, the
platform enables interaction with end-users distributed
across Europe.
| |
| The F-Interop project will develop a set of tools
enabling new forms of online testing on top of the existing
FIRE+ testbeds. This includes tests that require interactions
between a large number of devices, to verify scalability,
end-to-end Quality of Service (QoS) and energy efficiency.
Each of the federated platforms provides a different
type of hardware, depending on the testbed, allowing
for expansive testing scenarios to suit the needs of all
experimenters. |
|
This places the FIRE+ testbeds at the heart of the
F-Interop testing infrastructure, thereby significantly
increasing the usage of the FIRE+ testbeds by the
European industry.
F-Interop will research, develop and enable innovative
tools to support the development of new technologies and
standards, from their genesis to their maturity. It will provide
cloud-based remote interoperability and conformance testing
tools, as well as tools for testing and measuring the scalability,
end-to-end Quality of Service (QoS) and energy efficiency of
any tested solution. This will extend the capability of existing
FIRE+ testbeds (Fed4Fire, OneLab, IoT lab), which will play a
central role the proposed testing architecture.
Objectives of F-Interop
The goal of F-Interop is to design and develop a FIRE+-
based online conformance, interoperability and
performance test and validation platform to support
researchers, product development by SME, and
standardization processes. More specifically, F-Interop will:
1. Extend FIRE+ through research on an online testing
tools
- Conformance testing tools: an Implementation Under
Test (IUT) located at the vendor remotely connects to a
test system that verifies its conformance to a standard.
- Interoperability testing tools: remote IUTs run
interoperability tests through a variety of real-life
scenarios, by interacting with (or directly on) FIRE+
testbeds.
- Scalability tests, by leveraging the FIRE+
infrastructure, test and analyses the capacity of a
given implementation to handle large numbers of
connections and interactions.
- End-to-end Quality of Service (QoS) and Quality of
Experience (QoE) tests: through interactions between
FIRE+ testbed(s), verify QoS and QoE performance
metrics such as end-to-end reliability, latency and jitter.This is particularly applicable to SDN/NFV standards.
- Energy efficiency tests, including when the IUT is
deployed at large scale.
2. Integrate several FIRE+ testbeds into a shared
platform “Testbed as a Service”
- Mutualize 3 FIRE+ federating testbeds: Fed4FIRE,
OneLab and IoT lab, bringing together over 32 testbeds
and 4755 nodes.
- Design a common reference architecture model
for on-line test and standardization support in close
collaboration with ETSI, IETF, W3C, ITU and IEEE.
- Security by design: the architecture will consider the
security and privacy of the test results, gathered data
and any test code/scripts passed over shared networks.
- Develop a “Testbed as a Service” (TBaaS) model: based on virtualization and Software as a Service
(SaaS), simplify remote access and interaction with the
experimental platform.
- Adopt a future-proof and flexible design, allowing
seamless integration of future testbeds.
3. Support standardization and enable closer
cooperation with the industry
- Close collaboration with standardization bodies,
directly contributing to three global emerging standards:
oneM2M, IETF 6TiSCH and Web of Things (W3C).
- Supporting and enabling new online certification
and labelling mechanisms with a direct use by the
IPv6 Ready logo.
- Enabling easier participation of researchers and
industry in the standardization process.
- Open call for SMEs and developers to use and enrich
the developed testing tools.
The clear set of objectives of F-Interop and the new tools
developed during the project will have a significant impact
on standardization activities and the availability of compliant
products:
- Standardization support with emerging standards at
ETSI, IETF and W3C. The flexible F-Interop architecture
will enable new standards to be added to the available
test plans. Standards and test plans will be developed
side-by-side resulting in faster publication of better
standards. This will also allow certification and
labeling for those standards.
- Researchers and SMEs can remotely test
implementations, products and applications.
By streamlining conformance, interoperability and
performance test, we evaluate the time-to-market of
standards-based will be reduced by 6-12 months.
Faster availability of standards-based products results
in wider adopt standards and a larger portion of
interoperable products.
- Improving European leadership and influence on
global standardization processes.
The consortium includes the leading figures of the
3 targeted FIRE+ testbeds (Fed4Fire, OneLab, IoT lab). Its
members bring expertise from over research projects on
IoT, with strong links with standardization bodies (ETSI,
IETF, IEEE, ITU, etc.) and international fora (IoT Forum, IPv6
Forum, etc.). This will ensure the alignment of the research
with the industry needs, and an effective exploitation and
transfer of F-Interop results into standards, products and
services.
Overview: Best Practice in Conformance and
Interoperability Testing
Testing is arguably the most important activity during the
development of an implementation. When implementing
an existing standard, two types of tests are typically done:
conformance tests and interoperability. Today’s best
practices on both types of tests are highlighted in Sections
1.3.1.1 and 1.3.1.2, resp.
A typical test (including conformance and
interoperability test) involve an Implementation Under Test
(IUT) and a “test harness”. The test harness (also referred
to as “test execution engine” or “test script repository”)
interfaces to various devices, and monitors their activity
while executing a set of tests. The ISO/IEC 9646 series and the ETSI document EG202 237 describe a generic approach
for interoperability testing. These foundational documents
will be followed by the F-Interop project.
 |
| Figure 2. Relationship between Standards, Validation &
Testing |
Conformance testing
According to ISO/IEC 9646, the components of conformance
testing specifications are the Abstract Test Method/
Architecture, Test Purposes, the Abstract Test Suite (ATS) and
its Test Suite Structure, and Implementation Conformance
Statements (ICS).
The Abstract Test Architecture describes how an
Implementation Under Test (IUT) is tested independently
of any Means of Testing. It includes the IUT, the testing
interfaces called PCO (Point of Control and Observation)
and the chosen category of testing from the four
possibilities: Remote, Local, Distributed and Coordinated.
The architecture is a description of the interfaces where
test system and IUT are attached, the relationship between
the IUT, the type of test system, cabling/connection
requirements, and test and IUT operator locations. In some
cases, the IUT cannot be directly connected to the test
system; it is then tested through a SUT (System Under Test)
in which the IUT resides.
A Test Purpose is a plain-English description of a
testing objective focusing on a single requirement or a set
of requirements to be tested on the IUT. An abstract test
case is a complete and independent specification (at the
Abstract Test Method’s level of abstraction) of the actions
required to achieve a specific test purpose. It is abstract
insofar that it does not depend on the specific hardware
and/or interfaces used for test execution. Run-time
interfaces, software libraries and drivers are required for
interfacing the executable tests to the test equipment. The
Abstract Test Suite (ATS) is a collection of abstract test cases
organized into a Test Suite Structure, allowing multiple test
cases to be grouped into logical test groups; a test group
verifying the conformance of a particular part of a standard.
The ICS (Implementation Conformance Statements) is a
questionnaire filled in by the supplier (the vendor) that
provides a statement about which capabilities and options
of the standard have been implemented. It is used for
selecting and parameterizing the tests to be run at the end
on the IUT.
 |
| Figure 3. From standard to
executable test suite |
Figure 3 shows how a standard is translated into an
executable test suite. Figure 4 then shows how this test
suite, when executed against an IUT, is used for labeling and
certification.
 |
| Figure 4. General overview of
conformance testing execution |
Interoperability Testing
A framework for interoperability testing follows the same
process as conformance testing, except that there may
be several standards for abstract interoperability test
generation, and that the interoperability test suites are
executed on two or more devices (not only one as in
conformance testing).
The purpose of interoperability testing is to demonstrate
that an IUT works with other products, and to prove that
end-to-end functionality between (at least) two devices per
the implemented standard(s).
Interoperability test specifications include:
- A test architecture, an abstract framework within
which any SUT scenario or configuration (a SUT being
composed of a number of IUTs coming from different
suppliers) will fit. This test architecture must clearly
identify the different IUTs and their functional role, the
communication paths between them and protocols, APIs and/or data models used for communication.
- An Interoperable Functions Statement (IFS) which
identifies standardized functions that an IUT may
support. An IFS is used as a pro-forma by suppliers
to state which functions their IUT supports when
interoperating with other IUTs.
- A Test Description (TD) which specifies detailed
steps to be followed to achieve stated test objectives,
usually written in a structured and tabulated natural
language which allows a manual execution. Automated
interoperability Test Descriptions are also possible.
A Test Environment can be required: a combination of
other equipment and procedures enabling interoperability
testing. The Test Environment ensures the selection,
interpretation and execution of the Test Descriptions,
coordination and synchronization of the actions on the test
interfaces, and provides mechanisms for logging, monitoring
and observing the interactions among IUTs.
 |
| Figure 5. Different levels of interoperability |
A taxonomy of interoperability tests
Figure 5 illustrates the four categories of interoperability
tests, following ETSI’s taxonomy:
- Technical Interoperability is associated with hardware/
software components, systems and platforms that
enable machine-to-machine communication. Technical
interoperability is centered on (communication)
protocols and the infrastructure needed for those
protocols to operate.
- Syntactical Interoperability is associated with data
formats. Messages exchanged through a communication
protocol need to conform to the agreed-upon syntax and
encoding. High-level transfer syntaxes such as HTML,
XML or ASN.1 are used to represent the exchanged data.
- Semantic interoperability is associated with the meaning
of content, and concerns human rather than machine
interpretation of the content. There needs to be a
common understanding of the meaning of the content
(information) being exchanged.
- Organizational interoperability is the ability of
organizations to effectively communicate over different
information systems, infrastructures, geographic regions
and cultures. Organizational interoperability depends
on successful technical, syntactical and semantic
interoperability.
| |
F-Interop focuses on conformance, technical
interoperability testing (including syntactical
interoperability checking and conformance testing) and
semantic interoperability testing.
The project will develop complementary tools that help
testing scalability, Quality of Service (QoS) and Quality of
Experience (QoE), as well as energy consumption, using the
FIRE+ testbeds. |
|
The complexity of constrained network testing
F-interop aims to develop several tools for testing different
technologies. Among the tools proposed, particularly
challengeable will be the design and implementation of
those testing constrained protocols for low-power wireless
mesh networks, (e.g., IETF 6TiSCH, 6LoWPAN and CoAP).
The added complexity of testing those standards (in
comparison for example to testing IPv6 or HTTP) is due to
the constrained nature of the wireless devices and (lossy)
environments, where they run. Therefore, the testing tools
will be developed according to the strict requirements in
term of time synchronization, and energy consumption.
Interop tests, plugfests and plugtests
Partners in the consortium have a good experience in
organizing interop tests, plugfests and plugtests. The most
important such events are:
- TODO ETSI: CoAP, 6LoWPAN plugtests. Dates, #
participants, input ot WGs (3-4 lines)
- ETSI is organizing the first IETF 6TiSCH plugtest, to be
held at IETF93 Prague in July 2015. The purpose of the
event is to test interoperability of implementations of
draft-ietf-6tisch-minimal. 10 participants are expected.
- Inria co-organized an IPSO plugtest even in 2011
around IEEE802.15.4e and CoAP, held in Santa Clara
over the course of two days, with 5 implementations
from industrial/academic partners. With one company
participating remotely from France, this event resulted
in the first trans-Atlantic communication between CoAPenabled
low-power devices.
- The EANTC and Upperside Conferences invited
interested vendors to a public multi-vendor
interoperability test and showcase at the MPLS SDN
World Congress 2015 (http://www.uppersideconferences.
com/mpls-sdn/index.html). Together with the NFV &
SDN Summit and the V6 World Congress 2015 taking
place at the same time, EANTC presented the results
live to the combined audience of all three conferences.
By presenting a realistic example of a next generation interoperable service provider network, participants
demonstrated the maturity and applicability of their
devices and solutions.
- Inria participated in several specialist task forces
(STF) at ETSI related to Test Specifications for IPv6
Interoperability. Since 2001, Inria brings expertise to the
ETSI Plugtest services for the organization of the several
sessions of IPv6 related interoperability events. More
recently, in 2012, Inria contributed to the one 6LoWPAN
and the two first CoAP interoperability Plugtests events
organized by ETSI.
- Since 1995, the EANTC organizes interoperability test
events, usually 1-3 times a year, executed in the EANTC
lab. Results are demonstrated at shows (including the
Carrier Ethernet World Congress).
- Inria is a founding and active member of the worldwide
IPv6 Ready Logo certification program (supported by
the IPv6 Forum). It is the representative of Europe in this
program, both as technical expert for test specification
development, and technical advisor for test results
analysis in the certification process. In 2006-2008, Inria
participated in the Go4IT Project (the FP7 European
project on TTCN-3 based Test Tools and Services for
IPv6 protocols) and more recently in the PROBE-IT
(Pursuing Roadmap and Benchmark in IoT) dedicated to
deployment, testing and interoperability issues of IoT.
- Inria co-organized 2 plugfests around IETF 6TiSCH
technology (IETF89 London, March 2014, IETF90 Toronto,
July 2014), each attended by 8 academic/industrial teams.
The purpose was to bring together implementations
around 6TiSCH technology, including demonstrations of a
federated backbone by Cisco and Linear Technology, and an
IEEE802.15.4e stack by the OpenWSN team.
The main objective of these interoperability events is the
validation of the specification which gives a higher chance
for interoperable implementations, and allowed vendors
to detect and correct errors in a controlled manner. At the
same time, they are an opportunity for vendors to test their
implementations and prototypes, to reduce time to market,
to promote technology and to originate new ideas.
F-Interop remote interoperability tools will allow
validating of specifications and implementations in
a quicker and more responsive manner which, will
contribute to the technology success.
F-Interop will organize Plugtest event in which F-Interop
interoperability tools will be used and validated. |
|
References:
1 “Vision and Challenges for Realising the Internet of Things”,
Cluster of European Research Projects on the Internet
of Things (CERP-IoT), page 13, March 2010. [available online, consulted 17-Feb-2015].
2 “Ericsson CEO Predicts 50 Billion Internet Connected
Devices by 2020”, GigaOm, April 2010 [available online, consulted 17-Feb-2015].
3 “Disruptive technologies: Advances that will transform
life, business, and the global economy”, McKinsey Global
Institute, May 2013. [available online, consulted 17-Feb-2015]
4 Jittender Miglani’s blog on Forrester.com. [available online, consulted 17-
Feb-2015]
5 Global “What If” Analyzer of neTwork energy consumpTion
(G.W.A.T.T.), Alcatel-Lucent. [available online, consulted 17-Feb-2015]
6 “The Zettabyte Era—Trends and Analysis”, Cisco Systems,
June 2014. [available online, consulted 17-
Feb-2015]
7 “The Internet of Things”, Internet Report, International
Telecommunication Union (ITU), 2005. [available online,
consulted 17-Feb-2015]
8 Interoperability events are branded as “PlugtestsTM” when
organized by ETSI.
For more information visit:
www.ipv6forum.com
www.f-interop.eu |
