Issue 26 Articles
IPv6 Forum logo   F-Interop logo

IoT InterOp-WARE
A Heck of a Challenge

Latif Ladid, IPv6 Forum President

PDF icon Download article as PDF
Latif Ladid, IPv6 Forum President
Latif Ladid, IPv6 Forum President

By 2020, the Internet of Things (IoT) is expected to connect 50 to 100 Billion smart things and objects1&2, paving the way to great economic opportunities and challenges3. Forrester4 expects the digital universe to grow by a factor of 32 by 2020 compared to 2012. According to Bell labs5, in 2013, ICT consumed about 6% of global energy at about 108.4 gigawatts (or 109 nuclear plants, or 19 times the energy consumption of New York City).

According to Cisco6, by 2018, over half of all IP traffic will originate with non-PC devices and the machine-to-machine (M2M) traffic will grow at an annual rate of 84%. According to the IERC and the ITU, the largest barriers hindering the Internet of Things market development is the lack of interoperability 7.

Internet of Things, M2M and oneM2M are major emerging trends at the present and for the coming years of the communication society. Several challenges have already been identified in terms of governance, security/privacy and convergence.

The current approach to IoT is obsolete as it still uses old Internet models like gateways or NATed/Intranet or more IntraNAT. We are simply making old mistakes of the Internet in IoT today. This won’t move IoT to where it should go. In an Industry Forum session at Globecom with Vint Cerf and Geoff Mulligan, IPSO Chair and inventor of 6LoWPAN and we came to the conclusion that IoT, M2M and oneM2M are digging themselves into the same hole.

The Future Interop (F-Interop) project funded by the European Commission has just started its research work back in November 2015 for three years to research the issues of the new IoT interop paradigm.

Current applications often follow the “one-networkone- application” paradigm. We are shifting away from this paradigm as IoT applications often require merging different solutions from a vertical into a horizontal market. This results in an increased need for interoperable solutions, where “umbrella” networks are used to carry a large number of independent traffic.

Figure 1. F-Interop addresses the standards valuecreation process
Figure 1. F-Interop addresses the standards valuecreation process

In order to be widely adopted, new technologies, products and solutions go through the steps illustrated in Figure 1:

  1. Standardization: stakeholders discuss and align their views on a common standard.
  2. Conformance: test and validate that an implementation conforms to the standard.
  3. Optimization: in terms of Quality of Service, scalability, energy consumption, etc.
  4. Market Launch: the solution is ready for initial roll-out into the market.

Each phase requires extensive testing. Verifying conformance to a standard, and verifying interoperability with other vendors typically requires interoperability testing events and interaction with third-party certification labs. The traditional approach is to organize interoperability events8, where different vendors meet face-to-face to test interoperability by going through an exhaustive list of “interoperability tests”. Since this approach relies on face-to-face meetings, it has a direct impact on the speed of development of a standard, the cost of products implementing such standards, and their time-to-market. This is all the more problematic as it is commonplace for some tests to fail for trivial reasons (packet formats, addressing length, etc.). When this happens, vendors have to return home, change their implementation, and wait until the next event to re-test, usually months later.

The consequence is that:

  • The current process is extremely labor-intensive, as engineers travel across the globe only to find out they need to make a minor fix;
  • The cost associated with engineering time and travel expenses is often too high for SMEs;
  • Time-to-market is unnecessarily stretched, giving vendors who want to adopt emerging standards a disadvantage compared to vendors who come to market with entirely proprietary solutions.

This process often scares vendors away from standardsbased solutions. The end-user is therefore often and unnecessarily locked into proprietary solutions, as standards-based products haven’t hit the market when they bought their first product.

The F-Interop project proposes a ground-up redesign of the way conformance, interoperability and performance testing of protocols and standards is done. A novel architecture centered on the cloud will enable the vast majority of these tests to be done remotely.

The outcome is that typically only one interop event will be required for a company to develop standardsbased interoperable products, cutting time-to-market by 6-12 months, and significantly lowering the engineering/ financial overhead.

Interoperability tests require an implementation to interact with other devices running another implementation of the same standard. Scalability tests require a large number of such devices. It is impractical for an SME to buy a large number of devices “just” for testing.

The European FIRE+ community provides several largescale open testbeds, largely used by the academic research community. We propose to position them at the center of the cloud-based testing architecture. We are targeting the testbeds below, covering the full spectrum of networked devices. Partners of the consortium play the lead role in each of the following testbeds.

  • Fed4FIRE ( is a federation of (at the moment of writing) 24 FIRE+ testbeds spread over Europe, bringing together technologies such as cloud, IoT/wireless/wireless mobile, LTE, openflow, network emulation, all accessible through the same toolset and account. All testbeds speak the same API (Aggregate Manager API, XMLRPC based) and are monitored continuously on their availability. The total number of nodes (physical servers, physical wireless devices, physical switches) is around 1000.
  • OneLab ( is an experimental facility made up of a federation of future Internet testbeds, which together offer large-scale experimentation across heterogeneous resources. The four platforms federated under OneLab include:

- FIT IoT-Lab (, a 2728-node testbed of wireless devices deployed in France, representative of tomorrow’s low-power wireless mesh networks.

- FIT CorteXlab (, a cognitive radio testbed with 80 wireless nodes deployed in Lyon, France

- FIT NITOS-Lab (, a wireless testbed with over 100 nodes divided between sites in Volos (Greece), Paris, Sophia-Antipolis, and Evry (France)

- PlanetLab Europe (PLE, an Internet overlay testbed with 300 servers located in 150 different sites across Europe.

The FIT IoT-Lab testbeds will primarily be used in F-Interop; however, all of the testbeds federated under OneLab will be accessible through the F-Interop platform and will be available to users for experiment deployment. This will enable those users who wish to extend their testing beyong the IoT domain to so do.

  • The European IoT lab project ( federates several IoT-related testbeds together with crowd sourcing and crowd sensing capabilities. It aims at researching the potential of crowdsourcing to extend IoT testbed infrastructure for multidisciplinary experiments with more end-user interactions. On the IoT testbed side, in includes several testbed, including smart campuses and smart buildings, as well as a smart office testbed. The platform gathers a large heterogeneity of IoT devices enabling F-Interop to expose all sorts of devices using distinct communication protocols. On the user side, the platform enables interaction with end-users distributed across Europe.
The F-Interop project will develop a set of tools enabling new forms of online testing on top of the existing FIRE+ testbeds. This includes tests that require interactions between a large number of devices, to verify scalability, end-to-end Quality of Service (QoS) and energy efficiency. Each of the federated platforms provides a different type of hardware, depending on the testbed, allowing for expansive testing scenarios to suit the needs of all experimenters.

This places the FIRE+ testbeds at the heart of the F-Interop testing infrastructure, thereby significantly increasing the usage of the FIRE+ testbeds by the European industry.

F-Interop will research, develop and enable innovative tools to support the development of new technologies and standards, from their genesis to their maturity. It will provide cloud-based remote interoperability and conformance testing tools, as well as tools for testing and measuring the scalability, end-to-end Quality of Service (QoS) and energy efficiency of any tested solution. This will extend the capability of existing FIRE+ testbeds (Fed4Fire, OneLab, IoT lab), which will play a central role the proposed testing architecture.

Objectives of F-Interop

The goal of F-Interop is to design and develop a FIRE+- based online conformance, interoperability and performance test and validation platform to support researchers, product development by SME, and standardization processes. More specifically, F-Interop will:

1. Extend FIRE+ through research on an online testing tools

- Conformance testing tools: an Implementation Under Test (IUT) located at the vendor remotely connects to a test system that verifies its conformance to a standard.

- Interoperability testing tools: remote IUTs run interoperability tests through a variety of real-life scenarios, by interacting with (or directly on) FIRE+ testbeds.

- Scalability tests, by leveraging the FIRE+ infrastructure, test and analyses the capacity of a given implementation to handle large numbers of connections and interactions.

- End-to-end Quality of Service (QoS) and Quality of Experience (QoE) tests: through interactions between FIRE+ testbed(s), verify QoS and QoE performance metrics such as end-to-end reliability, latency and jitter.This is particularly applicable to SDN/NFV standards.

- Energy efficiency tests, including when the IUT is deployed at large scale.

2. Integrate several FIRE+ testbeds into a shared platform “Testbed as a Service”

- Mutualize 3 FIRE+ federating testbeds: Fed4FIRE, OneLab and IoT lab, bringing together over 32 testbeds and 4755 nodes.

- Design a common reference architecture model for on-line test and standardization support in close collaboration with ETSI, IETF, W3C, ITU and IEEE.

- Security by design: the architecture will consider the security and privacy of the test results, gathered data and any test code/scripts passed over shared networks.

- Develop a “Testbed as a Service” (TBaaS) model: based on virtualization and Software as a Service (SaaS), simplify remote access and interaction with the experimental platform.

- Adopt a future-proof and flexible design, allowing seamless integration of future testbeds.

3. Support standardization and enable closer cooperation with the industry

- Close collaboration with standardization bodies, directly contributing to three global emerging standards: oneM2M, IETF 6TiSCH and Web of Things (W3C).

- Supporting and enabling new online certification and labelling mechanisms with a direct use by the IPv6 Ready logo.

- Enabling easier participation of researchers and industry in the standardization process.

- Open call for SMEs and developers to use and enrich the developed testing tools.

The clear set of objectives of F-Interop and the new tools developed during the project will have a significant impact on standardization activities and the availability of compliant products:

- Standardization support with emerging standards at ETSI, IETF and W3C. The flexible F-Interop architecture will enable new standards to be added to the available test plans. Standards and test plans will be developed side-by-side resulting in faster publication of better standards. This will also allow certification and labeling for those standards.

- Researchers and SMEs can remotely test implementations, products and applications. By streamlining conformance, interoperability and performance test, we evaluate the time-to-market of standards-based will be reduced by 6-12 months. Faster availability of standards-based products results in wider adopt standards and a larger portion of interoperable products.

- Improving European leadership and influence on global standardization processes.

The consortium includes the leading figures of the 3 targeted FIRE+ testbeds (Fed4Fire, OneLab, IoT lab). Its members bring expertise from over research projects on IoT, with strong links with standardization bodies (ETSI, IETF, IEEE, ITU, etc.) and international fora (IoT Forum, IPv6 Forum, etc.). This will ensure the alignment of the research with the industry needs, and an effective exploitation and transfer of F-Interop results into standards, products and services.

Overview: Best Practice in Conformance and Interoperability Testing

Testing is arguably the most important activity during the development of an implementation. When implementing an existing standard, two types of tests are typically done: conformance tests and interoperability. Today’s best practices on both types of tests are highlighted in Sections and, resp.

A typical test (including conformance and interoperability test) involve an Implementation Under Test (IUT) and a “test harness”. The test harness (also referred to as “test execution engine” or “test script repository”) interfaces to various devices, and monitors their activity while executing a set of tests. The ISO/IEC 9646 series and the ETSI document EG202 237 describe a generic approach for interoperability testing. These foundational documents will be followed by the F-Interop project.

Figure 2. Relationship between Standards, Validation & Testing
Figure 2. Relationship between Standards, Validation & Testing

Conformance testing

According to ISO/IEC 9646, the components of conformance testing specifications are the Abstract Test Method/ Architecture, Test Purposes, the Abstract Test Suite (ATS) and its Test Suite Structure, and Implementation Conformance Statements (ICS).

The Abstract Test Architecture describes how an Implementation Under Test (IUT) is tested independently of any Means of Testing. It includes the IUT, the testing interfaces called PCO (Point of Control and Observation) and the chosen category of testing from the four possibilities: Remote, Local, Distributed and Coordinated. The architecture is a description of the interfaces where test system and IUT are attached, the relationship between the IUT, the type of test system, cabling/connection requirements, and test and IUT operator locations. In some cases, the IUT cannot be directly connected to the test system; it is then tested through a SUT (System Under Test) in which the IUT resides.

A Test Purpose is a plain-English description of a testing objective focusing on a single requirement or a set of requirements to be tested on the IUT. An abstract test case is a complete and independent specification (at the Abstract Test Method’s level of abstraction) of the actions required to achieve a specific test purpose. It is abstract insofar that it does not depend on the specific hardware and/or interfaces used for test execution. Run-time interfaces, software libraries and drivers are required for interfacing the executable tests to the test equipment. The Abstract Test Suite (ATS) is a collection of abstract test cases organized into a Test Suite Structure, allowing multiple test cases to be grouped into logical test groups; a test group verifying the conformance of a particular part of a standard. The ICS (Implementation Conformance Statements) is a questionnaire filled in by the supplier (the vendor) that provides a statement about which capabilities and options of the standard have been implemented. It is used for selecting and parameterizing the tests to be run at the end on the IUT.

Figure 3. From standard to executable test suite
Figure 3. From standard to executable test suite

Figure 3 shows how a standard is translated into an executable test suite. Figure 4 then shows how this test suite, when executed against an IUT, is used for labeling and certification.

Figure 4. General overview of conformance testing execution
Figure 4. General overview of conformance testing execution

Interoperability Testing

A framework for interoperability testing follows the same process as conformance testing, except that there may be several standards for abstract interoperability test generation, and that the interoperability test suites are executed on two or more devices (not only one as in conformance testing).

The purpose of interoperability testing is to demonstrate that an IUT works with other products, and to prove that end-to-end functionality between (at least) two devices per the implemented standard(s).

Interoperability test specifications include:

- A test architecture, an abstract framework within which any SUT scenario or configuration (a SUT being composed of a number of IUTs coming from different suppliers) will fit. This test architecture must clearly identify the different IUTs and their functional role, the communication paths between them and protocols, APIs and/or data models used for communication.

- An Interoperable Functions Statement (IFS) which identifies standardized functions that an IUT may support. An IFS is used as a pro-forma by suppliers to state which functions their IUT supports when interoperating with other IUTs.

- A Test Description (TD) which specifies detailed steps to be followed to achieve stated test objectives, usually written in a structured and tabulated natural language which allows a manual execution. Automated interoperability Test Descriptions are also possible. A Test Environment can be required: a combination of other equipment and procedures enabling interoperability testing. The Test Environment ensures the selection, interpretation and execution of the Test Descriptions, coordination and synchronization of the actions on the test interfaces, and provides mechanisms for logging, monitoring and observing the interactions among IUTs.

Figure 5. Different levels of interoperability
Figure 5. Different levels of interoperability

A taxonomy of interoperability tests

Figure 5 illustrates the four categories of interoperability tests, following ETSI’s taxonomy:

  • Technical Interoperability is associated with hardware/ software components, systems and platforms that enable machine-to-machine communication. Technical interoperability is centered on (communication) protocols and the infrastructure needed for those protocols to operate.
  • Syntactical Interoperability is associated with data formats. Messages exchanged through a communication protocol need to conform to the agreed-upon syntax and encoding. High-level transfer syntaxes such as HTML, XML or ASN.1 are used to represent the exchanged data.
  • Semantic interoperability is associated with the meaning of content, and concerns human rather than machine interpretation of the content. There needs to be a common understanding of the meaning of the content (information) being exchanged.
  • Organizational interoperability is the ability of organizations to effectively communicate over different information systems, infrastructures, geographic regions and cultures. Organizational interoperability depends on successful technical, syntactical and semantic interoperability.
F-Interop focuses on conformance, technical interoperability testing (including syntactical interoperability checking and conformance testing) and semantic interoperability testing.

The project will develop complementary tools that help testing scalability, Quality of Service (QoS) and Quality of Experience (QoE), as well as energy consumption, using the FIRE+ testbeds.

The complexity of constrained network testing

F-interop aims to develop several tools for testing different technologies. Among the tools proposed, particularly challengeable will be the design and implementation of those testing constrained protocols for low-power wireless mesh networks, (e.g., IETF 6TiSCH, 6LoWPAN and CoAP). The added complexity of testing those standards (in comparison for example to testing IPv6 or HTTP) is due to the constrained nature of the wireless devices and (lossy) environments, where they run. Therefore, the testing tools will be developed according to the strict requirements in term of time synchronization, and energy consumption.

Interop tests, plugfests and plugtests

Partners in the consortium have a good experience in organizing interop tests, plugfests and plugtests. The most important such events are:

- TODO ETSI: CoAP, 6LoWPAN plugtests. Dates, # participants, input ot WGs (3-4 lines)

- ETSI is organizing the first IETF 6TiSCH plugtest, to be held at IETF93 Prague in July 2015. The purpose of the event is to test interoperability of implementations of draft-ietf-6tisch-minimal. 10 participants are expected.

- Inria co-organized an IPSO plugtest even in 2011 around IEEE802.15.4e and CoAP, held in Santa Clara over the course of two days, with 5 implementations from industrial/academic partners. With one company participating remotely from France, this event resulted in the first trans-Atlantic communication between CoAPenabled low-power devices.

- The EANTC and Upperside Conferences invited interested vendors to a public multi-vendor interoperability test and showcase at the MPLS SDN World Congress 2015 (http://www.uppersideconferences. com/mpls-sdn/index.html). Together with the NFV & SDN Summit and the V6 World Congress 2015 taking place at the same time, EANTC presented the results live to the combined audience of all three conferences. By presenting a realistic example of a next generation interoperable service provider network, participants demonstrated the maturity and applicability of their devices and solutions.

- Inria participated in several specialist task forces (STF) at ETSI related to Test Specifications for IPv6 Interoperability. Since 2001, Inria brings expertise to the ETSI Plugtest services for the organization of the several sessions of IPv6 related interoperability events. More recently, in 2012, Inria contributed to the one 6LoWPAN and the two first CoAP interoperability Plugtests events organized by ETSI.

- Since 1995, the EANTC organizes interoperability test events, usually 1-3 times a year, executed in the EANTC lab. Results are demonstrated at shows (including the Carrier Ethernet World Congress).

- Inria is a founding and active member of the worldwide IPv6 Ready Logo certification program (supported by the IPv6 Forum). It is the representative of Europe in this program, both as technical expert for test specification development, and technical advisor for test results analysis in the certification process. In 2006-2008, Inria participated in the Go4IT Project (the FP7 European project on TTCN-3 based Test Tools and Services for IPv6 protocols) and more recently in the PROBE-IT (Pursuing Roadmap and Benchmark in IoT) dedicated to deployment, testing and interoperability issues of IoT.

- Inria co-organized 2 plugfests around IETF 6TiSCH technology (IETF89 London, March 2014, IETF90 Toronto, July 2014), each attended by 8 academic/industrial teams. The purpose was to bring together implementations around 6TiSCH technology, including demonstrations of a federated backbone by Cisco and Linear Technology, and an IEEE802.15.4e stack by the OpenWSN team.

The main objective of these interoperability events is the validation of the specification which gives a higher chance for interoperable implementations, and allowed vendors to detect and correct errors in a controlled manner. At the same time, they are an opportunity for vendors to test their implementations and prototypes, to reduce time to market, to promote technology and to originate new ideas.

F-Interop remote interoperability tools will allow validating of specifications and implementations in a quicker and more responsive manner which, will contribute to the technology success.

F-Interop will organize Plugtest event in which F-Interop interoperability tools will be used and validated.


1 “Vision and Challenges for Realising the Internet of Things”, Cluster of European Research Projects on the Internet of Things (CERP-IoT), page 13, March 2010. [available online, consulted 17-Feb-2015].

2 “Ericsson CEO Predicts 50 Billion Internet Connected Devices by 2020”, GigaOm, April 2010 [available online, consulted 17-Feb-2015].

3 “Disruptive technologies: Advances that will transform life, business, and the global economy”, McKinsey Global Institute, May 2013. [available online, consulted 17-Feb-2015]

4 Jittender Miglani’s blog on [available online, consulted 17- Feb-2015]

5 Global “What If” Analyzer of neTwork energy consumpTion (G.W.A.T.T.), Alcatel-Lucent. [available online, consulted 17-Feb-2015]

6 “The Zettabyte Era—Trends and Analysis”, Cisco Systems, June 2014. [available online, consulted 17- Feb-2015]

7 “The Internet of Things”, Internet Report, International Telecommunication Union (ITU), 2005. [available online, consulted 17-Feb-2015]

8 Interoperability events are branded as “PlugtestsTM” when organized by ETSI.

IPv6 Forum logo   F-Interop logo

For more information visit:

Upcoming Events
Intercomms eBook
Intercomms ebook: click here
Valid XHTML 1.0 Strict
Other publications
by Intercomms: