The SignON communication service will be more than an advanced machine translation system. Behind the scenes, SignON will incorporate sophisticated machine learning capabilities that will allow (i) learning new sign and spoken languages (ii) style-, domain- and user-adaptation and (iii) automatic error correction, based on user feedback. To the user, SignON will deliver signed conversations via a life-like avatar built with latest graphics technologies.
Bringing individuals together through technology, while respecting privacy
A competitive and comprehensive work plan has been devised to deliver the objectives of the SignON project and to ensure that it fulfils its goals and vision.
Objective 1:
Co-creation Workflow and Community. We aim to bring researchers and developers into a close collaboration with the main stakeholder groups to drive the research and development in SignON.
Objective 2:
SignON Framework and Mobile Application. Design and development of a framework and a mobile application which will deliver the SignON service to the user.
Objective 3:
Automated Recognition and Understanding of SL and Verbal Language Input through advanced sign language recognition (SLR), automatic speech recognition (ASR), and natural language understanding (NLU).
Objective 4:
Research and Development of a Novel Language Independent Meaning Representation for interlingua M, based on current vector representations, symbolic components or hybrid representations of the input/output message.
Objective 5:
Sign, Speech and Text Synthesis. SignON will convert a SL specific syntactic-semantic representation in the target SL via a customizable 3D virtual signer (i.e. avatar). It will also produce text output in the different spoken languages adapted to the user, by, for example, simplifying the text.
Objective 6:
Wide-range of Supported Languages and Extensibility of the Framework. During the project we will provide support for the following SLs: Irish SL (ISL), British SL (BSL), Flemish SL (VGT), Dutch SL (NGT) and Spanish SL (LSE) as well as English, Irish, Dutch and Spanish spoken languages. However, we will design the SignON application and framework to be extensible to new sign and spoken languages.
SignON’s research work is split into nine Work Packages (WP). WPs 1-5 are the technical work packages, directly related to both the research and the community engagement. WPs 6 & 7 are related to project coordination, communication, dissemination and exploitation. WPs 8 & 9 look at the ethics involved in the project.
The SignON work plan follows a general strategy of simultaneous and gradual advancement of the SignON service, functionalities and the underlying components to ensure wide up take from all user groups.
WP1 Co-Creation and User Response
Through WP1 we will engage with the potential users to co-design and co-develop the SignON framework and application and lay out a roadmap of KPIs to guide their development. Through openness, fairness and gender equality we will also establish a user community with hands-on experience with realistic expectations towards the SignON service.
WP2 SignON Service and Mobile App
WP2 encapsulates the technical activities involved in designing and developing the SignON framework that will incorporate components delivered through WP3, WP4 and WP5 and the SignON mobile app. The framework and the app will be free and open-source to allow for easy adoption and evolution during and after the life cycle of the project. WP2 is also concerned with the instantiation and utilisation of the cloud-based platform and a repository for storing and organising collected data.
WP3 Source message recognition, analysis and understanding
In WP3 we will research and develop efficient methods for recognition and analysis of the input message in the source language:
Sign Language Recognition (SLR)
Automated Speech Recognition (ASR) for typical and atypical speech
Input and recognised text analysis and processing through a set of Natural Language Understanding (NLU) pipelines
We will realise this research as SLR, ASR and NLU components that are adaptable. The work will be conducted in parallel, and led by different partners, in order to achieve a synchronous delivery of source message understanding of both spoken and sign languages.
WP4 Transfer and Interlingual Representation
In WP4 we will formulate and implement a multimodal and multilingual intermediate representation - Interlingua - to bridge between the sign and spoken languages that are handled in the project as well as the transformation processes from and to Interlingua. We will start with two different representations - a symbolic one and a representation based on embeddings - that we will merge into a hybrid representation.
WP5 Target Message Synthesis
WP5 will also encapsulate the work for written language synthesis and normalisation based on the representation in Interlingua. Thereafter this text will serve as the basis for the synthesis of spoken language through a commercial text-to-speech platform. WP5 will deliver a pipeline for sending the target message to the user through the SignON app.
WP6 Communication, Dissemination and Exploitation
SignON will offer innovative opportunities to exploit and promote its results and WP6 will develop business plans and advance exploitation for the project, including joint and individual opportunities, maximising opportunities for the dissemination of project results and ensuring effective communication procedures are in place. The project will draw on significant media networks and established contacts to promote awareness, adoption, and further development of the SignON services and application.
WP7 Coordination and Management
WP7 will coordinate the overall project with WP leaders managing their own work packages and contributing to project management (meetings, reviews, reports). DCU’s extensive experience in managing and collaborating in multidisciplinary research as well as experience in organising multidisciplinary research-industry events will ensure smooth and productive work throughout the project.
WP8 Ethical Requirements
WP8 will set out the 'ethics requirements' that the project must comply with.
WP9 Ethics
The final WP oversees and coordinates all ethical aspects of the project and provides the ethical guidelines and protocols for the identification and recruitment of participants. It also ensures all consortium members abide by the EU Code of Research Integrity and provides support and advice to members of the community as well in terms of the use of their personal data.
About SignON
SignON is a user-centric and community-driven project that aims to facilitate the exchange of information among Deaf, hard of hearing and hearing individuals across Europe, targeting the Irish, British, Dutch, Flemish and Spanish sign as well as the English, Irish, Dutch and Spanish spoken languages.
