Project 4 Stage III

• Initial prototype of interactive, functional 3D environments (executable + instructions / description) functional (executable + instructions / description)
• Initial prototype of the digital annotation system, functional (executable + instructions / description) functional (executable + instructions / description)
• All articles realized in present moment

Title. Authors. Conf/Journal	Status
LibQuest initial prototype – Mixed-reality quests to revitalize libraries and reading Maria Anca Băluțoiu, Alexandru Grădinaru, Alin Moldoveanu, Florica Moldoveanu, Ana-Karina Nazare Accceptat la conferinta “16th eLearning and Software for Education Conference – eLSE 2020”, ISI	published
Rapid Modelling Techniques for 3D Reconstruction of Historical Buildings, for Immersive Virtual Environments Nazare Ana-Karina, Moldoveanu Alin, Moldoveanu Florica, Gradinaru Alexandru, Balutoiu Maria Anca, Asavei Victor Accceptat la conferinta “16th eLearning and Software for Education Conference – eLSE 2020”, ISI	published
Socio-Collaborative Reading through AR-Annotations over Physical Books Iulia Stănică, Alin Moldoveanu, Florica Moldoveanu, Alexandru Grădinaru, Maria Dascălu Accceptat la conferinta “16th eLearning and Software for Education Conference – eLSE 2020”, ISI	published
Evolving the Very Act of Reading, with Socio-Collaborative Dimensions – AR-Annotations over Physical Books Acceptat spre publicare in University POLITEHNICA of Bucharest Scientific Bulletin, Series C-electrical Engineering And Computer Science (jurnal ISI).	published

Socio-collaborative digital annotations for physical books

Accomplishments thus far

The - “Lib2Life Annotations” – mobile app includes in its current state the following features::

• Account creation via an email + password, or a social media account (Facebook, Twitter, or Google)via an email + password, or social media account (Facebook, Twitter sau Google)
• Adding books to a favourite from the ones existing in the database manually or by scanning and automatically detecting the ISBN code by using the mobile phone
• Adding new books by the user through completing the information fields of the respective books
• Book ratings and comments .
• Automatic text recognition by scanning the respective page with the phone (using OCR)
• Annotations in the form of texts, videos (YouTube), or links, accompanied by hashtag topics
• The visualization of annotations added on a scanned page of a book, using matching-text algorithms (patenting potential)
• The visualization of detailed information regarding annotations, annotation ratings, and comments )
• Annotation sharing via social media apps (Facebook, Twitter, Whatsapp, or any other related app within the user’s phone)
• Automatic translation of annotations, as well as their respective comments
• Automatic image correction made by the users by adding/visualizing annotations
• Newsfeed function,with the latest users activities via the app
• Navigation through lists of annotations and books by using the "search" and "sort by" commands
• Vizualization through lists of annotations and books under different appearances (list/grid/table)
• Vizualization of all annotations associated with a topic (#)
• Loading 3D objects and viewing them in augmented reality module

Besides the essential functionalities of the application, in the last 3 months a series of functionalities for optimizing the application have been realized:

• Retrofit library integration to improve communication between client application (Android) and server; the bugs arising from this integration have been resolved;
• added the deskew option for image orientation correction an algorithm was developed to automatically correct the rotation of captured images using the device's camera; thus, both for adding new annotations or for visualizing the existing ones, the algorithm determines the reference lines on the basis of which the rotation (text lines) will be performed so that their angle after correction is as close as 0 degrees. The algorithm is set to correct any rotations up to 45 degrees (clockwise or counterclockwise). The differences between the images obtained without and with deskew can be observed here (Figure 1 - a and b).

Fig.1 Automatic image correction

a) Before correction

b) After correction, including the highlited anchor to which the annotation will be added

The process of adding advanced functions has started as well:

• adding the share option Annotation sharing via social media apps one of the indicators in the Act. 3.14 of the project; share works by default on all applications installed on the user's device and includes customized functions for sharing on Facebook (Figure 2 a and b), Twitter or Whatsapp; in case of sharing on Facebook, it will automatically open the application if it is installed on the device, otherwise the Facebook page will open in web mode; in the current stage of the prototype, in the case of text annotations is included a link to the APK of the application (from Google drive), which will be replaced with the link to the Google Play store (when the application will be launched);
• adding the share option automatic translation annotations and comments are translated (if the user opts for this) from the language in which they were made in the language in which the user's phone is set (detected automatically by the application); this functionality is achieved with the help of the Google MLKit module;
• integrated the API for Twitter (login, sharing of annotations in the form of "tweets") - this module is now completed, being initially delayed due to the late obtaining of development approval from Twitter.
• adding the augmented reality module - an innovative component of the application, that of adding and viewing 3D objects in augmented reality, associated with an image detected in the book; ARCore (for image augmentation) and Poli (3D library), both offered by Google, have been integrated into the application; the user can select the desired image from a screenshot of a book page to save it in a database (currently local on the device); the image magnification module recognizes the images saved in the database and displays 3D models associated with them (either present in the application Figure 2 c - not indicated, the size of the application would increase a lot - or searched in Poli - Figure 2 d); the user can interact with these 3D objects (scaling, rotating);

Figure 2 - a. Social media distribution options b. Video sharing on Facebook example c. Augumented image from a book (default 3D object) d. Image augumentation (3D object from the Poli library database)

Interactive 3D reconstructions

3D modeling of libraries

Achievement summary:

Library	Development stage	Achievement percent	Comments
BCUB	5.3	90%	Includes the 3 visitable bodies (Fundatia, Dacia, Bohemia). In the process of popularizing with objects, starting with the lighting fixtures, after which the test lights already present in the game scene will be repositioned.
BCUI	2.2	70%	The detailed reconstruction of the skeleton is completed. Reconstruction of architectural elements such as: stairs, columns, windows, etc.
BCUC	2.1 4	65% 20%	Reconstruction of the basic geometry of the building architecture. 20% of the library objects (11) are reconstructed and textured.
BCUT	4	90%	Completed reconstruction of the library and 90% of its objects. Both the geometry of the architecture and the objects are associated with test materials.

Activity description

The reconstruction process is a complete one, namely, it starts from the identification and acquisition of data (space recognition, analysis and formalization, and taking data of interest to achieve this form) and ends with the creation of virtual space in an interactive environment.

Accomplishments thus far

The first stage, data acquisition, was successfully completed for each partner's site.

Each visitable body of all partner buildings was approached in the construction of 3D virtual environments.

BCUB

The architecture of all 3 bodies of BCUB, Fundatia, Dacia and Boema, has been completely rebuilt (both geometry and materials and textures), including the connections between the bodies. Their main interior objects have also been reconstructed, including their textures. They are being rebuilt or some decorative objects, books, signs with indications have been omitted for the time being.

Fig. 1. Finalized reconstruction of BCUB's buildings architecture. The reconstruction, imported in the game development environment, presents its 3 bodies: The Foundation, Boema, Dacia.

At this time, all three bodies of BCUB have been completely rebuilt, including the application of textures corresponding to real materials, both on the architectural elements and on the objects that populate it. Exceptions are the performance hall or Aula, still untextured, and the access halls in it, whose reconstruction has the general geometry of the architecture completed, and the process of dividing the geometry according to decorative elements and materials is ongoing.

Fig. 2. Reconstruction of the architecture of the auditorium and of the access halls.

Right: detailed view of the performance hall. The reconstruction includes the lodges, the stage and the details of the ceiling.

The objects that populate BCUC are already reconstructed in proportions of approximately 20%.

Each texture is created from photos taken on the field. A new such visit was made on the occasion of the addition of references for the textures from the Dacia and Boema bodies of BCUB. The photographs are processed to recreate the chromatic from reality, then they are used to build repetitive visual structures, the delimitation of which is invisible. These can then be applied as maps on materials, to which are added attributes such as specularity, roughness, etc.

For example, for the texture of the Bohemian body, 47 textures were made corresponding to the paintings in the Bohemian body, 14 textures for uniform surfaces (materials such as marble, carpet, furniture surfaces).

Each building was imported into Unity and aligned to make the appropriate connections between the buildings. Lights were added for testing purposes, but only the first floor of the Fundatia body was populated with lighting objects.

The next steps are to recalibrate the materials imported from 3D Studio Max and then populate the virtual media with objects, in order of importance. The first series of objects that will be included in the project are the lighting fixtures, in order to keep the consistency in the development according to the reality and to be able to explore and test the virtual environment in the game environment.

Fig. 4. Frames from the virtual environment of the BCUB library reconstruction integrated in the Unity game engine

BCUT

The architecture of the BCUT buildings was also finalized, as well as the application of materials (which only chromatically suggest the real materials). These materials are to be replaced by materials created with the help of textures made from photographs taken in the field.

Fig. 5 .: Detailed reconstruction of the BCUT building. It includes the old body and the communication corridor with it and with the Western University

The reconstruction of the “Eugen Todoran” Central University Library architecture in Timisoara  presents the entire completed geometry, including the architecture of the two buildings, as well as the division of the geometry into test materials and the mapping of the geometric coordinates to the texturing ones.

The architecture is remade to a high level of detail, including decorative elements, columns, doors and windows, stairs and railings. The geometry is already divided into color-coded polygons that suggest the nuances of the materials in reality.

Fig. 6. Frames inside the BCUT reconstruction. Top: The new body of the library. Entrance to the performance hall. Reading room on the third floor. Bottom: Old body. Lobby with stairs from the old entrance and open reading room

Once the architecture was completed, the next stage focused on library objects. The objects inside the library are completely inventoried. 90% of them are modeled.

Materials with contrasting colors are applied to each object to highlight the difference between the types of materials that will be applied later, on a UV mapping already made. UV mapping means the correspondence between geometric and texturing coordinates (practically, coordinates on the image-texture) of the object.

Fig. 7. Objects of BCUT. Offices. Chairs. Plants. Instruments. Shelfes.

BCUC

The reconstruction of the BCUC architecture building is completed at a primary level, including the accessible bodies and respecting both the horizontal and the vertical projections of the building. It is the general skeleton of the building, comprising the halls, the reading room with dome, and the reading rooms upstairs, not separated.

About 20% of the objects that populate the library building are already rebuilt. These reconstructions include their textures and their application.

Fig. 8. Architectural reconstruction of the “Lucian Blaga” Central University Library in Cluj

Fig. 9. Reconstructions of BCUC objects examples. The office in the reading room "Alexandru Lepadatu". Catalog cabinets. Library in the reading room "Miko Imre". The map cabinet in the "Gheorghe Baritiu" hall. The office in the reading room "Emil Racovita"

BCUI

The reconstruction of the BCUI architecture includes the details of the hall (Lucian Blaga Library in Cluj does not present the performance hall) and the decorative domes, respectively those of the entrances of the custodians' offices in the reading rooms, the large marble stairs and a set of columns.

Fig. 10. Reconstruction of the architecture of the Central University Library "Mihai Eminescu" from Iasi. Left: Exterior view. Right: View from the inner hallway on the stairs and the entrance to the Hall, with visible architectural details (stairs, columns, stage)

Interactive 3D applications

Accomplishments thus far

Interactive 3D applications include all the basic functionalities set out in the requirements specification set out in the requirements specification

Thus, the following essential functionalities were realized:

• software client that works on all targeted platforms: windows, mobile, web (in browser) and virtual reality
• central server for managing resources, services and data persistence
• dedicated multi-user that manages the virtual scenes of the libraries to allow the connection of several users to the application, and to manage the interaction between them
• The following functionalities implemented on the client and server:
• creating an account and authenticationusing the email address and password
• central menu central menu from where the user can consult the profile data, the list of available libraries and an avatar editor
• avatar for user representation in virtual scenes, a human representation that can be controlled by the user through input means specific to the access device (keyboard, mobile, controller, etc.)
• avatar editor that allows you to customize various aspects of physiognomy, such as the size of body parts or the color of the skin or eyes, but also clothing or style
• virtual tour, in which users are guided by an avatar through the library, presenting different information at key points
• interactions with the virtual environment,such as the ability to open doors or access information about specific objects in static or multimedia format, even in the form of video streaming
• communication between users,through a real-time text messaging system (chat), currently managed by the multi-user server
• quest system
  • editor for designing and adding new quests
  • taking over and solving quests by the user
  • integration of an augmented reality system for quests and virtual tours
• integration in the virtual environment of the 3D model for BCUB

A. Avatar editing                                                                                             B.    The scene from BCUB