MusicMo: A Seamless E-commerce Solution for Instrument Retail in the Digital Age

Demo

Visit at: http://musicmo.demo.zixiaoma.com/

Overview

Our project aims to develop an attractive online instruments shop for a family-owned shop to alleviate the aforementioned problems. Firstly, our project has essential functions for online shopping. Secondly, to give customers immersive shopping experience, our project employs the usage of 3D instruments preview and instruments playing games on our website. Thirdly, our project is original in allowing customers to interact with 3D models to customise their preferred instruments, which have not gained much prevalence in current e-commerce. Most importantly, our project features intelligence, such as instrument recognition and search based on Machine Learning (ML) and Information Retrieval (IR) and AI customer service based on Natural Language Processing (NLP).

Design and Implementation

Functional Requirements Implementation - Employees Portal

Communication with Employees and Customers

One distinguishing feature of our communication function is its sidebar, where displays all messages ordered by time. Our group has discussed possible implementations for that and finally chose the following optimal approach. Firstly, we utilised Q and Django Model filter construct the sender_messages and recipient_messages dataset of messages related to the current user from the messages table of the database. After that, we utilised annotate to select the neaREST time and the number of unread messages of the sender and sort them by the sender’s unique identifier (ID) to construct the dataset sender_messages. Then we select the lasted time and sort them by recipients ID to construct the dataset recipient_messages. Now the recipient_messages and sender_messages have been sorted by ID.

Add Instrument (Blog, Wishlist, Cart)

The page where employees can add new instruments combines the model form provided by Django and the multi-file upload plugin of Bootstrap: the model form can generate a form according to its corresponding Django model, and the input tags of this form are automatically generated according to the fields of the model. This helps avoid the mismatch between the form and the database fields and frees the need to manually modify the corresponding form whenever the model fields are changed, greatly improving development efficiency and reducing the probability of web bugs. Bootstrap’s multi-file upload field is used to upload instrument images, allowing employees to upload and preview all instrument images simultaneously. When the administrator enters the instrument information, the instrument information will be posted to the back-end for instrument creation and storage.

Manage Personal Profile

We allow customers and employees to manage better their accounts, including username, first name, last name, email, and recent orders. Customers and employees can modify their first name, last name and avatar, which do not need to be verified. The click event of the first name and last name is listened to by jQuery. When jQuery detects a click, its location will be replaced with an input box, and after completing the input, a completion button will be provided next to the click-box. The contents of the box will be modified through the API provided by the REST framework for database-level operations to modify the user’s information. When the listening of avatar’s changes is detected, the previously hidden form for receiving the file will show up. After customers upload the avatar file and submit it, the image will be extracted and stored by the back-end. A separate page will be generated for email, passwords, and other information that needs to be verified. After customers submit the updated information, our website will encrypt the verification URL through Django’s built-in encryption method and send the encrypted URL to the customer’s email for verification. The information can only be modified after validation.

Functional Requirements Implementation - Customer Portal

3D Instruments Customization

We use the Blender and 3D MAX tool to divide and label the components with the names of our prepared 3D instruments models, which is labor work and very time-consuming. After spending many efforts and a pretty long time dividing and naming different parts of the instruments, we finally enable customers to modify the colour of different parts of the model of the 3D instrument. The work process is shown in the figure below.

Moreover, we also achieve the visually ascetic explosion effects of instruments models (see figure below). However, it is not always easy for our implementation. Some small parts are often difficult to select when customers select a particular model part, such as the guitar’s strings. To solve this problem, we use TweenLite to smoothly move model parts loaded by Three.js in different directions so that models can present an explosion effect. In this way, we can bring customers an unprecedented ascetic visually experience.

AI Customer Service based on NLP

Our project uses Natural Language Processing to implement automated customer service. We integrated RASA, which is an open-source machine learning framework for automated text generation. We defined intents, entities, and slots and supplied example sentences to understand the user’s intent fully. After training the model, customers can ask our intelligent customer service, for example, “I want to buy Chinese instruments,” and they will receive the link to the Chinese instruments zone. The following figure gives an illustration of our AI customer service.

The processes of NLP involved in our project are as follows.

• Training Data(stories.yml, rules.yml, nlu.yml): To recognise customers’ messages, we provide example messages for AI customer service to learn from and group examples based on their intent. Then use intents and their examples as the training data for the Natural Language Understanding (NLU) model.
• Domain(domain.yml): The context in which our customer service operates specifies the intents, entities, slots, responses, forms, and actions that the AI knows about in advance. The domain also defines a configuration for conversation sessions between customers and automated customer service.
• Configuration(config.yml): The mode configuration file defines the components(pre-trained entity extraction model) and policies that the automated customer service model will use to make predictions based on customers’ input.
• Actions(actions.py): Predict what the AI should perform after receiving customers’ messages.
• Evaluation: We have a test dataset with pre-configured dialogues and intent, which can be used to check if the model gives the right response.
• Channel Connectors: Because we want to make it easy for customers to use your bot, we use the chat interface that ships with RASA, where customers can test the bot if they are interested in it.

Search for Instruments Using Image Retrieval

We combine the instruments recognition with the search engine to allow customers to search for their preferred instruments by uploading an image of an instrument. We utilise a deep residual network model without the fully connected layer as a feature extractor. Thus, each image can be represented by a vector, and the distance between vectors indicates the degree of similarity between two images. The following shows other methods we have tried and compared to make the final decision using Resnet.

• Structural Similarity (SSIM): a metric for determining the quality of compressed pictures and measuring image similarity.
• Siamese network: a framework for a neural network to evaluate the similarity of two input samples.
• Mean Hash Algorithm: images are transformed to generate a set of binary numbers, and then similar images are found by comparing the Hash value distances of different images.
• Cosine similarity: An picture is represented as a vector, and the similarity of two images is determined by computing the cosine distance between the vectors. The closer the cosine value is to 1, the closer the angle is to 0 degrees, implying that the two vectors are more close to each other.
• Resnet: Our implementation process using Resnet is as follows. The image will be sent to channel cropping, where one channel image is converted into three channels, and if there are more than three channels, the first three channels will be used directly. Then the NumPy array with (w, h, 3) dimension is processed by resnet (convolution ->BatchNorm ->ReLU ->maxpool ->layers 1-4 of resnet ->mean pooling that removes the last layer of the original ResNet fully connected). The feature vector of the image is one dimension (2048), so we generate this feature vector for each image and store this feature vector in the database.

Content Update for Instrument Filters

We came up with two display methods for filtering results: overall page refresh and partial refresh of the page. We use the second option because it reduces unnecessary page refreshes. Moreover, we consider the advantage of page refresh because it can refresh the web page’s link so that the user can ensure that the page’s information can be completely conveyed in the usage scenario of accidentally touching the refreshed website or sharing the link. Therefore, while using Ajax to refresh the web page, we use Javascript to manually design the logic to update the link displayed on the web page, containing the filter conditions selected by customers. This reduces page refreshes and ensures that search results are not lost due to page refreshes under unexpected conditions

• 2023 6