Querying and exporting data from the DataBank management

To implement the databank management procedures, we need to provide the

when to show the button. We set the button with the consistent style of all the other buttons on the

edutalk-“display:none;” configuration to make the default. In the event handler of EduTalk ( databank button to show while

the user is on other web pages by given the c

“$('#show-dg-btn').hide();

Elasticsearch. For examples, “curl -XGET localhost: 9200/?pretty” shows the cluster http://localhost:9200/_cat/nodes” dumps the node settings.

final steps to configure the transfer of event logs from td We need to install Elasticsearch plug-in for td

configure the output destination of td-agent appropriately. After installing the Elasticsearch output plug-in, we replace the <match fluentd.test.**>

-agent with the content in Fig. 35 to Elasticsearch via port 9200.After the steps complete, we have to make all configurations effective.

. The configuration for Elasticsearch in the td-agent configuration file

4.2. Querying and exporting data from the DataBank management

To implement the databank management procedures, we need to provide the databank GUI, the query function, and the export function.

provides an interface for users to query the sampled data and download the export file. We modify the display web page of EduTalk

templates/tutorial.html) to implement the databank GUI. W

“show-dg-btn.” And when clicking the button, the databank GUI modify the event handler of EduTalk (edutalk/static/tutorial.js

when to show the button. We set the button with the consistent style of all the other -nav bar (Fig. 36 (b)), and give the databank button the configuration to make the GUI not to display

fault. In the event handler of EduTalk (edutalk/static/tutorial.js

databank button to show while the user is on the animation web page and to hide while other web pages by given the code “$(‘#show-dg

();” respectively.

XGET localhost: 9200/?pretty” shows the cluster node settings.

And when clicking the button, the databank GUI static/tutorial.js) to set when to show the button. We set the button with the consistent style of all the other , and give the databank button the databank button by static/tutorial.js), we configure the n the animation web page and to hide while dg-btn’).show();” and

Users can query the data sources of the specific sensors by clicking the checkbox with the sensor name and choosing the start and end time on the databank GUI.

are three labels in the databank GUI:

(d)), and Time End (Fig. 36 (e))

sensor name used in the lecture. And we provide checkboxes with the sensor name t allow the user to specify the sensor data to be exported

of the “Time Start” and “

before the present time, respectively minute, and second to display

export the data within 5 minutes from the current time by the default settings.

users can adjust “Time Start” and “Time End” to specify the period to The databank GUI also contains a

selection and closing the databank GUI

sending the information selected by the user to the query function and the export function to download the Excel file.

We modify the event handler of the lecture management (edutalk/lecture.py) support the query function and the export function.

selected sensors, the start time and the end time

in databank GUI. The export function sets the name of the exported Excel file to an easily discernible pattern.

Fig. 36. The databank GUI.

query the data sources of the specific sensors by clicking the checkbox with the sensor name and choosing the start and end time on the databank GUI.

are three labels in the databank GUI: Data Sources (Fig. 36 (c)),

(Fig. 36 (e)). The “Data Sources” label shows the corresponding sensor name used in the lecture. And we provide checkboxes with the sensor name t

specify the sensor data to be exported. To provide the

“Time End,” we use the present time and the time five minutes , respectively. And we divide them into four parts: date,

minute, and second to display “Time End” and “Time Start.”The

export the data within 5 minutes from the current time by the default settings.

users can adjust “Time Start” and “Time End” to specify the period to he databank GUI also contains a close button (Fig. 36 (g))

the databank GUI. It also has a download button (Fig. 36 (f)) the information selected by the user to the query function and the export to download the Excel file.

the event handler of the lecture management (edutalk/lecture.py) query function and the export function. The export function gets

the start time and the end time specified by the user through the form The export function sets the name of the exported Excel file to an cernible pattern. The name combines the start time and the end time with the query the data sources of the specific sensors by clicking the checkbox with the sensor name and choosing the start and end time on the databank GUI. There , Time Start (Fig. 36 label shows the corresponding sensor name used in the lecture. And we provide checkboxes with the sensor name to . To provide the default values time and the time five minutes . And we divide them into four parts: date, hour, .”Therefore, users can export the data within 5 minutes from the current time by the default settings. Also, users can adjust “Time Start” and “Time End” to specify the period to export data.

button (Fig. 36 (g)) for cancelling the a download button (Fig. 36 (f)) for the information selected by the user to the query function and the export

the event handler of the lecture management (edutalk/lecture.py) to he export function gets the user through the form The export function sets the name of the exported Excel file to an combines the start time and the end time with the

“&&” symbol. For example, respectively “2020/01/01 10:11:12 export Excel file is set to

selected sensor, the export function creates a new worksheet [31] to including the timestamp and the sensor data of each sample

created, we call the worksheet_init function to initialize it of the selected sensor data.

kind is a column to display the timestamp. The second

sensor. The number of columns of the second kind matches the dimension of the sensor data. It uses three columns if the

one column if the sensor

establishes a connection to string to obtain the desired da

“query_str” is defined as the line 156 in Fig.

function calls the worksheet_write function to

Fig. 37. Th

The worksheet_write function transforms the timestamp in a human readable format in GMT+8 time zone and writes all timestamps in this format in the first column of the worksheet. Then, it converts the sample to the floating

writes the data in the columns of the second kind. Next, it transforms the from the first sample of the data to the integer format and writes the interval to the last column of the worksheet.

For example, the start time and the end time speci

2020/01/01 10:11:12” and “2020/01/02 12:11:10.” Then the name of the export Excel file is set to

“2020-01-01-10-11-12&&2020-01-02-selected sensor, the export function creates a new worksheet [31] to

including the timestamp and the sensor data of each sample. Once the worksheet is created, we call the worksheet_init function to initialize it according to the dimension of the selected sensor data. The worksheet contains three kinds of colu

to display the timestamp. The second kind represents the data of the The number of columns of the second kind matches the dimension of the

It uses three columns if the sensor data is three-dimensional, a

data is one-dimensional. The last kind is a column indicating the arrival time interval, which is the time interval from the timestamp of the first

After the worksheet is initialized, the export function calls the query function to data from Elasticsearch DB and then writes the results to the corresponding worksheets. To query the Elasticsearch DB, the query function first

a connection to Elasticsearch. Then, the query function

string to obtain the desired data from the Elasticsearch DB. The query string

“query_str” is defined as the line 156 in Fig. 37. After receiving the results, the query function calls the worksheet_write function to write the results to the worksheet.

Fig. 37. The partial code of the query function.

The worksheet_write function transforms the timestamp in a human readable format in GMT+8 time zone and writes all timestamps in this format in the first

he worksheet. Then, it converts the sample to the floating writes the data in the columns of the second kind. Next, it transforms the from the first sample of the data to the integer format and writes the

to the last column of the worksheet.

the start time and the end time specified by the user is Then the name of the -12-11-10.” For each selected sensor, the export function creates a new worksheet [31] to store the values Once the worksheet is The last kind is a column indicating the arrival time interval, which is the time interval from the timestamp of the first

, the export function calls the query function to data from Elasticsearch DB and then writes the results to the . To query the Elasticsearch DB, the query function first Then, the query function sets up a query DB. The query string After receiving the results, the query write the results to the worksheet.

The worksheet_write function transforms the timestamp in a human readable format in GMT+8 time zone and writes all timestamps in this format in the first he worksheet. Then, it converts the sample to the floating-point format and writes the data in the columns of the second kind. Next, it transforms the time interval from the first sample of the data to the integer format and writes the arrival time

To export the Excel file, we need to add “-./data:/data” in the volumes of Edutalk by modifying the configuration file of EduTalk (docker/docker-compse.yml) and it will automatically add a directory for storing while executing. To export data to an Excel file and to perform the Elasticsearch query API, we need to add openpyxl and elasticsearch in the document (requirements.txt) that use to record all dependent packages.