2.1 Word Embedding based on Deep Learning

Embedding is the state in which a graph can be drawn so that edges do not cross each other on the surface ⁽²⁾. Recently, embedding techniques have been widely employed in the NLP sector. Typical embedding techniques include word embedding, in which words are represented as vectors, and the term frequency (TF)–inverse document frequency (IDF) approach, in which the importance of each word in a document is quantified.

First, for word embedding, there are prediction-based models, including neural probabilistic language models such as Word2Vec and FastText and matrix factorization-based models such as latent semantic analysis and the global word vector model ^(3-8). In this study, we selected Word2Vec, which showed good performance in evaluating word similarity and was widely used in various industries, as the word embedding technique. As shown in Figure 1, there are two training methods in Word2Vec, that is, the continuous bag of word (CBOW) and Skip-gram methods. The CBOW technique is a training method that predicts a target word based on the surrounding words (Fig. 1(a)). On the other hand, Skip-gram is a training method that predicts the surrounding words using a target word (Fig. 1(b)).

Second, TF-IDF, as a weight used in information searching and text mining, is a statistical number indicating how important a word is in a particular document ⁽⁹⁾. The TF indicates how often a specific word appears in a document, and the IDF is the inverse of the document frequency. The TF-IDF can be obtained by multiplying the TF times the IDF. The TF-IDF is used to rank search results in search engines and to measure the similarity between documents within a document cluster.

Minarro-Gimenez et al. conducted a study on improving the accessibility of medical knowledge by applying Word2Vec to medical documents, and Husain and Dih developed a mobile application recommendation system for travelers based on TF-IDF content ^(10-11). Similarly, research using deep learning-based word embedding and the TF-IDF has been actively underway in various industries. However, research on the application of this technology in the electric power sector is insufficient.

2.2 Framwork

A framework is a software environment that provides a reusable design and implementation of parts corresponding to specific software functions in a collaborative form to make the development of a software platform effective ⁽¹²⁾. The framework can be maintained through systematic code management and is highly reusable. It has high development productivity by providing a function library.

Regarding research on frameworks, Bedi and Toshniwal developed a deep learning framework for forecasting electricity demand using long short-term memory ⁽¹³⁾. Dowling et al. proposed an optimization framework for evaluating revenue opportunities provided by multi-scale hierarchies in the electric power market and determining optimal participation strategies for individual participants ⁽¹⁴⁾. In addition, Pinheiro and Davis Jr. endeavored to improve user convenience by managing the characteristics and structure of data collection target themes by developing ThemeRise, which was a framework for producing a volunteered geographic information application, a type of cloud sourcing, and Jack Jr. proposed the National Institute on Aging and Alzheimer's Association (NIA-AA) research framework to assist research on the biological definition of Alzheimer’s disease ^(15-16). As shown in those studies, framework research has been gaining attention not only in the electric power industry, but also in other industries. The introduction of a framework facilitates the utilization of existing technologies conveniently from the perspective of users and has the advantage of efficient platform development.

3.1 Framework Architecture

The framework architecture of the proposed Gen2Vec is shown in Fig. 2. Pretraining is performed using deep learning- based Word2Vec utilizing 1,348 expert knowledge documents for the electric generator. When the user enters a sentence or word in the search box, the preprocessing function is performed to extract only nouns. Then, the word recommendation function is performed by embedding based on the extracted and pretrained words. Next, Gen2Vecscore is calculated using the words extracted by the word recommendation function and the TF-IDF value of each document. Lastly, the document recommendation function is performed to recommend documents related to the search word.

3.2 Preprocessing Function

The preprocessing function of Gen2Vec is performed after the user enters a word or sentence that he/she wants to search in the search box. Firstly, the word or sentence is tokenized to separate each word into tokens, and then part of speech tagging is used to add the part of speech to each token. After this, the word recommendation function is performed by extracting only the noun tokens. KoNLPy that korean natural language processing package for python was used for the preprocessing function of Gen2Vec ⁽¹⁷⁾.

3.3 Word Recommendation Function

The word recommendation function was developed using the Gensim framework-based Word2Vec for the words extracted from the preprocessing function ⁽¹⁸⁾. The training parameters of the Skip-gram model for extracting embedding words are matrices u and v. The size of each matrix is the size of the vocabulary set (|v|) by the number of embedding dimensions (d). The probability that the target word (t) and content word (c) are positive samples is calculated using Eq. (1), and the probability that t and c are negative samples is calculated using Eq. (2).

Eq. 1(식 (1)) Positive Sample Calculation of Skip-gram

Eq. 2(식 (2)) Negative Sample Calculation of Skip-gram

The log-likelihood function of Skip-gram is Equation (3), and the word recommendation function can vectorize words in a document cluster after training to optimize Eq. (3). Using this approach, the words related to the search word can be recommended.

Eq. 3(식 (3)) Log-likelihood Function of Skip-gram

3.4 Document Recommendation Function

The document recommendation function of the proposed Gen2Vec is as follows. When the user enters a necessary word or sentence in the search box, the preprocessing function is used to extract the noun words ($Word_{x_{i}}$). The extracted words are input into the trained Word2Vec model to extract the TopN words ($Word_{y_{i}}$) with high cosine similarity for each $Word_{x_{i}}$. The formula for obtaining the cosine similarity is expressed in Eq. (4), and the method for obtaining $Word_{y_{i}}$ is expressed in Eq. (5).

Eq. 4(식 (4)) The formula for obtaining the cosine similarity

Eq. 5(식 (5)) The method for obtaining $Word_{y_{i}}$

After defining each word (w), each target word (t), each document (d), the total number of documents (D), and the total frequency (frequency (f ())) for the documents, TF-IDF is produced using Eq. (6). Then, the data frame is extracted for TF-IDF, which has the word list consisting of $Word_{x_{i}}$ and $Word_{y_{i}}$ as the column value and has the documents as the row values. Table 1 shows an example of the extraction results.

Eq. 6(식 (6)) The formula of TF-IDF

The above data frame contains the word $Word_{x_{i}}$ that the user directly enters in the search box and the related word $Word_{y_{i}}$ extracted by deep learning. Therefore, Gen2Vecweight is defined because it is necessary to give different weights to $Word_{x_{i}}$ and $Word_{y_{i}}$, as expressed in Eq. (7). Using this approach, the TF-IDF value in the data frame is updated.

Eq. 7(식 (7)) The formula of Gen2Vecweight

Next, Gen2Vecweight for each document in the data frame is summed and the TopK documents are extracted. The function used for this calculation was defined as Gen2Vecscore. If T is defined as the total number of $Word_{x_{i}}$ and $Word_{y_{i}}$, the method of obtaining Gen2Vecscore can be expressed as shown in Eq. (8).

Eq. 8(식 (8)) The formula of Gen2Vecscore

4.1 Expert Documents for Diagnostic Services of Power Generation Facility

KEPCO has operated electric generators at each of their power plants for about 20 years, and experts have directly diagnosed the generators, accumulating 1,348 documents. The experts utilized categories of boiler, electric generator, performance, gas turbine, and steam turbine in the diagnosis and designated subcategories such as fault diagnosis and precision diagnosis. Table 2 shows the statistics of expert documents on electric generator operation collected from 2000 to 2018. The Gen2Vec developed in this study was trained for these documents.

4.2 Preprocessing and Word Recommendation

The experimental example and results of the preprocessing function are shown in Fig. 3. After entering a word or sentence that the user wants to search in the search box, tokenization is applied to separate the word or sentence into tokens when the preprocessing function is applied (Fig. 3 (a), (b)). Next, the part of speech is tagged to each token, and the nouns are extracted (Fig. 3(c), (d)). The extracted nouns are input into the word recommendation function.

The experimental example and results of the word recommendation function are shown in Table 3. In this experiment, the nouns ($Word_{x_{i}}$) extracted by the preprocessing function as shown in Fig. 3 were pretrained with the Word2Vec model. For pretraining, the word vector dimension was set to 1,000, the window size was set to 4, and the downsample setting for frequently appearing words was set to 1e-3. By applying these parameters, the embedding words ($Word_{y_{i}}$) corresponding to TopN were extracted. Here, the embedding words were output in order of descending cosine similarity. In this experiment, N was assumed to be 5, and each extracted $Word_{y_{i}}$ had a cosine similarity value.

The experimental results show that the words that were highly related to each $Word_{x_{i}}$ were extracted as $Word_{y_{i}}$. The extracted words included duplicate words. Other than the duplicate words with high cosine similarity, the duplicate words were excluded from $Word_{y_{i}}$ to be used in the document recommendation function.

4.3 Document Recommendation Function

The experimental example and results of the document recommendation function are presented in Table 4. The TF-IDF value that was pretrained for the nouns ($Word_{x_{i}}$) extracted by the preprocessing function and words recommended by the word recommendation function ($Word_{y_{i}}$) was updated with the proposed Gen2Vecweight. Next, the documents extracted using Gen2Vecscore were presented to the user. Defining the K value used to obtain TopK as 10, in relation to the search words in the example employed in this study, the extracted documents are listed in Table 4.

The results in Table 4 confirm that the documents related to the nouns extracted from the search words, such as the gas turbine report, blade damage report, and high-temperature parts damage report, were extracted. The accuracy result of the document recommendation function is in Table 5. Precision, Recall, and F1 results were derived for Gen2Vec, Word2Vec, and TF-IDF. Gen2Vec proved about 3.9% and 10.8% higher than Word2Vec and TF-IDF.