Status message

Maintenant, vous regardez: Installation and configuration of elastic server, vuejs and lumen for advanced search

Installation and configuration of elastic server, vuejs and lumen for advanced search

Elastic Search performs real-time distributed search and analysis. In short, it is a platform or engine that performs a full-text search. It has a growing popularity over the years. It has powerful features and is easy to configure and uses RESTful API.

Let us try and understand the Elasticsearch basics in its pure form:

Documents:

Elasticsearch is schema-free data storing system. Elasticserach doesn’t indulge in database or table storage. It uses JSON formatted documents. All the operations like search, filler, sort are performed on this documents.

Index:

To perform search and other operations, Elasticsearch indexes the documents. It logically maps the data keys (or fields) together in documents that have similar structure. For instance, an index for users, posts, comments, etc. can be created separately.

Type:

Within the index, there are data types. The index usually carries one type, but we can add multiple types as and when required. It helps in faster-searching capabilities. Yet, types are critical, the same key (or field) must not hold multiple data types. Ignorance of this can lead to faulty search or unexpected results.

1. install java environment

  1. # Add the PPA
  2. sudo add-apt-repository ppa:webupd8team/java
  3. # Update the package repository
  4. sudo apt-get update
  5. # download java8 installer
  6. sudo apt-get install oracle-java8-installer
  7. //maybe config java version
  8. sudo update-alternatives --config java

verify java version

  1. java -version
  2. ava version "1.8.0_181"
  3. Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
  4. Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

2. install elastic search server , we will use 6.4.4 version
The installation docs can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/6.4/getting-star...
The analyzer ik for Chinese character can be found here: https://github.com/medcl/elasticsearch-analysis-ik/releases

  1. //wget <a href="https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.1.deb
  2. //wget">https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.1...</a> <a href="https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.1.deb.sha512
  3. shasum">https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.1...</a> -a 512 -c elasticsearch-6.4.2.deb.sha512
  4. sudo dpkg -i elasticsearch-6.4.2.deb
  5.  
  6. // /usr/share/elasticsearch/plugins
  7. cd /usr/share/elasticsearch/bin
  8. //./bin/elasticsearch-plugin plugin_url
  9. ./elasticsearch-plugin install <a href="https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.4.1/elasticsearch-analysis-ik-6.4.1.zip
  10.  
  11. sudo">https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6....</a> systemctl start elasticsearch.service
  12. sudo systemctl stop elasticsearch.service
  13. sudo systemctl restart elasticsearch.service

Config file of elastic server:

  1. sudo vim /etc/elasticsearch/elasticsearch.yml
  2. //Uncomment cluster.name and node.name

Verify status:

  1. $ curl -X GET '<a href="http://localhost:9200'
  2. #">http://localhost:9200'
  3. #</a> get all indices
  4. $ curl -X GET '<a href="http://localhost:9200/_cat/indices'

">http://localhost:9200/_cat/indices'
[/geshifilter-ruby]

3. install elastic driver for lumen 5.6 via composer:

  1. # <a href="https://github.com/ErickTamayo/laravel-scout-elastic
  2. composer">https://github.com/ErickTamayo/laravel-scout-elastic
  3. composer</a> require tamayo/laravel-scout-elastic

add the Scout service provider and the package service provider in your app.php config:

  1. // config/app.php
  2. 'providers' => [
  3. ...
  4. Laravel\Scout\ScoutServiceProvider::class,
  5. ...
  6. ScoutEngines\Elasticsearch\ElasticsearchProvider::class,
  7. ],

Setting up Elasticsearch configuration:

  1. // config/scout.php
  2. // Set your driver to elasticsearch
  3. 'driver' => env('SCOUT_DRIVER', 'elasticsearch'),
  4.  
  5. ...
  6. 'elasticsearch' => [
  7. 'index' => env('ELASTICSEARCH_INDEX', 'laravel'),
  8. 'hosts' => [
  9. env('ELASTICSEARCH_HOST', '<a href="http://localhost'">http://localhost'</a>),
  10. ],
  11. ],
  12. ...

4. create a console command to generate indice

  1. // app/Console/Commands/ESInit.php
  2. <?php
  3.  
  4. namespace App\Console\Commands;
  5.  
  6. use GuzzleHttp\Client;
  7. use Illuminate\Console\Command;
  8.  
  9. class ESInit extends Command
  10. {
  11. /**
  12. * The name and signature of the console command.
  13. *
  14. * @var string
  15. */
  16. protected $signature = 'es:init';
  17.  
  18. /**
  19. * The console command description.
  20. *
  21. * @var string
  22. */
  23. protected $description = 'Init es to create index';
  24.  
  25. /**
  26. * Create a new command instance.
  27. *
  28. */
  29. public function __construct()
  30. {
  31. parent::__construct();
  32. }
  33.  
  34. /**
  35. * Execute the console command.
  36. *
  37. * @return mixed
  38. */
  39. public function handle()
  40. {
  41. $client = new Client();
  42. $this->createTemplate($client);
  43. $this->createIndex($client);
  44. }
  45.  
  46. protected function createIndex(Client $client)
  47. {
  48. $url = config('scout.elasticsearch.hosts')[0] . ':9200/' . config('scout.elasticsearch.index');
  49. $client->put($url, [
  50. 'json' => [
  51. 'settings' => [
  52. 'refresh_interval' => '5s',
  53. 'number_of_shards' => 1,
  54. 'number_of_replicas' => 0,
  55. ],
  56. 'mappings' => [
  57. '_default_' => [
  58. '_all' => [
  59. 'enabled' => false
  60. ]
  61. ]
  62. ]
  63. ]
  64. ]);
  65. }
  66.  
  67. protected function createTemplate(Client $client)
  68. {
  69. $url = config('scout.elasticsearch.hosts')[0] . ':9200/' . '_template/rtf';
  70. $client->put($url, [
  71. 'json' => [
  72. 'template' => '*',
  73. 'settings' => [
  74. 'number_of_shards' => 1
  75. ],
  76. 'mappings' => [
  77. '_default_' => [
  78. 'dynamic_templates' => [
  79. [
  80. 'strings' => [
  81. 'match_mapping_type' => 'string',
  82. 'mapping' => [
  83. 'type' => 'text',
  84. 'analyzer' => 'ik_smart',
  85. 'ignore_above' => 256,
  86. 'fields' => [
  87. 'keyword' => [
  88. 'type' => 'keyword'
  89. ]
  90. ]
  91. ]
  92. ]
  93. ]
  94. ]
  95. ]
  96. ]
  97. ]
  98. ]);
  99.  
  100. }
  101. }
  1. <?php
  2.  
  3. namespace App\Console;
  4.  
  5. use Illuminate\Console\Scheduling\Schedule;
  6. use Laravel\Lumen\Console\Kernel as ConsoleKernel;
  7.  
  8. class Kernel extends ConsoleKernel
  9. {
  10. /**
  11.   * The Artisan commands provided by your application.
  12.   *
  13.   * @var array
  14.   */
  15. protected $commands = [
  16. //mail test
  17. Commands\SendMailCommand::class,
  18. Commands\RedisSubscribe::class,
  19. Commands\ESInit::class,
  20. ];
  21.  
  22. /**
  23.   * Define the application's command schedule.
  24.   *
  25.   * @param \Illuminate\Console\Scheduling\Schedule $schedule
  26.   * @return void
  27.   */
  28. protected function schedule(Schedule $schedule)
  29. {
  30. //
  31. }
  32. }

Then run artisan command to generate indice

  1. php artisan es:init

4. Vue form

  1. form: {
  2. gender: '',
  3. last_title: '',
  4. owner: '',
  5. recent_updated: '',
  6. age_min: '',
  7. age_max: '',
  8. language: '',
  9. service_year_min: '',
  10. service_year_max: '',
  11. status: '',
  12. education: '',
  13. now_location: '',
  14. forward_location: ''
  15. }
  1. <form class="col-sm-10">
  2.  
  3. <div class="input-group col-sm-6">
  4. <input type="text" class="form-control" v-model.trim="query" placeholder="请输入简历关键字" @keyup.enter="checkForm() ? showAlert() : search()">
  5. <input type="text" style="display:none" />
  6. <span class="input-group-append">
  7. <b-btn v-b-toggle.collapse1 variant="custom btn-bordered btn-sm" style="background-color:#15a1e5">高级选项</b-btn>
  8. </span>
  9. <span v-if="!searching" class="input-group-append">
  10. <button type="button" class="btn btn-custom btn-bordered btn-sm" v-on:click="checkForm() ? showAlert() : search()">搜索</button>
  11. </span>
  12.  
  13. <span v-else class="input-group-append btn-custom-grey">
  14. <button type="button" class="btn btn-custom btn-bordered btn-sm">搜索中...</button>
  15. </span>
  16. </div>
  17.  
  18. <b-collapse id="collapse1" class="mt-2 col">
  19. <b-card>
  20. <div class="row col-sm-12">
  21. <div class="form-group row col-sm-4" style="margin-top:auto;">
  22. <label for="sex" >性 别</label>
  23. <div class="col">
  24. <div class="radio radio-info form-check-inline">
  25. <input type="radio" v-model="form.gender" id="radio1" value="" checked>
  26. <label for="radio1">
  27. 不限
  28. </label>
  29. </div>
  30. <div class="radio radio-info form-check-inline">
  31. <input type="radio" v-model="form.gender" id="radio2" value="male">
  32. <label for="radio2">
  33. </label>
  34. </div>
  35. <div class="radio radio-info form-check-inline">
  36. <input type="radio" v-model="form.gender" id="radio3" value="female">
  37. <label for="radio3">
  38. </label>
  39. </div>
  40. </div>
  41. </div>
  42. <div class="form-group row col-sm-3">
  43. <label for="lasttitle" >目前职位</label>
  44. <div class="col-sm-8">
  45. <input type="text" parsley-trigger="change" v-model.trim="form.last_title" title="目前职位" :class="form.last_title ? 'ad-search-form' : ''" placeholder="" class="form-control" id="lasttitle">
  46. </div>
  47. </div>
  48. <div class="form-group row col-sm-4">
  49. <label for="update">更新日期</label>
  50. <div class="col-sm-8">
  51. <select id="update" v-model="form.recent_updated" :class="form.recent_updated ? 'ad-search-form' : ''" class="form-control custom-select" data-live-search="true">
  52. <option value="">不限</option>
  53. <option value="three-days">最近三天</option>
  54. <option value="one-week">最近一周</option>
  55. <option value="two-week">最近两周</option>
  56. <option value="one-month">最近一个月</option>
  57. <option value="two-month">最近两个月</option>
  58. </select>
  59. </div>
  60. </div>
  61.  
  62. <div class="form-group row col">
  63. <label for="followpeople" >操作人</label>
  64. <div class="col">
  65. <select id="followpeople" class="form-control custom-select" v-model="form.owner" :class="form.owner ? 'ad-search-form' : ''" data-live-search="true" onmousedown="if(options.length>7){this.size=8}" onblur="this.size=0" onchange="this.size=0" style="z-index: 1;position: absolute">
  66. <option value="">不限</option>
  67. <option v-for="singleUser in allUsers" :value="singleUser.id">{{ singleUser.name }}</option>
  68. </select>
  69. </div>
  70. </div>
  71.  
  72. </div>
  73. <div class="row col-sm-12">
  74. <div class="form-group row col-sm-4">
  75. <label for="age" >年 龄</label>
  76. <div class="input-group input-daterange col-sm-8" id="age">
  77. <input type="number" min="18" max="50" v-model="form.age_min" class="form-control" :class="form.age_min ? 'ad-search-form' : ''" placeholder="" title="年龄范围">
  78. <i class="mdi mdi-minus " style="margin-top:12px;width:15px;"></i>
  79. <input type="number" min="18" max="50" v-model="form.age_max" class="form-control" :class="form.age_max ? 'ad-search-form' : ''" placeholder="">
  80. </div>
  81. </div>
  82. <div class="form-group row col-sm-3">
  83. <label for="language" >语言能力</label>
  84. <div class="col-sm-8">
  85. <select id="language" class="form-control col custom-select" v-model="form.language" :class="form.language ? 'ad-search-form' : ''" data-live-search="true">
  86. <option value="">不限</option>
  87. <option value="english">英语</option>
  88. <option value="japanese">日语</option>
  89. <option value="korean">韩语</option>
  90. <option value="russian">俄语</option>
  91. <option value="franch">法语</option>
  92. <option value="german">德语</option>
  93. <option value="arabic">阿拉伯语</option>
  94. <option value="other">其他</option>
  95. </select>
  96. </div>
  97. </div>
  98. <div class="form-group row col-sm-4">
  99. <label for="workyear" >工作年限</label>
  100. <div class="input-group input-daterange col-sm-8" id="workyear">
  101. <input type="number" v-model="form.service_year_min" min="0" max="40" class="form-control" placeholder="" :class="form.service_year_min ? 'ad-search-form' : ''" title="工作年限">
  102. <i class="mdi mdi-minus " style="margin-top:12px;width:15px"></i>
  103. <input type="number" v-model="form.service_year_max" min="0" max="40" class="form-control" :class="form.service_year_max ? 'ad-search-form' : ''" placeholder="">
  104. </div>
  105. </div>
  106. <div class="form-group row col">
  107. <label for="followstate" >追踪状态</label>
  108. <div class="col">
  109. <select id="followstate" v-model="form.status" class="form-control custom-select" :class="form.status ? 'ad-search-form' : ''" data-live-search="true" style="position: absolute;">
  110. <option value="">不限</option>
  111. <option value="processing">流程中</option>
  112. <option v-for="(statu,index) in status" :value="statu.id">{{ statu.name }}</option>
  113. </select>
  114. </div>
  115. </div>
  116. </div>
  117. <div class="row col-sm-12">
  118. <div class="form-group row col-sm-4">
  119. <label for="education" class="">学 历</label>
  120. <div class="col-sm-8">
  121. <select id="education" v-model="form.education" class="form-control col custom-select" :class="form.education ? 'ad-search-form' : ''" data-live-search="true">
  122. <option value="">不限</option>
  123. <option value="1">大专</option>
  124. <option value="2">本科</option>
  125. <option value="3">硕士</option>
  126. <option value="4">博士</option>
  127. </select>
  128. </div>
  129. </div>
  130. <div class="form-group row col-sm-3">
  131. <label for="liveaddress" >现居住地</label>
  132. <div class="col-sm-8">
  133. <input type="text" v-model.trim="form.now_location" parsley-trigger="change" required title="现居住地" :class="form.now_location ? 'ad-search-form' : ''" placeholder="" class="form-control" id="liveaddress">
  134. </div>
  135. </div>
  136. <div class="form-group row col-sm-4">
  137. <label for="workingplace" class="">期待工作地</label>
  138. <div class="col-sm-8">
  139. <input type="text" v-model.trim="form.forward_location" name="text" parsley-trigger="change" required placeholder="" title="期待工作地" class="form-control" :class="form.forward_location ? 'ad-search-form' : ''" id="workingplace">
  140. </div>
  141. </div>
  142.  
  143. </div>
  144.  
  145. </b-card>
  146. </b-collapse>
  147.  
  148. </form>
  1. search: function () {
  2. this.resetSearch = true
  3. this.searching = true
  4. this.axios.post('search?q=' + this.query + '&arg=' + JSON.stringify(this.form) )
  5. .then((res) => {
  6. // console.log(res.data.data)
  7. })
  8. .catch((error) => {
  9. console.log(error)
  10. })
  11. },

5. Controller logic in lumen:
Update driver code for supporting gte range query(https://github.com/ErickTamayo/laravel-scout-elastic/issues/108):

  1. //vendor/tamayo/laravel-scout-elastic/src/ElasticsearchEngine.php 195 line
  2. protected function filters(Builder $builder)
  3. {
  4. return collect($builder->wheres)->map(function ($value, $key) {
  5.  
  6. if (is_array($value)) {
  7. if (isset($value['lt']) || isset($value['gt']) || isset($value['gte']) || isset($value['lte'])) {
  8. return ['range' => [$key => $value]];
  9. } else {
  10. return ['terms' => [$key => $value]];
  11. }
  12. } else if (strpos($key, '.')) {
  13. return ['match' => [$key => $value]];
  14. }
  15.  
  16. return ['match_phrase' => [$key => $value]];
  17. })->values()->all();
  18. }
  1. /**
  2.   * Search the Candidate table.
  3.   *
  4.   * @param Request $request
  5.   * @return mixed
  6.   */
  7. public function search(Request $request)
  8. {
  9. // First we define the error message we are going to show if no keywords
  10. // existed or if no results found.
  11.  
  12. $error = ['error' => '无搜索结果,请尝试其它关键字'];
  13.  
  14. // Making sure the user entered a keyword.
  15. if($request->has('q')) {
  16.  
  17. // Using the Laravel Scout syntax to search the resume table.
  18. //$results = Candidate::search($request->get('q'))->get();
  19. $arg = $request->get('arg');
  20. $args = json_decode($arg);
  21.  
  22. $results = Resume::search($request->get('q'))->orderBy('updated_at.date.keyword', 'desc');
  23.  
  24. if(isset($args->gender) && !empty($args->gender)){ //gender
  25. $results = $results->where('gender', $args->gender);
  26. }
  27.  
  28. if(isset($args->last_title) && !empty($args->last_title)){ // last title
  29. $results = $results->where('last_title', $args->last_title);
  30. }
  31.  
  32. if(isset($args->now_location) && !empty($args->now_location)){ //now location
  33. $results = $results->where('now_location', $args->now_location);
  34. }
  35.  
  36. if(isset($args->forward_location) && !empty($args->forward_location)){ //forward location
  37. $results = $results->where('forward_location', $args->forward_location);
  38. }
  39.  
  40. if( isset($args->age_min) && isset($args->age_max) && !empty($args->age_min) && !empty($args->age_max) && $args->age_max >= $args->age_min){ // age
  41.  
  42. $age_q = [];
  43. for($i= $args->age_min; $i<= $args->age_max; $i++){
  44. $age_q[] = $i;
  45. }
  46. $results = $results->where('age', $age_q);
  47. }
  48.  
  49.  
  50. if( isset($args->service_year_min) && isset($args->service_year_max) && !empty($args->service_year_min) && !empty($args->service_year_max) && $args->service_year_max >= $args->service_year_min){ // service year
  51.  
  52. $service_year_q = [];
  53. for($j= $args->service_year_min; $j<= $args->service_year_max; $j++){
  54. $service_year_q[] = $j;
  55. }
  56. $results = $results->where('service_year', $service_year_q);
  57. }
  58.  
  59.  
  60. if( isset($args->education) && !empty($args->education)){ // education
  61.  
  62. $education_q = [];
  63.  
  64. if($args->education == 1){
  65. $education_q[] = '大专';
  66. } elseif ($args->education == 2){
  67. $education_q[] = '本科';
  68. } elseif ($args->education == 3){
  69. $education_q = ['硕士', '研究生'];
  70. } else {
  71. $education_q[] = '博士';
  72. }
  73.  
  74. $results = $results->where('education', $education_q);
  75. }
  76.  
  77.  
  78. if( isset($args->language) && !empty($args->language) ){ // language
  79.  
  80. $lan_arr = [
  81. 'english' => '英语',
  82. 'korean' => '韩语',
  83. 'russian' => '俄语',
  84. 'japanese' => '日语',
  85. 'franch' => '法语',
  86. 'german' => '德语',
  87. 'arabic' => '阿拉伯语',
  88. 'other' => '其他'
  89. ];
  90.  
  91. $results = $results->where('language', $lan_arr[$args->language]);
  92. }
  93.  
  94.  
  95. if( isset($args->status) && !empty($args->status) ){ //status
  96.  
  97. if( $args->status == 'processing'){
  98. $results = $results->where('self_status', 'in_action');
  99. } else {
  100. $status_arr = Statu::get()->pluck('name', 'id');
  101. $results = $results->where('status', $status_arr[$args->status]);
  102. }
  103. }
  104.  
  105.  
  106. if( isset($args->owner) && !empty($args->owner)){ //owner
  107. $users = User::get()->pluck('name', 'id');
  108. $results = $results->where('owner', $users[$args->owner]);
  109. }
  110.  
  111. if(isset($args->recent_updated) && !empty($args->recent_updated)){ // recent updated
  112. //'2018-12-03 16:36:16'
  113. $recent_q = [
  114. 'three-days' => Carbon::today()->subDay(3)->toDateTimeString(),
  115. 'one-week' => Carbon::today()->subWeek()->toDateTimeString(),
  116. 'two-week' => Carbon::today()->subWeek(2)->toDateTimeString(),
  117. 'one-month' => Carbon::today()->subMonth()->toDateTimeString(),
  118. 'two-month' => Carbon::today()->subMonth(2)->toDateTimeString()
  119. ];
  120.  
  121. $results = $results->where('updated_at.date.keyword', ['gte' => $recent_q[$args->recent_updated]]);
  122. }
  123.  
  124.  
  125. $results = $results->paginate($request->input('pagesetting'));
  126.  
  127. // $results = Resume::search($request->get('q'))->orderBy('updated_at.date.keyword', 'desc')->where('service_year', $service_year_q)->paginate(10);
  128.  
  129. $results->load('candidate');
  130. // If there are results return them, if none, return the error message.
  131. return $results->count() ? $this->resumesReturn($results) : $error;
  132.  
  133. }
  134.  
  135. // Return the error message if no keywords existed
  136. return $error;
  137. }

6. Modify Resume model to be indexed:

  1. // app/Resume.php
  2. use Laravel\Scout\Searchable;
  3.  
  4. -------------------------
  5.  
  6. /**
  7.   * Get the indexable data array for the model.
  8.   *
  9.   * @return array
  10.   */
  11. public function toSearchableArray()
  12. {
  13. //return $array;
  14. return [
  15. 'objectId' => $this->id,
  16. 'updated_at' => $this->updated_at,
  17. 'name' => $this->candidate->name,
  18. 'gender' => $this->candidate->gender,
  19. 'age' => $this->candidate->age,
  20. 'mobile' => $this->candidate->mobile,
  21. 'email' => $this->candidate->email,
  22. 'education' => $this->candidate->education,
  23. 'now_location' => $this->candidate->now_location,
  24. 'service_year' => $this->candidate->service_year,
  25. 'language' => $this->candidate->language,
  26. 'forward_location' => $this->forward_location,
  27. 'last_title' => $this->last_title,
  28. 'status' => $this->statu->name,
  29. 'self_status' => $this->self_status,
  30. 'source'=> $this->sources->name,
  31. 'owner' => $this->user->name,
  32. 'career_objective' => $this->career_objective,
  33. 'self_evaluations' => $this->self_evaluations,
  34. 'works_info' => $this->works_info,
  35. 'projects_info' => $this->projects_info,
  36. 'education_info' => $this->education_info,
  37. 'comprehensive_skills' => $this->comprehensive_skills
  38. ];
  39. }
  40.  
  41.  
  42.  
  43. /**
  44.   * Get the index name for the model.
  45.   *
  46.   * @return string
  47.   */
  48. public function searchableAs()
  49. {
  50. return 'resumes_index';
  51. }
  52.  
  53. public function shouldBeSearchable()
  54. {
  55. return $this->is_valid;
  56. }

7. Import data from database to elasticsearch

  1. php artisan scout:import "App\Resume"

Then Volia , enjoy your search functionality