Satomi が 2021年05月16日03時53分32秒に編集

コメント無し

本文の変更

# 概要ブラウザ上でWebカメラを使用し、Face-api.jsを用いて眠気検知を行う。寝ていることを検知したらWi-Fi経由でマウスパッドに搭載されたObnizに指令が送られ、目覚ましアクションが発生する。目覚ましアクションは、眠気レベルによって変化する。 - レベル１　空調の温度を下げる - レベル２　マウスパッドを振動させる - レベル３　ブザーを鳴らす - レベル４　ライトを消す（疲れているため、目覚ましを諦める） # デモ動画 @[youtube](https://youtu.be/tXlF-e53FF8) # 構成図と部品

![構成図](https://camo.elchika.com/41fffe9888ae674daa774a86d08ad531227b7b2b/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f34306235383336642d363666622d346462642d396437662d373738393336653431316536/)

![ハードウェア構成図](https://camo.elchika.com/41fffe9888ae674daa774a86d08ad531227b7b2b/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f34306235383336642d363666622d346462642d396437662d373738393336653431316536/)

![ソフトウェア構成図](https://camo.elchika.com/5ef7fa8d2807735e73c96f4b3db2f22296083eb6/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f33633537303030382d316665612d343364352d613263312d306638653466376336333238/)

**部品** - obnizBoard　1個 - Buzzer　1個 - 赤外線受信モジュール　1個 - 赤外線LED　1個 - DCモーター　1個 - 抵抗(10kΩ) 　1個 - LEGOブロック # 配線とピン配置 ![配線](https://camo.elchika.com/cce1828c93d6fcf11938f4ac169ddce7421d618d/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f61303564653765622d313135322d343761322d616337612d323764653264646262363263/) **ピン配置** | PIN No | 用途 | |:---:|:---| | 0 | 赤外線受信モジュール Output | | 1 | 赤外線受信モジュール Gnd | | 2 | 赤外線受信モジュール Vcc | | 3 | 赤外線LED Cathode | | 4 | 赤外線LED Anode | | 5 | ブザー Signal | | 6 | ブザー Gnd | | 10 | DCモーター forward | | 11 | DCモーター back | # 機能説明

### 眠気検知について WebカメラとFace-api.jsで実現。動画 : [https://youtu.be/gzsYHE1Xhwg](https://youtu.be/gzsYHE1Xhwg) ![キャプションを入力できます](https://camo.elchika.com/2a8708befbfdde04397b63c1aa17fbe27bf1bae4/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f39396261323939392d333765372d343562662d386637322d616139383035346635303463/) 眠気検知方法はFace-api.jsを用いて目の特徴点を抽出。【参考】[https://github.com/justadudewhohacks/face-api.js/](https://github.com/justadudewhohacks/face-api.js/) 目の開閉判定は以下記事を参考にした。

【参考】[https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/](https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/>)

【参考】[https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/](https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/)

目の閉じている時間から、眠っているか否かの判定をする。眠っていると判定された回数が多くなるにつれ、眠気レベルを上げていく。 ### 家電の操作について Obnizに赤外線受信モジュールと赤外線LED を接続。 ![obniz：赤外線受信モジュール、赤外線LED、ブザー](https://camo.elchika.com/97fc94756adcdfeec8bafa8ae992c03969101207/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f36316439343836662d613035662d346134352d613965322d306466323038336463663561/) ブラウザ上の登録画面から、赤外線受信モジュールに向けてリモコンを操作することで家電の各操作の赤外線データを登録できる。 ![赤外線データ登録画面](https://camo.elchika.com/96f387c4c8b503ff363e320f1d2aa9b45bed2c40/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f39613932346439322d383639632d343462642d623835312d393666366632316162393438/) :::plantuml:登録状態遷移 @startuml (*) --> [アプリ開始] "確認中" If "登録済み？" then --> [Yes] "登録済み" else --> [No] "未登録" Endif --> [クリック] "登録中" --> [リモコン操作　　　　　　] "登録済み" --> [クリック　　] "登録中" @enduml ::: ### 振動機構について DCモーターを正逆交互に回転させ、レゴブロックで作ったラック・アンド・ピニオンを動作させる。動画：[https://youtu.be/2pcjLCafXPY](https://youtu.be/2pcjLCafXPY) ![LEGO 振動機構](https://camo.elchika.com/b5143742ba04e22813e2c3cfcb1d3ff6fa636169/687474703a2f2f73746f726167652e676f6f676c65617069732e636f6d2f656c6368696b612f76312f757365722f30323366303566662d306139382d343961382d396365392d3432336564613332333061332f37613230633832382d356166632d346133382d626439392d316163396638366233316535/) # ソースコード ```arduino:Main(GUI+眠気検知+Obniz操作) <html> <head> <meta charset="utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" /> <script src="https://code.jquery.com/jquery-3.2.1.min.js"></script> <script src="https://unpkg.com/obniz@3.x/obniz.js" crossorigin="anonymous" ></script> <script src="https://unpkg.com/obniz-parts-kits@0.16.0/ui/index.js"></script> <script src="https://obniz.com/users/5143/repo/storage.json"></script> <script src="https://obniz.com/users/5143/repo/register_data.js"></script> <script src="https://obniz.com/users/5143/repo/face-api.min.js"></script> </head> <body>  <div style="position: relative"> <video id="video" onloadedmetadata="onPlay(this)" muted autoplay width="640" height="480"></video> <canvas id="facecanvas" style=" position: absolute; top: 0; left: 0;" width="640" height="480"></canvas> </div> <div id="obniz-debug"></div> <div class="container"> <div class="text-center"> <h3>Sleepiness Detection</h3> <label>電源 ON : </label> <button class="btn btn-link" id="power_on_reg">確認中</button> <button class="btn btn-info" id="power_on_send">実行</button> <label id="power_on_sts"></label> <br> <label>電源 OFF : </label> <button class="btn btn-link" id="power_off_reg">確認中</button> <button class="btn btn-info" id="power_off_send">実行</button> <label id="power_off_sts"></label> <br> <label>ライト ON : </label> <button class="btn btn-link" id="light_on_reg">確認中</button> <button class="btn btn-info" id="light_on_send">実行</button> <label id="light_on_sts"></label> <br> <label>ライト OFF : </label> <button class="btn btn-link" id="light_off_reg">確認中</button> <button class="btn btn-info" id="light_off_send">実行</button> <label id="light_off_sts"></label> <br> <label>エアコン ON : </label> <button class="btn btn-link" id="aircon_on_reg">確認中</button> <button class="btn btn-info" id="aircon_on_send">実行</button> <label id="aircon_on_sts"></label> <br> <label>エアコン OFF : </label> <button class="btn btn-link" id="aircon_off_reg">確認中</button> <button class="btn btn-info" id="aircon_off_send">実行</button> <label id="aircon_off_sts"></label> <br> <label>温度上げる : </label> <button class="btn btn-link" id="thm_up_reg">確認中</button> <button class="btn btn-info" id="thm_up_send">実行</button> <label id="thm_up_sts"></label> <br> <label>温度下げる : </label> <button class="btn btn-link" id="thm_down_reg">確認中</button> <button class="btn btn-info" id="thm_down_send">実行</button> <label id="thm_down_sts"></label> <br> </div> </div> <script> var obniz = new Obniz("OBNIZ_ID_HERE"); var ObnizId = document.getElementById('ObnizId'); var power_on_reg = document.getElementById('power_on_reg'); var power_on_send = document.getElementById('power_on_send'); var power_on_sts = document.getElementById('power_on_sts'); var power_off_reg = document.getElementById('power_off_reg'); var power_off_send = document.getElementById('power_off_send'); var power_off_sts = document.getElementById('power_off_sts'); var light_on_reg = document.getElementById('light_on_reg'); var light_on_send = document.getElementById('light_on_send'); var light_on_sts = document.getElementById('light_on_sts'); var light_off_reg = document.getElementById('light_off_reg'); var light_off_send = document.getElementById('light_off_send'); var light_off_sts = document.getElementById('light_off_sts'); var aircon_on_reg = document.getElementById('aircon_on_reg'); var aircon_on_send = document.getElementById('aircon_on_send'); var aircon_on_sts = document.getElementById('aircon_on_sts'); var aircon_off_reg = document.getElementById('aircon_off_reg'); var aircon_off_send = document.getElementById('aircon_off_send'); var aircon_off_sts = document.getElementById('aircon_off_sts'); var thm_up_reg = document.getElementById('thm_up_reg'); var thm_up_send = document.getElementById('thm_up_send'); var thm_up_sts = document.getElementById('thm_up_sts'); var thm_down_reg = document.getElementById('thm_down_reg'); var thm_down_send = document.getElementById('thm_down_send'); var thm_down_sts = document.getElementById('thm_down_sts'); var res_promise; const canvas = document.getElementById( 'facecanvas' ); const videoEl = document.getElementById( 'video' ); const ctx = canvas.getContext("2d"); ctx.font = "32px serif"; ctx.fillStyle = "Red"; const inputSize = 224; const scoreThreshold = 0.5; const options = new faceapi.TinyFaceDetectorOptions({ inputSize, scoreThreshold }); var PassSec = 0; // 秒数カウント用変数 var WakeupCount = 0; // 秒数カウント用変数 var flg = true; var sleepinessCount = 0; var sleepiness_level = 0; var power_off_flag = false; var thm_down_flag = false; var PassSeccount_flag = false; timerID = setInterval('countup()',1000); //1秒毎にcountup()を呼び出し const xml = new XMLHttpRequest(); xml.open("POST", "https://maker.ifttt.com/trigger/WakeupAlarm/with/key/d5nIKdxHkXiL3bjtNe6_t_", false); xml.setRequestHeader("content-type", "application/x-www-form-urlencoded;charset=UTF-8","Access-Control-Allow-Origin", "https://maker.ifttt.com/trigger/WakeupAlarm/with/key/d5nIKdxHkXiL3bjtNe6_t_"); function countup() { PassSec++; PassSeccount_flag = true; } async function onPlay() { if(videoEl.paused || videoEl.ended || !faceapi.nets.tinyFaceDetector.params) return setTimeout(() => onPlay()) // face-apiを使って, 両目の特徴点を取得 const result = await faceapi.detectSingleFace(videoEl, options).withFaceLandmarks() if (result) { const dims = faceapi.matchDimensions(canvas, videoEl, true) const resizedResult = faceapi.resizeResults(result, dims) const leftEye = result.landmarks.getLeftEye() const rightEye = result.landmarks.getRightEye() // EAR(Eye Aspect Ratio)を算出 var a_l = Math.sqrt( Math.pow( leftEye[1].x-leftEye[5].x, 2 ) + Math.pow( leftEye[1].y-leftEye[5].y, 2 ) ) ; var b_l = Math.sqrt( Math.pow( leftEye[2].x-leftEye[4].x, 2 ) + Math.pow( leftEye[2].y-leftEye[4].y, 2 ) ) ; var c_l = Math.sqrt( Math.pow( leftEye[0].x-leftEye[3].x, 2 ) + Math.pow( leftEye[0].y-leftEye[3].y, 2 ) ) ; var EAR_L = ( a_l + b_l ) / ( 2 * c_l ) ; var a_r = Math.sqrt( Math.pow( rightEye[1].x-rightEye[5].x, 2 ) + Math.pow( rightEye[1].y-rightEye[5].y, 2 ) ) ; var b_r = Math.sqrt( Math.pow( rightEye[2].x-rightEye[4].x, 2 ) + Math.pow( rightEye[2].y-rightEye[4].y, 2 ) ) ; var c_r = Math.sqrt( Math.pow( rightEye[0].x-rightEye[3].x, 2 ) + Math.pow( rightEye[0].y-rightEye[3].y, 2 ) ) ; var EAR_R = ( a_r + b_r ) / ( 2 * c_r ) ; var EAR = ( EAR_R + EAR_L ) / 2; ctx.fillText("眠気検知回数：" + sleepinessCount, 20, 20); ctx.fillText("レベル：" + sleepiness_level, 20, 40); ctx.fillText(flg, 20, 100); //目があいているフレーム数をカウント if(EAR > 0.33){ // 使用者によりチューニングが必要!! WakeupCount++; } //目があいているフレーム数をが5フレーム以上は起きている if(WakeupCount > 5){ PassSec = 0; WakeupCount = 0; } if(PassSec == 5){ WakeupCount = 0; if(PassSeccount_flag == true){ sleepinessCount ++ ; if(sleepinessCount < 2){ sleepiness_level = 1 ; }else if(sleepinessCount < 4){ sleepiness_level = 2 ; }else if(sleepinessCount < 6){ sleepiness_level = 3 ; }else if(sleepinessCount < 7){ sleepiness_level = 4 ; }else{ sleepiness_level = 5 ; } } ctx.fillText("SLEEP", 20, 120); flg = false; }else if(PassSec > 5){ ctx.fillText("SLEEP", 20, 120); }else{ ctx.fillText("AWAKE", 20, 120); flg = true; } } PassSeccount_flag = false; setTimeout(() => onPlay()) }; async function run(){ await faceapi.nets.tinyFaceDetector.load('https://obniz.com/users/5143/repo/') await faceapi.loadFaceLandmarkModel('https://obniz.com/users/5143/repo/') const stream = await navigator.mediaDevices.getUserMedia({ video: {} }) videoEl.srcObject = stream; } $(document).ready(function() { run(); }); // called on online obniz.onconnect = async function() { var sensor = obniz.wired('IRSensor', {vcc:2, gnd:1, output: 0}); var led = obniz.wired('InfraredLED', {anode: 4, cathode: 3}); var speaker = obniz.wired("Speaker", {signal:5 , gnd:6 }); var dcmotor = obniz.wired("DCMotor",{forward:10, back:11}); var power_on = []; var power_off = []; var light_on = []; var light_off = []; // Load console.log('Load Start'); var power_on_reg_data = new Register(obniz, "PowerOn" , sensor, led, power_on_reg , power_on_sts); var power_off_reg_data = new Register(obniz, "PowerOff" , sensor, led, power_off_reg , power_off_sts); var light_on_reg_data = new Register(obniz, "light_on" , sensor, led, light_on_reg , light_on_sts); var light_off_reg_data = new Register(obniz, "light_off" , sensor, led, light_off_reg , light_off_sts); var aircon_on_reg_data = new Register(obniz, "aircon_on" , sensor, led, aircon_on_reg , aircon_on_sts); var aircon_off_reg_data = new Register(obniz, "aircon_off" , sensor, led, aircon_off_reg , aircon_off_sts); var thm_up_reg_data = new Register(obniz, "thm_up" , sensor, led, thm_up_reg , thm_up_sts); var thm_down_reg_data = new Register(obniz, "thm_down" , sensor, led, thm_down_reg , thm_down_sts); power_on_reg_data.LoadData(); power_off_reg_data.LoadData(); light_on_reg_data.LoadData(); light_off_reg_data.LoadData(); aircon_on_reg_data.LoadData(); aircon_off_reg_data.LoadData(); thm_up_reg_data.LoadData(); thm_down_reg_data.LoadData(); $("#power_on_reg").click(function() { power_on_reg_data.RegisterData(); sensor.ondetect = function(arr) { power_on_reg_data.RegisterComp(arr); } }); $("#power_off_reg").click(function() { power_off_reg_data.RegisterData(); sensor.ondetect = function(arr) { power_off_reg_data.RegisterComp(arr); } }); $("#light_on_reg").click(function() { light_on_reg_data.RegisterData(); sensor.ondetect = function(arr) { light_on_reg_data.RegisterComp(arr); } }); $("#light_off_reg").click(function() { light_off_reg_data.RegisterData(); sensor.ondetect = function(arr) { light_off_reg_data.RegisterComp(arr); } }); $("#aircon_on_reg").click(function() { aircon_on_reg_data.RegisterData(); sensor.ondetect = function(arr) { aircon_on_reg_data.RegisterComp(arr); } }); $("#aircon_off_reg").click(function() { aircon_off_reg_data.RegisterData(); sensor.ondetect = function(arr) { aircon_off_reg_data.RegisterComp(arr); } }); $("#thm_up_reg").click(function() { thm_up_reg_data.RegisterData(); sensor.ondetect = function(arr) { thm_up_reg_data.RegisterComp(arr); } }); $("#thm_down_reg").click(function() { thm_down_reg_data.RegisterData(); sensor.ondetect = function(arr) { thm_down_reg_data.RegisterComp(arr); } }); // Send $("#power_on_send").click(async function() { power_on_reg_data.SendData(); }); $("#power_off_send").click(async function() { power_off_reg_data.SendData(); }); $("#light_on_send").click(async function() { light_on_reg_data.SendData(); }); $("#light_off_send").click(async function() { light_off_reg_data.SendData(); }); $("#aircon_on_send").click(async function() { aircon_on_reg_data.SendData(); }); $("#aircon_off_send").click(async function() { aircon_off_reg_data.SendData(); }); $("#thm_up_send").click(async function() { thm_up_reg_data.SendData(); }); $("#thm_down_send").click(async function() { thm_down_reg_data.SendData(); }); // called while online. obniz.onloop = async function() { const _sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); switch (sleepiness_level){ case 1: //空調の温度を下げる if(flg == false){ if( thm_down_flag == false){ aircon_on_reg_data.SendData(); await _sleep(500); thm_down_reg_data.SendData(); thm_down_flag = true; } } break; case 2: //マウスパットを振動させる if(flg == false){ dcmotor.power(70); for (let i = 0; i < 3; i++) { dcmotor.move(false); await _sleep(300); dcmotor.stop(); await _sleep(250); dcmotor.move(true); await _sleep(300); dcmotor.stop(); await _sleep(250); } } break; case 3:　//ブザーを鳴らす if(flg == false){ speaker.play(1000); } else{ speaker.stop(); } break; case 4:　//ライトを消す speaker.stop(); if( power_off_flag == false){ power_off_reg_data.SendData(); power_off_flag = true; } break; default: break; } }; }; // called on offline obniz.onclose = async function() { }; </script> </body> </html> ``` ```arduino:赤外線データ登録 var Register = function(obniz, key, sensor, led, reg_status, load_status) { this.obniz = obniz; this.key = key; this.sensor = sensor; this.led = led; this.reg_status = reg_status; this.load_status = load_status; this.data = []; this.LoadData = async function(){ var res_promise; res_promise = (ObnizUI.Util.loadFromStorage(this.key)); res_promise.then( response => { if (Array.isArray(response)){ this.reg_status.innerText = "登録済み"; this.data = response.concat(); } else{ this.reg_status.innerText = "未登録"; } }, error => { this.reg_status.innerText = "読み出し失敗"; } ); return 0; } this.RegisterData = async function(){ var res_data = []; if(this.reg_status.innerText == "登録中") { this.LoadData (); } else { this.reg_status.innerText = "登録中"; this.obniz.display.print("detecting..."); this.sensor.start(); } } this.RegisterComp = async function(arr){ var res_data = []; if (this.reg_status.innerText == "登録中"){ this.reg_status.innerText = "登録完了"; this.obniz.display.print("detecte success"); res_data = arr.concat(); ObnizUI.Util.saveToStorage(this.key, res_data); this.LoadData (); } } this.SendData = function(){ if ((this.reg_status.innerText == "登録済み") || (this.reg_status.innerText == "登録完了")) { this.led.send(this.data); } else { alert("未登録です"); } } }; ```