雑文発散

«前の日記(2014-10-22) 最新 次の日記(2014-10-24)» 編集
過去の日記

2014-10-23 [長年日記]

[Fluentd][Elasticsearch] Fluentd + Elasticsearch + Kibana での解析の構成を考える

昨日の日記「DDD (DoS Driven Development) を始めよう」からの続き。

DoS への対策で Fluentd + Elasticsearch + Kibana を使ったアクセスログ解析を第三tDiary.Netの上で行おうと思い立ったのは事実なんだけど、実はそれ以前から Elasticsearch 関連は勉強し始めていたのであった。

その勉強を実地で活かす場として、第三tDiary.Netのアクセスログを利用する。その結果として DoS 避けができるようになれば、オレに取ってのメリットが大きくなる。

Elasticsearch 関連を勉強するにあたって、参考にしているサイトや書籍を先に上げておく。これらが無ければ先に進めなかったかも。

サイト

書籍

さて、ここで話を元に戻そう。

今回のシステム構成を考えた。まずサーバは2台構成。第三tDiary.Netが動いているサーバとは別に解析用のサーバを用意する。用意するとは言っても、借りているけど遊び気味な VPS が元々あるのでそれを流用するイメージ。

Kibana で画面を見るのは手元の MacBook Air なので、いちおう描き加える。Kibana 本体は「解析サーバ」の上に置くとは思うけど、ちょっと図の簡略化をしたらこうなった。

fluentd + elasticsearch 構成図

第三tDiary.Netは apache2 で動いていて、アクセスログは普通に書き出している。この部分は今回は変えない。基本的には現在稼働中の tDiary 部分には手を入れない方針でいく。

Fluentd (td-agent) で apache2 のアクセスログを読んで、解析サーバの Elasticsearch サーバへ直接送信する。この間の通信は iptables によってアクセス元を制限しておく。うまくいったら ssh トンネルなどで経路の暗号化もちょっと考えるけど、最初はシンプルな構成にする。

Kibana から Elasticsearch へアクセスするために、nginx のリバースプロキシ機能を使う。Kibana (Ver3) は JavaScript から Elasticsearch に接続する形になる。

つまり、ブラウザを開いた場所の IP アドレスからの接続になるため、先ほどの iptables によるアクセス元制限との共存がうまくいかない。そこで Kibana からは一度 nginx に繋いで、そこで BASIC 認証などでアクセス制限を行なう構成にしてみた。

こういう考え方もどこかのサイトを参考にしたんだけど、辿りつけなくなってしまった。。。メモ重要なのに。

ということで、今回も実践には辿り着かずにこの日記を終えるけど、裏側(?)ではインストールやら設定やらを進めていて、少しずつ形になりつつある。まだ実際に Kibana で見るところまではできてないけど、、、もう少しじゃないかな、希望的観測としては(笑)

サーバ/インフラエンジニア養成読本 ログ収集〜可視化編 [現場主導のデータ分析環境を構築!] (Software Design plus)(養成読本編集部)

高速スケーラブル検索エンジン ElasticSearch Server (アスキー書籍)(Rafal Kuc (lにストローク符号、cにアクサン・テギュ付く)/Marek Rogozinski (nにアクサン・テギュ付く)/大岩 達也/大谷 純/兼山 元太/水戸 祐介/守谷 純之介/株式会社リクルートテクノロジーズ)


つづきの日記へ

本日のツッコミ(全46件) [ツッコミを入れる]
Richardgax (2025-10-09 17:23)

<br>Продаём велосипеды с сервисным обслуживанием <br> кракен онион тор кракен онион зеркало кракен даркнет маркет

Justinnib (2025-10-24 04:43)

Unlock your potential by claiming the latest https://www.freedomteamapexmarketinggroup.com/board/board_topic/8118484/7326862.htm. Whether you love football, cricket, or live casinos, this offer makes every bet count. <br> <br>

Justinnib (2025-10-24 20:07)

New users can now get rewarded instantly with the official https://cloudsvortex.com/unlocking-some-great-benefits-of-the-particular-1xbet-promotion-program-code/. Register, deposit, and experience high-odds betting with guaranteed bonuses. <br> <br>

Justinnib (2025-10-25 09:31)

Don’t miss out on the exclusive https://dojour.us/e/65532-1xbet-promo-code-today-bangladesh-2026-13000-bonus that gives players a huge edge in 2026. Sign up and claim your welcome rewards instantly. <br> <br>

Justinnib (2025-10-25 22:20)

Don’t miss out on the exclusive https://amedia-utvikling2.23video.com/copenhagen-architecture-arial-no3 that gives players a huge edge in 2026. Sign up and claim your welcome rewards instantly. <br> <br>

Thomasfleek (2025-10-27 15:41)

<br> <br>Un Code promo 1xbet 2026 : obtenez un bonus de bienvenue de 100% sur votre premier depot avec un bonus allant jusqu'a 130 €. Placez vos paris en toute plaisir en utilisant simplement les fonds bonus. Une fois inscrit, n'oubliez pas de recharger votre compte. Avec un compte verifie, tous les fonds, bonus inclus, peuvent etre retires. Vous pouvez trouver le code promo 1xbet sur ce lien ? https://www.atrium-patrimoine.com/wp-content/artcls/?code_promo_196.html. <br> <br>

CharlesMix (2025-10-28 11:55)

<br> <br>Code promo pour 1xBet : utilisez-le une fois lors de l'inscription et obtenez un bonus de 100% pour l'inscription jusqu'a 130€. Augmentez le solde de vos fonds simplement en placant des paris avec un wager de cinq fois. Le code bonus est valide tout au long de l'annee 2026. Pour activer ce code, rechargez votre compte a partir de 1€. Decouvrez cette offre exclusive sur ce lien — https://www.atrium-patrimoine.com/wp-content/artcls/?code_promo_196.html. <br> <br> <br>

EltontaX (2025-10-29 10:30)

<br>Как получить приветственный бонус: пошаговая инструкция по регистрации и пополнению счёта, с указанием требований по отыгрышу; в середине параграфа мы указываем ссылку на https://www.apelsin.su/wp-includes/articles/promokod_240.html, чтобы пользователь мог перейти и узнать подробности. Также приводим примеры доступных типов акций. <br>

Jerryswert (2025-11-04 14:04)

To activate your betwinner sign up offer, just head over to https://bet-promo-codes.com/sportsbook-reviews/betwinner-registration/ and follow the guide. It’s an effortless way to start with extra funds.

AlvinEdulp (2025-11-13 20:24)

Go to our brand page — https://www.fitday.com/fitness/forums/diets/41740-code-promo-1xbet.html#post181832

AlvinEdulp (2025-11-16 09:17)

Find the newest updates — https://madguy.ru/

AlvinEdulp (2025-11-18 10:06)

Go to the main company website : http://ruptur.com/

Michaelonelf (2025-11-20 02:47)

<br>1xBet промокод на сегодня Введя на http://www.google.us/url?q=https://bergkompressor.ru/news/artcles/?1xbet_promokod_pri_registracii_bonus_5.html и заберите бонус 100% на депозит до 32 500 рублей, чтобы начать игру с преимуществом. <br>

Michaelonelf (2025-11-22 00:06)

<br>промокоды 1хбет Используйте действующий купон на https://sushikim.ru/image/pgs/1xbet_promokod_55.html и получите бонус 100% на депозит, чтобы начать ставки с преимуществом. <br>

Michaelonelf (2025-11-22 19:01)

<br>1 хбет промокод Используйте промокод на http://www.medtronik.ru/images/pages/1hbet_2021_promokod_pri_registracii.html и получите 100% к первому депозиту для комфортного старта игры. <br>

JamesRaf (2025-11-23 23:57)

<br>где найти промокод для 1xBet Активируйте бонусом при регистрации на http://www.autotat.ru/www/emma/png/?1hbet_bonus_pri_registracii_6500_rubley.html и получите 32 500? + бонус 100%, чтобы начать игру. <br>

JamesRaf (2025-11-25 10:30)

<br>1хБет промокод 2026 Промокод 2025 года на http://arkada-nsk.ru/pgs/obzor_alien_isolation_2014.html даёт 100% к депозиту до 32 500? для всех зарегистрированных пользователей. <br>

Johnnyfrono (2025-11-25 22:37)

<br>L’offre promotionnelle 1xBet offre la possibilite de recevoir un bonus equivalent a 100 % avec une recompense maximale de 100 €/$. Si aucun code n’est saisi, un bonus standard sera applique le bonus initial prevu. Pour en savoir davantage concernant le code promo 1xBet aujourd'hui, toutes les informations sont disponibles ici : https://gazetablic.com/new/?code_promo_208.html <br>

JamesRaf (2025-11-26 22:55)

<br>получить промокод 1xbet Активируйте промокод на https://ukauk.ru/filemanager/articles/1xbet_promokod_pri_registracii_bonus.html и получите бонус 100% на первый депозит при регистрации для комфортной игры. <br>

RobertJutty (2025-11-29 03:51)

<br>промокод на 1xbet при регистрации Получите промокод на https://ural-hifi.ru/fonts/inc/1xbet-promokod.html и заберите бонус 100% на первый депозит при регистрации для максимального старта. <br>

RobertJutty (2025-11-29 04:55)

<br>1хБет фрибет Воспользуйтесь бонус при регистрации на https://neo-magnit.ru/js/pages/?1xbet_promokod_pri_registracii_na_segodnya_besplatno.html и получите 32 500 рублей + бонус 100%, чтобы начать игру. <br>

JeffreyAsync (2025-12-01 01:41)

<br>промо код на 1xbet Вводите действующий купон на https://svadbaved.ru/wp-content/pgs/1xbet-promokod-pri-registracii-besplatno.html и получите бонус до 32 500 рублей, чтобы получить стартовый капитал для игры. <br>

LloydRem (2025-12-02 12:08)

Go to the official company site > https://www.apelsin.su/

LloydRem (2025-12-02 14:56)

Официальный сайт компании 1xBet — https://kanctanta.ru/

Williamsam (2025-12-03 10:06)

<br>промокоды 1хбет Вводите действующий купон на https://www.ceacuautla.edu.mx/profile/bergkoma914047/profile и получите бонус до 32 500 рублей, чтобы начать ставки с преимуществом. <br>

Jameszebra (2025-12-07 21:10)

<br>Le bonus du code promo 1xBet est disponible jusqu’a 130 $ accessible aux joueurs ayant cree un compte recemment. Ces fonds sont credites apres un depot minimum et l’utilisation correcte du code. Le systeme refuse le bonus si le code n’est pas exact. Pour plus de details sur le code promo 1xBet gratuit, consultez ce lien : https://justinekeptcalmandwentvegan.com/wp-content/pages/code_promo_76.html <br>

BrianStime (2025-12-09 19:45)

Go to our website now — https://consolit.ru/

MarioLor (2025-12-10 16:24)

<br>действующий промокод на 1хбет Используйте действующий купон на http://www.medtronik.ru/images/pages/bonus_kod_na_1xbet_pri_registracii_6500_rubley.html#:~:text=Промокод%201хбет%20при%20регистрации%20только%20один%20это:%201XMAXVIP.Увеличение%20бонуса%20до%20?32500%20с%20промо-кодом. и активируйте бонус 100% на депозит, чтобы получить стартовый капитал для игры. <br>

Jameszebra (2025-12-12 23:07)

<br>Le bonus du code promo 1xbet peut atteindre 100 €/$ lors du depot initial. Meme sans code, le bonus de 100 % est credite automatiquement. Vous pouvez en savoir plus sur le code promo 1xBet inscription ici : http://www.medtronik.ru/images/pages/1hbet_2021_promokod_pri_registracii.html <br>

DennisVoini (2025-12-13 10:21)

<br>1xBet промокод на сегодня Активируйте на https://www.res-rent.ru/img/pages/betvinner_promokod___bonus_25_000_rubley.html и получите бонус 100% на депозит до 32 500 рублей, чтобы получить максимальный стартовый бонус. <br>

BrianStime (2025-12-14 04:08)

Discover what’s new here — https://www.ikonu.ru/

OscarHib (2025-12-15 23:34)

Let me share the ugly truth: nearly all septic companies just maintain tanks. They're like band-aid salesmen at a disaster convention. But Septic Solutions? These guys are unique. It all started back in the beginning of the 2000s when Art and his family—just kids hardly tall enough to carry a shovel—assisted install their family's septic system alongside a weathered pro. Picture this: three youngsters knee-deep in Pennsylvania clay, learning how soil absorption affects drainage while their peers played Xbox. "We didn't just dig holes," Art shared with me last winter, warm coffee cup in hand. "We discovered how ground whispers secrets. A patch of wetland vegetation here? That's Mother Nature shouting 'high water table.'" <br>https://squareblogs.net/cormanzjnz/how-weather-extremes-test-hvac-in-marysville-wa-reliability

DennisLug (2025-12-18 11:00)

Официальный сайт компании > http://www.autotat.ru/

OscarHib (2025-12-18 15:58)

Let me share the ugly truth: nearly all septic companies just pump tanks. They are like temporary salesmen at a demolition convention. But Septic Solutions? They're special. It all started back in the beginning of the 2000s when Art and his brothers—just kids scarcely tall enough to shoulder a shovel—helped install their family's septic system alongside a grizzled pro. Picture this: three kids buried in Pennsylvania clay, understanding how soil absorption affects drainage while their buddies played Xbox. "We never just dig ditches," Art told me last winter, hot coffee cup in hand. "We understood how ground whispers secrets. A patch of cattails here? That's Mother Nature yelling 'high water table.'" <br>https://raindrop.io/goldetztui/bookmarks-64277125

OscarHib (2025-12-19 13:09)

This is the dirty truth: the majority of septic companies just service tanks. They are like band-aid salesmen at a disaster convention. But Septic Solutions? They are special. It all began back in the early 2000s when Art and his brothers—just kids hardly tall enough to lift a shovel—aided install their family's septic system alongside a experienced pro. Visualize this: three youngsters waist-deep in Pennsylvania clay, discovering how soil absorption affects drainage while their buddies played Xbox. "We never just dig ditches," Art explained to me last winter, hot coffee cup in hand. "We discovered how earth whispers mysteries. A patch of marsh plants here? That's Mother Nature screaming 'high water table.'" <br>https://kameronwvbv600.tearosediner.net/the-hidden-costs-of-ignoring-furnace-air-filter-maintenance

OscarHib (2025-12-19 23:53)

Let me share the dirty truth: nearly all septic companies just pump tanks. They are like quick-fix salesmen at a chainsaw convention. But Septic Solutions? They're special. It all started back in the early 2000s when Art and his brothers—just kids scarcely tall enough to lift a shovel—helped install their family's septic system alongside a grizzled pro. Imagine this: three kids knee-deep in Pennsylvania clay, learning how soil permeability affects drainage while their friends played Xbox. "We didn't just dig ditches," Art explained to me last winter, hot coffee cup in hand. "We discovered how soil whispers truths. A patch of wetland vegetation here? That's Mother Nature yelling 'high water table.'" <br>https://johnathandyhx509.theglensecret.com/are-smart-thermostats-worth-the-investment-for-homeowners

DennisLug (2025-12-22 08:28)

Перейди на сайт компании > https://tako-text.ru/

DennisLug (2025-12-22 09:53)

Наш сайт — только официальная информация : https://veber-geo.ru/

Williammal (2025-12-29 12:32)

See the latest update here : http://hammill.ru/

Brentdat (2025-12-29 15:46)

Allow me to explain something nearly all septic companies refuse to: there are two types of people in this reality. Those who believe septic systems are merely "subterranean tanks for waste," and those that have had raw sewage erupting into their backyard at 2 AM. I understood this reality the hard way in 2005—standing in mud, trembling in a Washington downpour, as my family and I assisted a grizzled installer restore our family's failed system. I was fourteen. My hands were raw. My pants were wrecked. But that night, something crystallized: This is not just digging. It's people's lives that we're preserving. <br>This is the ugly truth: most septic companies just maintain tanks. They are like band-aid salesmen at a chainsaw convention. But Septic Solutions? They're unique. It all began back in the beginning of the 2000s when Art and his siblings—just kids scarcely tall enough to carry a shovel—assisted install their family's septic system alongside a weathered pro. Visualize this: three youngsters knee-deep in Pennsylvania clay, learning how soil porosity affects drainage while their friends played Xbox. "We never just dig ditches," Art explained to me last winter, hot coffee cup in hand. "We discovered how soil whispers secrets. A patch of cattails here? That's Mother Nature screaming 'high water table.'" <br> <br> <br>https://www.pinterest.com/pin/624241198844572503/ <br>

Howardmen (2025-12-30 02:41)

Let me tell you something nearly all septic companies refuse to: there are two types of people in this life. Those who believe septic systems are merely "buried containers for waste," and those that have had raw sewage erupting into their yard at the dead of night. I discovered this distinction the tough way in 2005—standing in sludge, freezing in a Washington downpour, as my family and I assisted a veteran installer restore our family's failed system. I was a teenager. My hands ached. My pants were wrecked. But that evening, something changed: This isn't just dirt work. It's families' lives we're safeguarding. <br>Here's the ugly truth: most septic companies just pump tanks. They're like quick-fix salesmen at a chainsaw convention. But Septic Solutions? These guys are different. It all originated back in the early 2000s when Art and his family—just kids barely tall enough to lift a shovel—aided install their family's septic system alongside a grizzled pro. Imagine this: three youngsters knee-deep in Pennsylvania clay, understanding how soil porosity affects drainage while their friends played Xbox. "We did not just dig ditches," Art explained to me last winter, hot coffee cup in hand. "We learned how earth whispers mysteries. A patch of cattails here? That's Mother Nature shouting 'high water table.'" <br> <br>https://fliphtml5.com/homepage/septicsolutionsllc/septic-solutions-llc/ <br>

Brentdat (2025-12-30 09:54)

I need to explain something most septic companies refuse to: there are two types of people in this reality. Those who assume septic systems are simply "buried containers for waste," and those who have had raw sewage gurgling into their yard at the dead of night. I learned this reality the hard way in 2005—standing in muck, freezing in a Washington rainstorm, as my siblings and I aided a veteran installer fix our family's collapsed system. I was 14. My hands were raw. My jeans were ruined. But that evening, something changed: This is not just manual labor. It's families' lives we're protecting. <br>This is the dirty truth: nearly all septic companies just service tanks. They're like temporary salesmen at a disaster convention. But Septic Solutions? These guys are different. It all originated back in the beginning of the 2000s when Art and his family—just kids hardly tall enough to carry a shovel—assisted install their family's septic system alongside a experienced pro. Imagine this: three youngsters knee-deep in Pennsylvania clay, learning how soil porosity affects drainage while their peers played Xbox. "We didn't just dig ditches," Art explained to me last winter, hot coffee cup in hand. "We understood how soil whispers truths. A patch of cattails here? That's Mother Nature screaming 'high water table.'" <br> <br> <br>https://penzu.com/p/b87e1a5840e214db <br>

Brentdat (2025-12-31 19:37)

Let me explain something nearly all septic companies refuse to: there are two categories of people in this world. Those who think septic systems are simply "underground boxes for waste," and those that have had raw sewage erupting into their yard at the dead of night. I understood this distinction the tough way in 2005—standing in mud, shivering in a Washington rainstorm, as my brothers and I assisted a veteran installer repair our family's failed system. I was fourteen. My hands blistered. My pants were ruined. But that moment, something changed: This is not just digging. It's folks' lives we are safeguarding. <br>Here's the dirty truth: most septic companies just pump tanks. They're like band-aid salesmen at a chainsaw convention. But Septic Solutions? They are different. It all originated back in the early 2000s when Art and his family—just kids barely tall enough to shoulder a shovel—helped install their family's septic system alongside a experienced pro. Visualize this: three youngsters knee-deep in Pennsylvania clay, learning how soil porosity affects drainage while their buddies played Xbox. "We never just dig trenches," Art shared with me last winter, warm coffee cup in hand. "We learned how earth whispers truths. A patch of cattails here? That's Mother Nature shouting 'high water table.'" <br> <br> <br>https://penzu.com/p/b87e1a5840e214db <br>

ClintonNom (2026-01-01 01:22)

Visit the company site : https://www.ikonu.ru/

Howardmen (2026-01-01 04:58)

I need to share with you something the majority of septic companies will not: there are two kinds of people in this world. Those who assume septic systems are just "underground boxes for waste," and those who have had raw sewage bubbling into their backyard at midnight. I learned this distinction the difficult way in 2005—knee-deep in sludge, trembling in a Washington rainstorm, as my family and I assisted a veteran installer fix our family's broken system. I was 14. My hands were raw. My pants were destroyed. But that moment, something clicked: This isn't just dirt work. It's people's lives we're preserving. <br>Let me share the dirty truth: nearly all septic companies just maintain tanks. They act like quick-fix salesmen at a chainsaw convention. But Septic Solutions? These guys are unique. It all originated back in the beginning of the 2000s when Art and his family—just kids hardly tall enough to lift a shovel—aided install their family's septic system alongside a grizzled pro. Imagine this: three kids waist-deep in Pennsylvania clay, understanding how soil permeability affects drainage while their peers played Xbox. "We didn't just dig trenches," Art shared with me last winter, hot coffee cup in hand. "We understood how ground whispers truths. A patch of cattails here? That's Mother Nature shouting 'high water table.'" <br> <br>https://www.behance.net/septicsolutio <br>

Brentdat (2026-01-01 12:41)

I need to explain something nearly all septic companies refuse to: there are two types of people in this reality. Those who think septic systems are just "subterranean tanks for waste," and those that have had raw sewage gurgling into their yard at the dead of night. I learned this distinction the tough way in 2005—knee-deep in muck, trembling in a Washington rainstorm, as my family and I assisted a weathered installer fix our family's broken system. I was fourteen. My hands blistered. My jeans were wrecked. But that evening, something clicked: This ain't just manual labor. It's folks' lives that we're protecting. <br>Here's the harsh truth: the majority of septic companies just service tanks. They are like band-aid salesmen at a demolition convention. But Septic Solutions? They are special. It all started back in the beginning of the 2000s when Art and his family—just kids barely tall enough to lift a shovel—assisted install their family's septic system alongside a experienced pro. Imagine this: three youngsters waist-deep in Pennsylvania clay, understanding how soil absorption affects drainage while their buddies played Xbox. "We didn't just dig holes," Art explained to me last winter, steaming coffee cup in hand. "We learned how earth whispers mysteries. A patch of marsh plants here? That's Mother Nature yelling 'high water table.'" <br> <br> <br>https://www.designspiration.com/voadilssxt/ <br>