AppSuite:OX Abuse Shield
OX Abuse Shield
OX Abuse Shield is a standalone server, that works with any authentication system to prevent abuse of that system. It is pre-integrated as part of both Dovecot Pro and OX App Suite as a component to protect against login/authentication abuse.
Abuse Shield runs on a cluster of servers, and integrates with authenication systems to detect abuse, brute force attacks and also to enforce common authentication/authorization policies across the platform. Authentication systems which are not pre-integrated can use the simple REST API to integrate with OX Abuse Shield.
Support Commitment
Open-Xchange encourages administrators to regularly update to the latest available release. To ensure a stable and up to date environment please note the different versions supported. An overview of the latest supported Major, Minor and Public Patch Releases can be found in the OXpedia at: https://oxpedia.org/wiki/index.php?title=OXAbuseShield:Version_Support_Commitment
Key Features
Features of OX Abuse Shield include:
- Replicated/clustered architecture – Login reports are shared between all the servers in a cluster so there is a single view of abuse
- Scriptable Policy Language – Using the Lua language, the functionality of the daemon can be extended to record and protect against a large variety of abusive behavior, as well as implement specific customer policies.
- DNS Lookup Feature – For looking up IPs or domains in blacklists
- GepIP Lookup Feature – GeoIP lookups can be made, and incorporated into policy decisions.
- Ratelimiting and Tarpitting – Extremely flexible, these can be enabled and enforced based on IP address, login name, geoip location, time windows, etc.
- Flexible In-Memory Statistics Database – A versatile and extensible in-memory database is used to store statistics information about abuse over time periods from a few minutes to many hours.
- Integration with Customer Authentication/Authorization Systems – Customers can use the open HTTP REST API to benefit from the protection of the anti-abuse daemon in their own authentication/authorization systems.
- Admin Console – To retrieve statistics and query server state
- Persistent Replicated Blacklist – Configurable via a REST API or the Lua policy engine, supports auto-expiry of entries, replication between all cluster nodes, and optionally uses a Redis DB for persistence.
- Webhooks – Integrate Abuse Shield with other systems using webhooks to send events.
- Integration with ELK - Storage of report data in ELK stack (Elasticsearch, Logstash, Kibana).
- Suspicious Login Detection - Using the long-term report data stored in Elasticsearch to detect anomalous logins to send alerts about suspcious logins to administrators or end-users.
- Scheduled Reports on abusive IPs and compromised logins - Sent as webhooks to abuse teams.
In General
The goal of OX Abuse Shield is to detect brute forcing of passwords across many servers, services and instances, as well as enforce policy for authentication and authorization. In order to support the real world, brute force detection policy can be tailored to deal with "bulk, but legitimate" users of your service, as well as botnet-wide slow-scans of passwords.
Here is how it works:
- Report successful logins via JSON http-api
- Report unsuccessful logins via JSON http-api
- Query if a login should be allowed to proceed, should be delayed, or ignored via JSON http-api
Various other API functions are available, please see https://documentation.open-xchange.com/components/weakforce-core/2.0.0/ for full API documentation.
OX App Suite and Dovecot's POP/IMAP server are pre-integrated with OX Abuse Shield. For more information on how to configure them to work with Abuse Shield, see https://documentation.open-xchange.com/7.10.0/middleware/components/weakforced.html#configuration and http://wiki2.dovecot.org/Authentication/Policy.
However it is also aimed to receive message from other services over a simple REST API, including:
- Other IMAP/POP servers
- Other Webmail logins
- FTP logins
- Authenticated SMTP
- Self-service logins
- Password recovery services
By gathering failed and successful login attempts from as many services as possible, brute forcing attacks can be detected and prevented more effectively.
The service runs as a daemon (called wforce), and can be clustered in a way that report information is shared between all members of the cluster.
OX Abuse Shield Server-side Installation and Configuration
This chapter describes how the backend components of OX Abuse Shield are installed and configured on the server.
OX Abuse Shield is made up of three components, some of which are optional depending on your requirements:
- (Mandatory) wforce - A network daemon, which stores the statistics needed for Abuse Shield to operate.
- (Optional) wforce-policy - A set of example policies for use with the wforce daemon
- (Optional) wforce-trackalert - A network daemon which takes reports from wforce and looks for suspicious logins.
- (Optional) Open-Xchange AppSuite Plugin - A connector from AppSuite to the wforce daemon.
- (Optional) Dovecot Authentication Policy - A core feature of Dovecot from 2.2.25 onwards, which connects Dovecot to the wforce daemon (see http://wiki2.dovecot.org/Authentication/Policy). No additional packages are needed.
Mandatory Packages
The following package is mandatory for OX Abuse Shield to function:
- wforce - Wforce daemon package
Installation on the server varies depending on the underlying distribution, details are as follows.
Debian GNU/Linux 9.0 (Stretch)
Add the Dovecot GPG key to your installation:
$ wget -O- https://software.open-xchange.com/products/abuseshield/abuseshield.pub | apt-key add -
Add the following repositories to your apt configuration (e.g. /etc/apt/sources.list.d/wforce.list):
deb https://USERNAME:PASSWORD@software.open-xchange.com/products/abuseshield/stable/abuseshield-wforce/DebianStretch ./
and run
$ apt-get update $ apt-get install wforce
Debian GNU/Linux 10.0 (Buster)
Add the Dovecot GPG key to your installation:
$ wget -O- https://software.open-xchange.com/products/abuseshield/abuseshield.pub | apt-key add -
Add the following repositories to your apt configuration (e.g. /etc/apt/sources.list.d/wforce.list):
deb https://USERNAME:PASSWORD@software.open-xchange.com/products/abuseshield/stable/abuseshield-wforce/DebianBuster ./
and run
$ apt-get update $ apt-get install wforce
Redhat Enterprise Linux 7
If not already done, add the following repositories to your yum configuration (e.g. /etc/yum.repos.d/wforce.repo):
[abuseshield-stable] name=RHEL $releasever - $basearch - OX Abuse Shield baseurl=https://USERNAME:PASSWORD@software.open-xchange.com/products/abuseshield/stable/abuseshield-wforce/CentOS7 gpgkey=https://software.open-xchange.com/products/abuseshield/abuseshield.pub gpgcheck=1 enabled=1
There are slightly different procedures depending on whether you are running Centos 7 or RHEL7.
For Centos 7:
% yum install epel-release % yum update % yum install wforce
For RHEL7:
First install the EPEL repositories as described here: https://fedoraproject.org/wiki/EPEL#How_can_I_use_these_extra_packages.3F
and then run:
% yum update % yum install wforce
Amazon Linux 2
- First install EPEL repositories
- If not already done, add the following repositories to your yum configuration (e.g. /etc/yum.repos.d/wforce.repo):
[abuseshield-stable] name=Amazon $releasever - $basearch - OX Abuse Shield baseurl=http://<USERNAME>:<PASSWORD>@http://software.open-xchange.com/products/abuseshield/stable/abuseshield-wforce/AmazonLinux2 gpgkey=https://software.open-xchange.com/products/abuseshield/abuseshield.pub gpgcheck=1 enabled=1
Systemd Configuration
The wforce package installs with systemd support. To enable wforce under systemd, run:
$ systemctl enable wforce $ systemctl start wforce
To check that wforce is running correctly under systemd, run:
$ systemctl status wforce
* wforce.service - Weakforce Anti-Abuse Daemon
Loaded: loaded (/usr/lib/systemd/system/wforce.service; enabled)
Active: active (running) since Wed 2016-08-17 05:27:36 EDT; 1h 40min ago
Docs: man:wforce(1)
Main PID: 1500 (wforce)
CGroup: /system.slice/wforce.service
└─1500 /usr/bin/wforce -s
Aug 17 05:27:36 debian-devserv wforce[1500]: Read configuration from '/etc/wforce.conf'
Aug 17 05:27:36 debian-devserv wforce[1500]: ACL allowing queries from: 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/
Aug 17 05:27:36 debian-devserv wforce[1500]: Listening on 0.0.0.0:4000
Aug 17 05:27:36 debian-devserv wforce[1500]: Starting stats reporting thread
Aug 17 05:27:36 debian-devserv wforce[1500]: Webserver launched on 0.0.0.0:8084
Aug 17 05:27:36 debian-devserv wforce[1500]: Accepting control connections on 0.0.0.0:4004
To view the systemd journal logs for wforce, run:
$ journalctl _SYSTEMD_UNIT=wforce.service -- Logs begin at Wed 2016-08-17 05:21:41 EDT, end at Wed 2016-08-17 07:12:36 EDT. -- Aug 17 05:27:36 debian-devserv wforce[1500]: Read configuration from '/etc/wforce.conf' 17 05:27:36 debian-devserv wforce[1500]: ACL allowing queries from: 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/ Aug 17 05:27:36 debian-devserv wforce[1500]: Listening on 0.0.0.0:4000 Aug 17 05:27:36 debian-devserv wforce[1500]: Starting stats reporting thread Aug 17 05:27:36 debian-devserv wforce[1500]: Webserver launched on 0.0.0.0:8084 Aug 17 05:27:36 debian-devserv wforce[1500]: Accepting control connections on 0.0.0.0:4004 ...
You can also view the wforce logs in the normal syslog files in /var/log.
Optional Packages
The following packages are optional for Abuse Shield to function:
- wforce-policy - Wforce example policy package (available from v1.4.0)
- wforce-trackalert - Wforce trackalert daemon (available from v2.0.0)
- open-xchange-weakforced - Open-Xchange App Suite plugin
- wforce-replfwd - Wforce forward replication messages daemon (available from v2.4.0)
Installation on the server varies depending on the underlying distribution, details are as follows.
Optional: wforce-policy
Please use the same repository locations as for Mandatory Packages
Debian GNU/Linux 9.0
Optionally run:
$ apt-get install wforce-policy
Debian GNU/Linux 10.0
Optionally run:
$ apt-get install wforce-policy
Redhat Enterprise Linux 7 / CentOS 7 / Amazon 2
Optionally run:
$ yum install wforce-policy
Optional: wforce-trackalert
Please use the same repository locations as for Mandatory Packages
Debian GNU/Linux 9.0
Optionally run:
$ apt-get install wforce-trackalert
Debian GNU/Linux 10.0
Optionally run:
$ apt-get install wforce-trackalert
Redhat Enterprise Linux 7 / CentOS 7 / Amazon 2
Optionally run:
$ yum install wforce-trackalert
Optional: wforce-replfwd
Please use the same repository locations as for Mandatory Packages
Debian GNU/Linux 9.0
Optionally run:
$ apt-get install wforce-replfwd
Debian GNU/Linux 10.0
Optionally run:
$ apt-get install wforce-replfwd
Redhat Enterprise Linux 7 / CentOS 7 / Amazon 2
Optionally run:
$ yum install wforce-replfwd
Systemd Configuration
The wforce-trackalert package installs with systemd support. To enable trackalert under systemd, run:
$ systemctl enable trackalert $ systemctl start trackalert
To check that trackalert is running correctly under systemd, run:
- systemctl status trackalert -l
trackalert.service - Trackalert Anti-Abuse Daemon
Loaded: loaded (/usr/lib/systemd/system/trackalert.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2018-11-14 09:28:31 UTC; 1s ago
Docs: man:trackalert(1)
Main PID: 218 (trackalert)
CGroup: /docker/ff12caab9337ec6035f3ecaafacfcf58fa933d315b2e56edeab610b68df3e567/system.slice/trackalert.service
└─218 /usr/bin/trackalert -s
Nov 14 09:28:31 ff12caab9337 systemd[1]: Starting Trackalert Anti-Abuse Daemon... Nov 14 09:28:31 ff12caab9337 trackalert[218]: Read configuration from '/etc/wforce/trackalert.conf' Nov 14 09:28:31 ff12caab9337 trackalert[218]: Read configuration from '/etc/wforce/trackalert.conf' Nov 14 09:28:31 ff12caab9337 trackalert[218]: ACL allowing queries from: 127.0.0.0/8, 192.168.0.0/16, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10 Nov 14 09:28:31 ff12caab9337 trackalert[218]: Starting stats reporting thread Nov 14 09:28:31 ff12caab9337 systemd[1]: Started Trackalert Anti-Abuse Daemon. Nov 14 09:28:31 ff12caab9337 trackalert[218]: WforceWebserver launched on 0.0.0.0:8085 Nov 14 09:28:31 ff12caab9337 trackalert[218]: Accepting control connections on 127.0.0.1:4005
To view the systemd journal logs for wforce, run:
- journalctl _SYSTEMD_UNIT=trackalert.service
-- Logs begin at Wed 2018-11-14 09:22:57 UTC, end at Wed 2018-11-14 09:29:00 UTC. -- Nov 14 09:22:58 ff12caab9337 trackalert[50]: Read configuration from '/etc/wforce/trackalert.conf' Nov 14 09:22:58 ff12caab9337 trackalert[50]: Read configuration from '/etc/wforce/trackalert.conf' Nov 14 09:22:58 ff12caab9337 trackalert[50]: ACL allowing queries from: 127.0.0.0/8, 192.168.0.0/16, 10 .0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 172.16.0.0/12, ::1/128, fc00::/7, fe80::/10 Nov 14 09:22:58 ff12caab9337 trackalert[50]: Starting stats reporting thread Nov 14 09:22:58 ff12caab9337 trackalert[50]: Accepting control connections on 127.0.0.1:4005 Nov 14 09:22:58 ff12caab9337 trackalert[50]: WforceWebserver launched on 0.0.0.0:8085 Nov 14 09:23:00 ff12caab9337 trackalert[50]: Ran background thread Nov 14 09:24:00 ff12caab9337 trackalert[50]: Ran background thread
...
You can also view the wforce logs in the normal syslog files in /var/log.
Optional: open-xchange-weakforced
Redhat Enterprise Linux 6 or CentOS 6
If not already done, add the following repositories to your Open-Xchange yum configuration:
[open-xchange-backend-weakforced] name=Open-Xchange-backend-weakforced baseurl=https://software.open-xchange.com/products/weakforced/1.0.2/backend-weakforced/RHEL6/ gpgkey=https://software.open-xchange.com/oxbuildkey.pub enabled=1 gpgcheck=1 metadata_expire=0m
[open-xchange-backend-weakforced-updates] name=Open-Xchange-backend-weakforced-updates baseurl=https://LDBUSER:LDBPASSWORD@software.open-xchange.com/products/weakforced/1.0.2/backend-weakforced/updates/RHEL6/ gpgkey=https://software.open-xchange.com/oxbuildkey.pub enabled=1 gpgcheck=1 metadata_expire=0m
and run
$ yum update $ yum install open-xchange-weakforced
Redhat Enterprise Linux 7 or CentOS 7
If not already done, add the following repositories to your Open-Xchange yum configuration:
[open-xchange-backend-weakforced] name=Open-Xchange-backend-weakforced baseurl=https://software.open-xchange.com/products/weakforced/stable/backend-weakforced/RHEL7/ gpgkey=https://software.open-xchange.com/oxbuildkey.pub enabled=1 gpgcheck=1 metadata_expire=0m
[open-xchange-backend-weakforced-updates] name=Open-Xchange-backend-weakforced-updates baseurl=https://LDBUSER:LDBPASSWORD@software.open-xchange.com/products/weakforced/stable/backend-weakforced/updates/RHEL7/ gpgkey=https://software.open-xchange.com/oxbuildkey.pub enabled=1 gpgcheck=1 metadata_expire=0m
and run
$ yum update $ yum install open-xchange-weakforced $ yum install wforce-policy
Debian GNU/Linux 9.0
If not already done, add the following repositories to your Open-Xchange apt configuration:
deb https://software.open-xchange.com/products/weakforced/stable/backend-weakforced/DebianStretch /
deb https://LDBUSER:LDBPASSWORD@software.open-xchange.com/products/weakforced/stable/backend-weakforced/updates/DebianStretch /
and run
$ apt-get update $ apt-get install open-xchange-weakforced
Debian GNU/Linux 10.0
If not already done, add the following repositories to your Open-Xchange apt configuration:
deb https://software.open-xchange.com/products/weakforced/stable/backend-weakforced/DebianBuster /
deb https://LDBUSER:LDBPASSWORD@software.open-xchange.com/products/weakforced/stable/backend-weakforced/updates/DebianBuster /
and run
$ apt-get update $ apt-get install open-xchange-weakforced
SUSE Linux Enterprise Server 12
Add the package repository using zypper if not already present:
$ zypper ar https://software.open-xchange.com/products/weakforced/1.0.2/backend-weakforced/SLE_12 backend-weakforced
$ zypper ar https://LDBUSER:LDBPASSWORD@software.open-xchange.com/products/weakforced/1.0.2/backend-weakforced/updates/SLE_12 backend-weakforced-updates
and run
$ zypper ref $ zypper in open-xchange-weakforced
Optional: Configuration of OX App Suite
OX App Suite and Dovecot's POP/IMAP server are pre-integrated with OX Abuse Shield. For more information on how to configure them to work with Abuse Shield, see https://documentation.open-xchange.com/7.8.4/middleware/components/weakforced.html#configuration and http://wiki2.dovecot.org/Authentication/Policy.
Policies
Configuration and control of policy is almost entirely through a configuration file which is based on the Lua scripting language. There is a sensible default configuration in wforce.conf, and extensive support for crafting your own policies using the Lua scripting language.
Note that although there is a single Lua configuration file, the report and allow functions run in different lua states from the rest of the configuration, thus you cannot share state.
The following sample is from the default wforce.conf file:
-- set up the things we want to track
field_map = {}
-- use hyperloglog to track cardinality of (failed) password attempts
field_map["diffFailedPasswords"] = "hll"
-- track those things over 6x10 minute windows
newStringStatsDB("OneHourDB", 600, 6, field_map)
-- this function counts interesting things when "report" is invoked
function twreport(lt)
sdb = getStringStatsDB("OneHourDB")
if (not lt.success)
then
sdb:twAdd(lt.remote, "diffFailedPasswords", lt.pwhash)
addrlogin = lt.remote:tostring() .. lt.login
sdb:twAdd(addrlogin, "diffFailedPasswords", lt.pwhash)
end
end
function allow(lt)
sdb = getStringStatsDB("OneHourDB")
if(sdb:twGet(lt.remote, "diffFailedPasswords") > 50)
then
return -1, "", "", {} -- BLOCK!
end
// concatenate the IP address and login string
addrlogin = lt.remote:tostring() .. lt.login
if(sdb:twGet(addrlogin, "diffFailedPasswords") > 3)
then
return 3, "tarpitted", "diffFailedPasswords", {} -- must wait for 3 seconds
end
return 0, "", "", {} -- OK!
end
The main way to track statistics is by using a sliding-time-window based, in-memory stats database, that enables extremely efficient storage and retrieval of three types of statistics:
- Integer - Counting the number of times relevant events happened
- HyperLogLog - Counting the uniqueness or cardinality of a data set
- CountMin - Distinct count of multiple items in a data set
Country-level IPv4 and v6 IP lookups are available, as are a variety of DNS lookup functions, including querying RBLs.
Many more metrics are available to base decisions on, and are documented in the wforce.conf man page, available on any system with OX Abuse Shield installed. It also ships with wforce.conf.example, which gives many examples of how to configure the service and policy.
wforce-policy Package
If you have installed the optional wforce-policy package, a full-featured example policy is available to use, either as-is, or as the basis for customised site-specific policies. This package contains policies for both wforce and trackalert.
The example policies are located in /etc/wforce/example-config.
In order to use them, do the following:
$ cp -r /etc/wforce/example-config/policy /etc/wforce $ cp -r /etc/wforce/example-config/config /etc/wforce $ cp -r /etc/wforce/example-config/es_queries /etc/wforce $ cp -r /etc/wforce/example-config/userdb_plugins /etc/wforce $ cp -r /etc/wforce/example-config/alert_plugins /etc/wforce $ cp /etc/wforce/wforce.conf /etc/wforce/wforce.conf.bak $ cp /etc/wforce/example-config/wforce.conf /etc/wforce $ cp /etc/wforce/trackalert.conf /etc/wforce/trackalert.conf.bak $ cp /etc/wforce/example-config/trackalert.conf /etc/wforce
Before restarting wforce and trackalert daemons, you must then replace the 'webserver' and 'setKey' directives in the new config files with the values from the .bak files, otherwise the daemons will not function correctly.
Then do the following:
$ systemctl restart wforce $ systemctl restart trackalert
You may need to modify the example configuration, in which case consult the following man page:
$ man wforce_policy
Simple Testing
To report (if you configured with 'webserver("127.0.0.1:8084", "super")'):
$ for a in {1..101}
do
curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234'$a'", "success":"false"}' \
http://127.0.0.1:8084/?command=report -u wforce:super
done
This reports 101 failed logins for one user, but with different password hashes.
Now to look up if we're still allowed in:
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234"}' \
http://127.0.0.1:8084/?command=allow -u wforce:super
{"status": -1, "msg": "diffFailedPasswords"}
It appears we are not!
You can also provide additional information for use by Abuse Shield using the optional "attrs" object. An example:
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1",
"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":"val2"}}' \
http://127.0.0.1:8084/?command=allow -u wforce:super
{"status": 0, "msg": ""}
An example using the optional attrs object using multi-valued attributes:
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1",
"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":["val2","val3"]}}' \
http://127.0.0.1:8084/?command=allow -u wforce:super
{"status": 0, "msg": ""}
There is also a command to reset the stats for a given login and/or IP Address, using the 'reset' command, the logic for which is also implemented in Lua. The default policy for reset is as follows:
function reset(type, login, ip)
sdb = getStringStatsDB("OneHourDB")
if (string.find(type, "ip"))
then
sdb:twReset(ip)
end
if (string.find(type, "login"))
then
sdb:twReset(login)
end
if (string.find(type, "ip") and string.find(type, "login"))
then
iplogin = ip:tostring() .. login
sdb:twReset(iplogin)
end
return true
end
To test it out, try the following to reset the login 'ahu':
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu"}'\
http://127.0.0.1:8084/?command=reset -u wforce:super
{"status": "ok"}
You can reset IP addresses also:
$ curl -X POST -H "Content-Type: application/json" --data '{"ip":"128.243.21.16"}'\
http://127.0.0.1:8084/?command=reset -u wforce:super
{"status": "ok"}
Or both in the same command (this helps if you are tracking stats using compound keys combining both IP address and login):
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "ip":"FE80::0202:B3FF:FE1E:8329"}'\
http://127.0.0.1:8084/?command=reset -u wforce:super
{"status": "ok"}
You can retrieve all the known stats for a given IP or login with the 'getDBStats' command:
$ curl -X POST -H "Content-Type: application/json" --data '{"ip":"127.0.0.1"}'\
http://127.0.0.1:8084/?command=getDBStats -u wforce:super
{"blacklisted": false, "ip": "127.0.0.1", "stats": {"OneHourDB": {"diffFailedPasswords": 1}}}
There is also a "ping" command, to check the server is up and answering requests:
$ curl -X GET http://127.0.0.1:8084/?command=ping -u wforce:super {"status": "ok"}
Console
Available over TCP/IP, like this:
setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=")
controlSocket("0.0.0.0:4004")
Although wforce normally runs under systemd, you can launch wforce manually (`wforce -C "config file"`), or as a daemon (`wforce --daemon`). To connect using the console, run `wforce -c "config file"`. Comes with autocomplete and command history. If you put an actual IP address in place of 0.0.0.0, you can use the same config to listen and connect remotely.
To get some stats, try:
> stats() 40 reports, 8 allow-queries, 40 entries in database
Clustering wforce
For high-availability or performance reasons it may be desirable to run multiple instances of wforce. We need to add a loadbalancer to distribute load to the individual wforce instances, and we need to have some "intra-cluster communication" between the wforce instances to present a unified view of status.
Load balancing
We have gathered HAproxy configuration information on our HAproxy article.
Siblings
To present a unified view of status however, these instances then need to share data. To do so, wforce implements a simple knowledge-sharing system. To do this, wforce broadcasts incremental changes to the underlying state databases, namely the stats dbs and the blacklist.
The sibling list is parsed such that we don't broadcast messages to ourselves accidentally, and can thus be identical across all servers.
Even if you configure siblings, stats db data is not replicated by default. To do this, use the "twEnableReplication()" command on each stats db for which you wish to enable replication. Blacklist information is automatically replicated if you have configured siblings.
To define siblings, use:
setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=")
addSibling("192.168.1.79")
addSibling("192.168.1.30")
addSibling("192.168.1.54")
siblingListener("0.0.0.0")
The first line sets the authentication and encryption key for our sibling communications. To make your own key (recommended), run `makeKey()` on the console and paste the output in all your configuration files.
This last line configures that we also listen to our other siblings (which is nice). The default port is 4001, the protocol is UDP.
To view sibling stats:
> siblings() Address Sucesses Failures Note 192.168.1.79:4001 18 7 192.168.1.30:4001 25 0 192.168.1.54:4001 0 0 Self
With this setup, several wforce instances are all kept in sync, and can be load balanced behind for example haproxy, which incidentally can also offer SSL.
Replication is performed over UDP by default, but it can be configured to use TCP instead; use the following:
addSibling("192.168.1.54:4001:tcp")
Note that the port must be specified when using TCP.
Integrating with ELK
To integrate with the ELK stack, the correct config must be enabled for Logstash and Elasticsearch. The wforce package provides a sample logstash configuration file, as well as a sample Elasticsearch template which is automatically loaded by Logstash, i.e.:
/usr/share/doc/wforce-VERSION/logstash.conf /usr/share/doc/wforce-VERSION/wforce_template.json
These configuration files are compatible only with Elasticsearch 6 and above.