Back to home page

Enduro/X

 
 

    


0001 # endurox
0002 Enduro/X is Open Source Middleware Platform for Distributed Transaction 
0003 Processing
0004 
0005 It is modern, microservices based middleware for writing distributed, open
0006 systems (program consists of several executables) based applications. Thus 
0007 by using Enduro/X programmers do not have to worry about threads and concurrency 
0008 anymore, the load balacing and multi-CPU loading is done by Enduro/X middleware by it self,
0009 administrator only have to determine how many copies of particular services should
0010 be started. Of-course Enduro/X supports multi-threaded applications too, but now
0011 system architects have a choice either to design multi-threaded executables or
0012 just configure number of executables to start.
0013 
0014 For local inter-process-communication (IPC) Enduro/X uses kernel memory based Posix
0015 queues to avoid overhead of the TCP/IP protocol which is used in other middlewares
0016 or REST based microservice architectures. Thus this approach greatly increases
0017 application speed, as kernel queues is basically a matter of block memory
0018 copy from one process to another (by contrast of 7 layers of TCP/IP stack and
0019 streaming nature of the sockets vs block copy). 
0020 
0021 Enduro/X provides SOA architecture for C/C++ applications and allows to cluster 
0022 application in fault tolerant way over multiple physical servers. 
0023 Enduro/X provides Oracle(R) Tuxedo(R) FML/FML32 library emulation, 
0024 including boolean expressions. Other Tuxedo specific APIs are supported, 
0025 such as tpforward() and work with distributed transactions (tpbegin(), 
0026 tpcommit(), etc.).
0027 
0028 Platform provides X/Open XATMI and XA interfaces/APIs for C/C++ applications. Enduro/X can 
0029 be considered as replacement for Oracle(R) Tuxedo(R), Jboss Blacktie (Narayan), 
0030 Hitachi OpenTP1 and other XATMI middlewares.
0031 
0032 Dual licensed under Affero General Public License Version 3 for use in Open
0033 Source project or Commercial license Acquired from Mavimax Ltd 
0034 (https://www.mavimax.com),
0035 
0036 EnduroX have binddings for:
0037 - Golang (client & server)
0038 - PHP (client)
0039 - Perl (client & server)
0040 - Python (client & server)
0041 - Node.js (client)
0042 
0043 Enduro/X provides following features:
0044 
0045 * Standards based APIs - SCA, The Open Group XATMI
0046 Communication types - Synchronous, Asynchronous, Conversational, Publish/subscribe
0047 * Typed buffers
0048   * UBF (Unified Buffer Format) which provides emulation of Tuxedo's FML/FML32 
0049         format. UBF if high performance binary protocol buffer format. Buffer is 
0050         indexed by binary search on fixed data types.
0051   * STRING buffer format.
0052   * CARRAY (byte array) buffer format.
0053   * JSON buffer format, automatic conversion between JSON and UBF available.
0054   * VIEW buffer (starting from version 5.0+). This offer C structure sending
0055         between processes in cross platform way. Also this allows to map UBF 
0056         fields to VIEW fields, thus helping developer quicker to develop applications, 
0057         by combining UBF and VIEW buffers.
0058 * Transaction Management - Global Transactions - Two-phase commit protocol - X/Open XA
0059 * Clustering - on peer-to-peer basis
0060 * Event broker (also called publish and subscribe messaging)
0061 * Security - Cluster link encryption with GNU PGP framework
0062 * System process monitoring and self healing (pings and restarts)
0063 * SOA Service cache. XATMI services can be cached to LMDB database. Resulting 
0064         that next call to service from any local client receives results 
0065         directly from cache (mainly from direct memory read).
0066 * Dynamic re-configuration
0067 * Custom server polling extensions
0068 * XATMI sub-system is able to work with out main application server daemon (ndrxd)
0069 * Main application server daemon (ndrxd) can be restarted (if crashed). 
0070         When started back it enters in learning mode for some period of time, 
0071         in which in gathers information about system, what services are running, 
0072         etc. After learning =-period, it starts to do normal operations
0073 * tpforward() call
0074 * ATMI server threads may become clients, and can do tpcall()
0075 * Extensive logging & debugging. Enduro/X logging can be configured per binary 
0076         with different log levels. As ATMI servers can be started outside of 
0077         appserver, it is possible to debug them from programming IDE or with 
0078         tools like valgrind.
0079 * For quality assurance project uses automated unit-testing and integration-testing
0080 * Built in ATMI service profiling.
0081 * Environment variables can be updated for XATMI server processes with 
0082         out full application reboot.
0083 * Generic client process monitor (cpm). Subsystem allows to start/stop/monitor 
0084         client executables. At client process crashes, cpm will start it back.
0085 * Starting with version 5.2 Enduro/X provides configuration data encryption feature, 
0086         so that software which is built on top of Enduro/X may comply with 
0087         Payment Card Industry Data Security Standard (PCI/DSS).
0088 
0089 ![Alt text](doc/Endurox-product.jpg?raw=true "Enduro/x overview")
0090 
0091 # Load balancing
0092 
0093 ![Alt text](doc/endurox-load-balance.jpg?raw=true "Enduro/x service load 
0094 balancer")
0095 
0096 Supported operating system: GNU/Linux, starting from 2.6.12 kernel 
0097 (needed for POSIX Queues). Starting with Enduro/X Version 3.1.2 IBM AIX (6.1 and 
0098 7.1), 
0099 Oracle Solaris 11, MAC OS X (experimental) and Cygwin (experimental) 
0100 support is added. Supported compilers: gcc, LLVM clang, IBM xlC.
0101 
0102 - Build and installation guides are located at: www.endurox.org/dokuwiki
0103 
0104 - Support forum: http://www.endurox.org/projects/endurox/boards
0105 
0106 - Binary builds (RPMs & DEBs) for various platforms are here: 
0107 http://www.endurox.org/projects/endurox/files
0108 
0109 - Documentation available here: http://www.endurox.org/dokuwiki/doku.php
0110 
0111 - Source corss reference: http://www.silodev.com/lxr/source
0112 
0113 PS, feel free to contact me at madars@mavimax.com. I will gladly help you
0114  to get EnduroX running ;)
0115 
0116 # Call/message forwarding
0117 
0118 ## Instead of doing calls to each server separately...
0119 
0120 ![Alt text](doc/exforward_tpcall.png?raw=true "Classical service orchestration")
0121 
0122 This is typciall way for example if doing micro-services with HTTP/REST. 
0123 You need to do the calls to each service separately. And that is extra 
0124 overhead to system (multiple returns) and the caller must orchestrate the all 
0125 calls.
0126 
0127 ## ...Enduro/X offers much effective way - tpforward() where request is 
0128 passed around the system
0129 
0130 ![Alt text](doc/exforward_forward.png?raw=true "Enhanced service 
0131 orchestration by forwarding the call")
0132 
0133 In this case the destination service orchestrates the system, it is up to 
0134 service to choose the next service which will continue the call processing. 
0135 The caller is not aware of final service which will do the reply back 
0136 (tpreturn()).
0137 
0138 Note that service after doing "tpforward()" becomes idle and can consume next 
0139 call.
0140 
0141 # High availability and self healing
0142 
0143 Enduro/X is capable of detecting stalled/hanged XATMI servers and gracefully 
0144 can reboot them. The ping times are configurable on per server basis. In case 
0145 if main daemon dies, it is possible to configure special XATMI server named 
0146 "tprecover" which monitors "ndrxd" and vice versa, ndrxd can monitor tprecover. 
0147 Thus if any of these services are not cleanly shut down, then system will 
0148 heal it self, and boot service/daemons back.
0149 
0150 The ndrxd learning mode is some period of time (configurable) in which ndrxd 
0151 will request all the system for active servers and their services, 
0152 to get back the view of the system and be in control. Note that system is able 
0153 to work even with out ndrxd, but at that time no server healing 
0154 and management will be done.
0155 
0156 ![Alt text](doc/server_monitoring_and_recovery.png?raw=true "Enduro/X high 
0157 availability facility")
0158 
0159 # Real time system patching
0160 
0161 It is possible to patch the Enduro/X system (update stateless XATMI servers) 
0162 without service interruption. This is achieved by following scheme:
0163 
0164 ![Alt text](doc/rt-patching.png?raw=true "Enduro/X real time patching")
0165 
0166 # Performance
0167 
0168 Due to fact that Enduro/X uses memory based queues, performance numbers are 
0169 quite high:
0170 
0171 ## ATMI client calls server in async way (tpacall(), no reply from server)
0172 
0173 ![Alt text](doc/benchmark/04_tpacall.png?raw=true "Local tpcall() performance")
0174 
0175 
0176 ## Local one client calls server, and gets back response:
0177 
0178 ![Alt text](doc/benchmark/01_tpcall.png?raw=true "Local tpcall() performance")
0179 
0180 ## Multiple clients calls multiple servers
0181 
0182 ![Alt text](doc/benchmark/03_tpcall_threads.png?raw=true "Multiprocessing 
0183 tpcall() performance")
0184 
0185 ## Single client calls single server via network (two app servers bridged)
0186 
0187 ![Alt text](doc/benchmark/02_tpcall_network.png?raw=true "Network tpcall() 
0188 performance")
0189 
0190 ## Persistent storage (message enqueue to disk via tpenqueue())
0191 The number here are lower because messages are being saved to disk. 
0192 Also internally XA transaction is used, which also requires logging to stable 
0193 storage.
0194 
0195 ![Alt text](doc/benchmark/05_persistent_storage.png?raw=true "Network 
0196 tpenqueue() performance")
0197 
0198 
0199 ## Tpcall cache benchmark
0200 This benchmark shows the performance of cached XATMI service calls.
0201 
0202 ![Alt text](doc/benchmark/06_tpcache.png?raw=true "tpcall() cache performance")
0203 
0204 # Releases
0205 
0206 - Version 2.5.1 released on 18/05/2016. Support for transactional 
0207 persistent message queues. Provides APIs: tpenqueue(), tpdequeue() and 
0208 command line management tools. See doc/persistent_message_queues_overview.adoc, 
0209 doc/manpage/tmqueue.adoc, doc/manpage/xadmin.adoc, doc/manpage/q.config.adoc. 
0210 Use cases can be seen under atmitests/test028_tmq
0211 
0212 - Version 3.1.2 released on 25/06/2016. Added support for other Unix systems. 
0213 Currently production grade support is provided for following operating systems: 
0214 GNU/Linux, FreeBSD, IBM AIX and Oracle Solaris. MAC OS X and Cygwin versions 
0215 are 
0216 considered as experimental. FreeBSD installation guides are located at 
0217 doc/freebsd_notes.adoc, AIX install guide: doc/aix_notes.adoc, 
0218 Solaris install guide: doc/solaris_notes.adoc and OS X guide: 
0219 doc/osx_notes.adoc. Also with this version lot of core processing bug fixes are 
0220 made.
0221 
0222 - Version 3.2.1 released on 06/07/2016. Major UBF optimization. Now for each 
0223 data type memory offset is maintained. For fixed data types (short, long, char, 
0224 float, double) binary search is performend when field is read from buffer.
0225 
0226 - Version 3.2.2 released on 15/07/2016. Bugfixes for UBF binary search. Added 
0227 UBF benchmarking scripts and documents.
0228 
0229 - Version 3.3.1 released on 05/09/2016. Implemented common configuration engine 
0230 (can use ini files in alternative to environment variables, debug config and 
0231 persistent queue config).
0232 
0233 - Version 3.3.2 released on 01/10/2016. Works on TP logger, user accessible 
0234 logging framework. Set of tplog\* functions.
0235 
0236 - Version 3.3.5 released on 10/01/2017. Fixes for RPM build and install on 
0237 Red-hat enterprise linux platform.
0238 
0239 - Version 3.3.6 released on 12/01/2017. Fixes for integration mode, changes in 
0240 tpcontinue().
0241 
0242 - Version 3.4.1 released on 13/01/2017. New ATMI server flag for developer - 
0243 reloadonchange - if cksum change changed on binary issue sreload.
0244 
0245 - Version 3.4.2 released on 20/01/2017. Network protocol framing moved from 2 
0246 bytes to 4 bytes.
0247 
0248 - Version 3.4.3 released on 26/01/2017. Memory leak fixes for tmsrv and tmqueue 
0249 servers.
0250 
0251 - Version 3.4.4 released on 26/01/2017. Works on documentation. Fixes on 
0252 PScript.
0253 
0254 - Version 3.4.5 released on 14/02/2017. Fixes in generators.
0255 
0256 - Version 3.5.1 released on 03/03/2017. Updates on tpreturn() - more correct 
0257 processing as in Tuxedo, free up return buffer + free up auto buffer.
0258 
0259 - Version 3.5.2 released on 24/03/2017. Bug #105 fix. Abort transaction if it 
0260 was in "preparing" stage, and tmsrv loaded it from logfile. Meaning that caller 
0261 alaready do not wait for tpcommit() anymore and got inconclusive results, thus 
0262 better to abort. 
0263 
0264 - Version 3.5.3 released on 26/03/2017. Fixes for Solaris, Sun Studio (SUNPRO) 
0265 compiler
0266 
0267 - Version 3.5.4 released on 01/03/2017. Fixed bug #108 - possible CPM crash
0268 
0269 - Version 3.5.5 released on 23/03/2017. Works Feature #113 - mqd_t use as file 
0270 descriptor - for quick poll operations.
0271 
0272 - Version 3.5.6 released on 02/04/2017. Fixed issues with possible deadlock for 
0273 poll mode on bridge service calls. Fixed issuw with 25 day uptime problem for 
0274 bridges on 32bit platfroms - the messages becomes expired on target server. 
0275 Issues: #112, #145, #140, #139, #113
0276 
0277 - Version 3.5.7 release on 23/04/2017. Fixed Bug #148 (Bdelall corrupts ubf 
0278 buffer)
0279 Bug #110 (tpbridge does not report connection status to ndrxd after ndrxd is 
0280 restarted for recovery)
0281 
0282 - Version 3.5.9 released on 22/06/2017. Bug fix #160 - retry xa_start with 
0283 xa_close/xa_open. New env variable NDRX_XA_FLAGS, tag "RECON".
0284 
0285 - Version 4.0.1 released on 29/06/2017. Multi-threaded bridge implementation. 
0286 tpnotify(), tpbroadcast(), tpsetunsol(), tpchkunsol() API implementation.
0287 
0288 - Version 4.0.2 released on 19/07/2017. Fixed ndrxd core dump issue Bug #174.
0289 
0290 - Version 4.0.3 released on 26/07/2017. Fixes of #176 #178. NDRX_DUMP, UBF_DUMP 
0291 fixes.
0292 
0293 - Version 4.0.4 released on 29/07/2017. Feature #162 - added evt_mib.h
0294 
0295 - Version 4.0.5 released on 01/08/2017. Feature #180 - refactoring code towards 
0296 ISO/IEC 9899:201x standard.
0297 
0298 - Version 5.0.1 released on 21/08/2017. Feature #96 - Typed View support, Bug 
0299 #182 - fix in bridge protocol.
0300 
0301 - Version 5.0.2 released on 22/08/2017. Bug #183
0302 
0303 - Version 5.0.3 released on 22/08/2017. Bug #186
0304 
0305 - Version 5.0.4 released on 22/08/2017. Bug #186
0306 
0307 - Version 5.0.5 released on 27/08/2017. Bug #184, Bug #191, Bug #190, Bug #192, 
0308 Bug #193
0309 
0310 - Version 5.0.6 released on 28/08/2017. Bug #195
0311 
0312 - Version 5.0.7 released on 01/09/2017. Feature #161 - tmsrv database pings & 
0313 automatic reconnecting in case of network failures.
0314 
0315 - Version 5.0.8 released on 17/09/2017. Works on dynamic view access. #99, #206, 
0316 #207, #210
0317 
0318 - Version 5.0.9 released on 27/09/2017. (development) Works on #224
0319 
0320 - Version 5.0.10 released on 13/10/2017. (stable) Feature #232, Feature #231, 
0321 -O2 optimization by default
0322 
0323 - Version 5.1.1  released on 13/10/2017. (development) Feature #127, Bug #229, 
0324 Feature #230, Bug #234, Feature #244, Bug #243, Feature #248, Bug #240, Bug #238 
0325 
0326 - basically big message size support (over the 64K)
0327 
0328 - Version 5.1.2  released on 18/10/2017. (stable) Bug #250
0329 
0330 - Version 5.2.1  released on 18/12/2017. (development) Major work on support for PCI/DSS mandatory
0331 configuration encryption. Introduction of plugin architecture (currently used for
0332 crypto key providers). Implemented tpconvert() ATMI call. Now @global section for ini files are 
0333 read twice. Thus ini file can reference to previosly defined env/global variable. 
0334 Fixes: #261 Bug, #118 Feature, #237 Feature, #236 Bug, #245 Feature, #258
0335 Support, #259 Support, #255 Bug, #254 Bug.
0336 
0337 - Version 5.2.2 released on 21/12/2017. (stable) First stable version of Enduor/X 5.2. Rolled
0338 
0339 - Version 5.2.4 released on 22/12/2017. (stable) Fixed Bug #268.
0340 
0341 - Version 5.2.6 released on 02/01/2018. (stable) Happy New Year! Fixed Bug #269.
0342 
0343 - Version 5.2.8 released on 26/01/2018. (stable) Fixed Bug #274 - too many open files,
0344 when threaded logger using for multi-contexting (like go runtime)
0345 
0346 - Version 5.2.10 released on 27/01/2018. (stable) Feature #275 - allow to mask
0347 
0348 - Version 5.2.12 released on 27/01/2018. (stable) Feature #275 - fixes for server un-init (not critical)
0349 
0350 - Version 5.2.14 released on 03/02/2018. (stable) Feature #278 - new fields for compiled connection id
0351 
0352 - Version 5.2.15 released on 08/02/2018. (development) Feature #282 - new UBF api Baddfast()
0353 
0354 - Version 5.2.16 released on 09/02/2018. (stable) Feature #282 - new UBF api Baddfast(), finished
0355 documentation and added unit tests.
0356 
0357 - Version 5.3.2 released on 25/03/2018 (stable) Fixes on Feature #272, Bug #291, Feature #287, Bug #290, Support #279
0358 
0359 - Version 5.3.4 released on 26/03/2018 (stable) Feature #294, Bug #293
0360 
0361 - Version 5.3.5 released on 28/03/2018 (Development) Working progress on Feature #295
0362 
0363 - Version 5.3.6 released on 01/04/2018 (stable) Finished works on Feature #295
0364 
0365 - Version 5.3.8 released on 06/04/2018 (stable) Implemented Feature #299 - new flag TPNOABORT
0366 
0367 - Version 5.3.9 released on 09/04/2018 (development) Fixed Bug #300
0368 
0369 - Version 5.3.10 released on 10/04/2018 (stable) Fixed Support #301
0370 
0371 - Version 5.3.12 released on 12/04/2018 (stable) Support #302 - added last 8 chars of hostname to logfile
0372 
0373 - Version 5.3.14 released on 21/04/2018 (stable) Bug #306
0374 
0375 - Version 5.3.15 released on 23/04/2018 (development) Feature #307
0376 
0377 - Version 5.3.16 released on 16/05/2018 (stable) Bug #317
0378 
0379 - Version 5.3.17 released on 18/05/2018 (development) Feature #320 - cmdline tag for server in xml config
0380 
0381 - Version 5.3.18 released on 27/06/2018 (stable) Bug #325
0382 
0383 - Version 5.3.20 released on 10/10/2018 (stable) Backport Bug #321 fix to 5.3
0384 
0385 # Build configurations
0386 
0387 ## Configure make with: 
0388 
0389 $ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/dist .
0390 
0391 ### Flags:
0392 
0393 - To disable GPG_ME, pass additional flag to cmake '-DDEFINE_DISABLEGPGME=ON'
0394 
0395 - To disable documentation building add '-DDEFINE_DISABLEDOC=ON'
0396 
0397 - To enable poll() use instead of epoll() in Linux use '-DDEFINE_FORCEPOLL=ON'
0398 
0399 - To disable platform script building use '-DDEFINE_DISABLEPSCRIPT=ON'
0400 
0401 - To do release build, use '-DDEFINE_RELEASEBUILD=ON'
0402 
0403 - To force use emulated message queue, add '-DDEFINE_FORCEEMQ=ON'
0404 
0405 - To log the memory allocation to user log add '-DNDRX_MEMORY_DEBUG=1' 
0406 
0407 - To trace of the Object-API use '-DNDRX_OAPI_DEBUG=1' 
0408 
0409 - To enable file descriptor based treating of the queues 
0410 '-DDEFINE_FORCEFDPOLL=ON'
0411 
0412 - To trace of the Semaphore handling use '-DNDRX_SEM_DEBUG=1' 
0413 
0414 - To enable test047 with Oracle PRO*C database access use '-DENABLE_TEST47=ON'
0415 Note that *proc* must be in path and ORACLE_HOME must be set. Also Oracle DB 
0416 libraries must be present in LD_LIBRARY_PATH (or equivalent environment for 
0417 target OS).