Back to home page

Enduro/X

 
 

    


0001 # endurox
0002 Enduro/X is Open Source Middleware Platform for Distributed Transaction 
0003 Processing
0004 
0005 It is modern, microservices based middleware for writing distributed, open
0006 systems (program consists of several executables) based applications. Thus 
0007 by using Enduro/X programmers do not have to worry about threads and concurrency 
0008 anymore, the load balacing and multi-CPU loading is done by Enduro/X middleware by it self,
0009 administrator only have to determine how many copies of particular services should
0010 be started. Of-course Enduro/X supports multi-threaded applications too, but now
0011 system architects have a choice either to design multi-threaded executables or
0012 just configure number of executables to start.
0013 
0014 For local inter-process-communication (IPC) Enduro/X uses kernel memory based Posix
0015 queues to avoid overhead of the TCP/IP protocol which is used in other middlewares
0016 or REST based microservice architectures. Thus this approach greatly increases
0017 application speed, as kernel queues is basically a matter of block memory
0018 copy from one process to another (by contrast of 7 layers of TCP/IP stack and
0019 streaming nature of the sockets vs block copy). 
0020 
0021 Enduro/X provides SOA architecture for C/C++ applications and allows to cluster 
0022 application in fault tolerant way over multiple physical servers. 
0023 Enduro/X provides Oracle(R) Tuxedo(R) FML/FML32 library emulation, 
0024 including boolean expressions. Other Tuxedo specific APIs are supported, 
0025 such as tpforward() and work with distributed transactions (tpbegin(), 
0026 tpcommit(), etc.).
0027 
0028 Platform provides X/Open XATMI and XA interfaces/APIs for C/C++ applications. Enduro/X can 
0029 be considered as replacement for Oracle(R) Tuxedo(R), Jboss Blacktie (Narayan), 
0030 Hitachi OpenTP1 and other XATMI middlewares.
0031 
0032 EnduroX have binddings for:
0033 - Golang (client & server)
0034 - PHP (client)
0035 - Perl (client & server)
0036 - Python (client & server)
0037 - Node.js (client)
0038 
0039 Enduro/X provides following features:
0040 
0041 * Standards based APIs - SCA, The Open Group XATMI
0042 Communication types - Synchronous, Asynchronous, Conversational, Publish/subscribe
0043 * Typed buffers
0044   * UBF (Unified Buffer Format) which provides emulation of Tuxedo's FML/FML32 format. UBF if high performance binary protocol buffer format. Buffer is indexed by binary search on fixed data types.
0045   * STRING buffer format.
0046   * CARRAY (byte array) buffer format.
0047   * JSON buffer format, automatic conversion between JSON and UBF available.
0048   * VIEW buffer (starting from version 5.0+). This offer C structure sending between processes in cross platform way. Also this allows to map UBF fields to VIEW fields, thus helping developer quicker to develop applications, by combining UBF and VIEW buffers.
0049 * Transaction Management - Global Transactions - Two-phase commit protocol - X/Open XA
0050 * Clustering - on peer-to-peer basis
0051 * Event broker (also called publish and subscribe messaging)
0052 * Security - Cluster link encryption with GNU PGP framework
0053 * System process monitoring and self healing (pings and restarts)
0054 * SOA Service cache. XATMI services can be cached to LMDB database. Resulting that next call to service from any local client receives results directly from cache (mainly from direct memory read).
0055 * Dynamic re-configuration
0056 * Custom server polling extensions
0057 * XATMI sub-system is able to work with out main application server daemon (ndrxd)
0058 * Main application server daemon (ndrxd) can be restarted (if crashed). When started back it enters in learning mode for some period of time, in which in gathers information about system, what services are running, etc. After learning =-period, it starts to do normal operations
0059 * tpforward() call
0060 * ATMI server threads may become clients, and can do tpcall()
0061 * Extensive logging & debugging. Enduro/X logging can be configured per binary with different log levels. As ATMI servers can be started outside of appserver, it is possible to debug them from programming IDE or with tools like valgrind.
0062 * For quality assurance project uses automated unit-testing and integration-testing
0063 * Built in ATMI service profiling.
0064 * Environment variables can be updated for XATMI server processes with out full application reboot.
0065 * Generic client process monitor (cpm). Subsystem allows to start/stop/monitor client executables. At client process crashes, cpm will start it back.
0066 * Starting with version 5.2 Enduro/X provides configuration data encryption feature, so that software which is built on top of Enduro/X may comply with Payment Card Industry Data Security Standard (PCI/DSS).
0067 
0068 ![Alt text](doc/Endurox-product.jpg?raw=true "Enduro/x overview")
0069 
0070 # Load balancing
0071 
0072 ![Alt text](doc/endurox-load-balance.jpg?raw=true "Enduro/x service load 
0073 balancer")
0074 
0075 Supported operating system: GNU/Linux, starting from 2.6.12 kernel 
0076 (needed for POSIX Queues). Starting with Enduro/X Version 3.1.2 IBM AIX (6.1 and 
0077 7.1), 
0078 Oracle Solaris 11, MAC OS X (experimental) and Cygwin (experimental) 
0079 support is added. Supported compilers: gcc, LLVM clang, IBM xlC.
0080 
0081 - Build and installation guides are located at: www.endurox.org/dokuwiki
0082 
0083 - Support forum: http://www.endurox.org/projects/endurox/boards
0084 
0085 - Binary builds (RPMs & DEBs) for various platforms are here: 
0086 http://www.endurox.org/projects/endurox/files
0087 
0088 - Documentation available here: http://www.endurox.org/dokuwiki/doku.php
0089 
0090 - Source corss reference: http://www.silodev.com/lxr/source
0091 
0092 PS, feel free to contact me at madars@mavimax.com. I will gladly help you
0093  to get EnduroX running ;)
0094 
0095 # Call/message forwarding
0096 
0097 ## Instead of doing calls to each server separately...
0098 
0099 ![Alt text](doc/exforward_tpcall.png?raw=true "Classical service orchestration")
0100 
0101 This is typciall way for example if doing micro-services with HTTP/REST. 
0102 You need to do the calls to each service separately. And that is extra 
0103 overhead to system (multiple returns) and the caller must orchestrate the all 
0104 calls.
0105 
0106 ## ...Enduro/X offers much effective way - tpforward() where request is 
0107 passed around the system
0108 
0109 ![Alt text](doc/exforward_forward.png?raw=true "Enhanced service 
0110 orchestration by forwarding the call")
0111 
0112 In this case the destination service orchestrates the system, it is up to 
0113 service to choose the next service which will continue the call processing. 
0114 The caller is not aware of final service which will do the reply back 
0115 (tpreturn()).
0116 
0117 Note that service after doing "tpforward()" becomes idle and can consume next 
0118 call.
0119 
0120 # High availability and self healing
0121 
0122 Enduro/X is capable of detecting stalled/hanged XATMI servers and gracefully 
0123 can reboot them. The ping times are configurable on per server basis. In case 
0124 if main daemon dies, it is possible to configure special XATMI server named 
0125 "tprecover" which monitors "ndrxd" and vice versa, ndrxd can monitor tprecover. 
0126 Thus if any of these services are not cleanly shut down, then system will 
0127 heal it self, and boot service/daemons back.
0128 
0129 The ndrxd learning mode is some period of time (configurable) in which ndrxd 
0130 will request all the system for active servers and their services, 
0131 to get back the view of the system and be in control. Note that system is able 
0132 to work even with out ndrxd, but at that time no server healing 
0133 and management will be done.
0134 
0135 ![Alt text](doc/server_monitoring_and_recovery.png?raw=true "Enduro/X high 
0136 availability facility")
0137 
0138 # Real time system patching
0139 
0140 It is possible to patch the Enduro/X system (update stateless XATMI servers) 
0141 without service interruption. This is achieved by following scheme:
0142 
0143 ![Alt text](doc/rt-patching.png?raw=true "Enduro/X real time patching")
0144 
0145 # Performance
0146 
0147 Due to fact that Enduro/X uses memory based queues, performance numbers are 
0148 quite high:
0149 
0150 ## ATMI client calls server in async way (tpacall(), no reply from server)
0151 
0152 ![Alt text](doc/benchmark/04_tpacall.png?raw=true "Local tpcall() performance")
0153 
0154 
0155 ## Local one client calls server, and gets back response:
0156 
0157 ![Alt text](doc/benchmark/01_tpcall.png?raw=true "Local tpcall() performance")
0158 
0159 ## Multiple clients calls multiple servers
0160 
0161 ![Alt text](doc/benchmark/03_tpcall_threads.png?raw=true "Multiprocessing 
0162 tpcall() performance")
0163 
0164 ## Single client calls single server via network (two app servers bridged)
0165 
0166 ![Alt text](doc/benchmark/02_tpcall_network.png?raw=true "Network tpcall() 
0167 performance")
0168 
0169 ## Persistent storage (message enqueue to disk via tpenqueue())
0170 The number here are lower because messages are being saved to disk. 
0171 Also internally XA transaction is used, which also requires logging to stable 
0172 storage.
0173 
0174 ![Alt text](doc/benchmark/05_persistent_storage.png?raw=true "Network 
0175 tpenqueue() performance")
0176 
0177 
0178 ## Tpcall cache benchmark
0179 This benchmark shows the performance of cached XATMI service calls.
0180 
0181 ![Alt text](doc/benchmark/06_tpcache.png?raw=true "tpcall() cache performance")
0182 
0183 # Releases
0184 
0185 - Version 2.5.1 released on 18/05/2016. Support for transactional 
0186 persistent message queues. Provides APIs: tpenqueue(), tpdequeue() and 
0187 command line management tools. See doc/persistent_message_queues_overview.adoc, 
0188 doc/manpage/tmqueue.adoc, doc/manpage/xadmin.adoc, doc/manpage/q.config.adoc. 
0189 Use cases can be seen under atmitests/test028_tmq
0190 
0191 - Version 3.1.2 released on 25/06/2016. Added support for other Unix systems. 
0192 Currently production grade support is provided for following operating systems: 
0193 GNU/Linux, FreeBSD, IBM AIX and Oracle Solaris. MAC OS X and Cygwin versions 
0194 are 
0195 considered as experimental. FreeBSD installation guides are located at 
0196 doc/freebsd_notes.adoc, AIX install guide: doc/aix_notes.adoc, 
0197 Solaris install guide: doc/solaris_notes.adoc and OS X guide: 
0198 doc/osx_notes.adoc. Also with this version lot of core processing bug fixes are 
0199 made.
0200 
0201 - Version 3.2.1 released on 06/07/2016. Major UBF optimization. Now for each 
0202 data type memory offset is maintained. For fixed data types (short, long, char, 
0203 float, double) binary search is performend when field is read from buffer.
0204 
0205 - Version 3.2.2 released on 15/07/2016. Bugfixes for UBF binary search. Added 
0206 UBF benchmarking scripts and documents.
0207 
0208 - Version 3.3.1 released on 05/09/2016. Implemented common configuration engine 
0209 (can use ini files in alternative to environment variables, debug config and 
0210 persistent queue config).
0211 
0212 - Version 3.3.2 released on 01/10/2016. Works on TP logger, user accessible 
0213 logging framework. Set of tplog\* functions.
0214 
0215 - Version 3.3.5 released on 10/01/2017. Fixes for RPM build and install on 
0216 Red-hat enterprise linux platform.
0217 
0218 - Version 3.3.6 released on 12/01/2017. Fixes for integration mode, changes in 
0219 tpcontinue().
0220 
0221 - Version 3.4.1 released on 13/01/2017. New ATMI server flag for developer - 
0222 reloadonchange - if cksum change changed on binary issue sreload.
0223 
0224 - Version 3.4.2 released on 20/01/2017. Network protocol framing moved from 2 
0225 bytes to 4 bytes.
0226 
0227 - Version 3.4.3 released on 26/01/2017. Memory leak fixes for tmsrv and tmqueue 
0228 servers.
0229 
0230 - Version 3.4.4 released on 26/01/2017. Works on documentation. Fixes on 
0231 PScript.
0232 
0233 - Version 3.4.5 released on 14/02/2017. Fixes in generators.
0234 
0235 - Version 3.5.1 released on 03/03/2017. Updates on tpreturn() - more correct 
0236 processing as in Tuxedo, free up return buffer + free up auto buffer.
0237 
0238 - Version 3.5.2 released on 24/03/2017. Bug #105 fix. Abort transaction if it 
0239 was in "preparing" stage, and tmsrv loaded it from logfile. Meaning that caller 
0240 alaready do not wait for tpcommit() anymore and got inconclusive results, thus 
0241 better to abort. 
0242 
0243 - Version 3.5.3 released on 26/03/2017. Fixes for Solaris, Sun Studio (SUNPRO) 
0244 compiler
0245 
0246 - Version 3.5.4 released on 01/03/2017. Fixed bug #108 - possible CPM crash
0247 
0248 - Version 3.5.5 released on 23/03/2017. Works Feature #113 - mqd_t use as file 
0249 descriptor - for quick poll operations.
0250 
0251 - Version 3.5.6 released on 02/04/2017. Fixed issues with possible deadlock for 
0252 poll mode on bridge service calls. Fixed issuw with 25 day uptime problem for 
0253 bridges on 32bit platfroms - the messages becomes expired on target server. 
0254 Issues: #112, #145, #140, #139, #113
0255 
0256 - Version 3.5.7 release on 23/04/2017. Fixed Bug #148 (Bdelall corrupts ubf 
0257 buffer)
0258 Bug #110 (tpbridge does not report connection status to ndrxd after ndrxd is 
0259 restarted for recovery)
0260 
0261 - Version 3.5.9 released on 22/06/2017. Bug fix #160 - retry xa_start with 
0262 xa_close/xa_open. New env variable NDRX_XA_FLAGS, tag "RECON".
0263 
0264 - Version 4.0.1 released on 29/06/2017. Multi-threaded bridge implementation. 
0265 tpnotify(), tpbroadcast(), tpsetunsol(), tpchkunsol() API implementation.
0266 
0267 - Version 4.0.2 released on 19/07/2017. Fixed ndrxd core dump issue Bug #174.
0268 
0269 - Version 4.0.3 released on 26/07/2017. Fixes of #176 #178. NDRX_DUMP, UBF_DUMP 
0270 fixes.
0271 
0272 - Version 4.0.4 released on 29/07/2017. Feature #162 - added evt_mib.h
0273 
0274 - Version 4.0.5 released on 01/08/2017. Feature #180 - refactoring code towards 
0275 ISO/IEC 9899:201x standard.
0276 
0277 - Version 5.0.1 released on 21/08/2017. Feature #96 - Typed View support, Bug 
0278 #182 - fix in bridge protocol.
0279 
0280 - Version 5.0.2 released on 22/08/2017. Bug #183
0281 
0282 - Version 5.0.3 released on 22/08/2017. Bug #186
0283 
0284 - Version 5.0.4 released on 22/08/2017. Bug #186
0285 
0286 - Version 5.0.5 released on 27/08/2017. Bug #184, Bug #191, Bug #190, Bug #192, 
0287 Bug #193
0288 
0289 - Version 5.0.6 released on 28/08/2017. Bug #195
0290 
0291 - Version 5.0.7 released on 01/09/2017. Feature #161 - tmsrv database pings & 
0292 automatic reconnecting in case of network failures.
0293 
0294 - Version 5.0.8 released on 17/09/2017. Works on dynamic view access. #99, #206, 
0295 #207, #210
0296 
0297 - Version 5.0.9 released on 27/09/2017. (development) Works on #224
0298 
0299 - Version 5.0.10 released on 13/10/2017. (stable) Feature #232, Feature #231, 
0300 -O2 optimization by default
0301 
0302 - Version 5.1.1  released on 13/10/2017. (development) Feature #127, Bug #229, 
0303 Feature #230, Bug #234, Feature #244, Bug #243, Feature #248, Bug #240, Bug #238 
0304 
0305 - basically big message size support (over the 64K)
0306 
0307 - Version 5.1.2  released on 18/10/2017. (stable) Bug #250
0308 
0309 - Version 5.2.1  released on 18/12/2017. (development) Major work on support for PCI/DSS mandatory
0310 configuration encryption. Introduction of plugin architecture (currently used for
0311 crypto key providers). Implemented tpconvert() ATMI call. Now @global section for ini files are 
0312 read twice. Thus ini file can reference to previosly defined env/global variable. 
0313 Fixes: #261 Bug, #118 Feature, #237 Feature, #236 Bug, #245 Feature, #258
0314 Support, #259 Support, #255 Bug, #254 Bug.
0315 
0316 - Version 5.2.2 released on 21/12/2017. (stable) First stable version of Enduor/X 5.2. Rolled
0317 
0318 - Version 5.2.4 released on 22/12/2017. (stable) Fixed Bug #268.
0319 
0320 - Version 5.2.6 released on 02/01/2018. (stable) Happy New Year! Fixed Bug #269.
0321 
0322 - Version 5.2.8 released on 26/01/2018. (stable) Fixed Bug #274 - too many open files,
0323 when threaded logger using for multi-contexting (like go runtime)
0324 
0325 - Version 5.2.10 released on 27/01/2018. (stable) Feature #275 - allow to mask
0326 
0327 - Version 5.2.12 released on 27/01/2018. (stable) Feature #275 - fixes for server un-init (not critical)
0328 
0329 - Version 5.2.14 released on 03/02/2018. (stable) Feature #278 - new fields for compiled connection id
0330 
0331 - Version 5.2.15 released on 08/02/2018. (development) Feature #282 - new UBF api Baddfast()
0332 
0333 - Version 5.2.16 released on 09/02/2018. (stable) Feature #282 - new UBF api Baddfast(), finished
0334 documentation and added unit tests.
0335 
0336 - Version 5.3.2 released on 25/03/2018 (stable) Fixes on Feature #272, Bug #291, Feature #287, Bug #290, Support #279
0337 
0338 - Version 5.3.4 released on 26/03/2018 (stable) Feature #294, Bug #293
0339 
0340 - Version 5.3.5 released on 28/03/2018 (Development) Working progress on Feature #295
0341 
0342 - Version 5.3.6 released on 01/04/2018 (stable) Finished works on Feature #295
0343 
0344 - Version 5.3.8 released on 06/04/2018 (stable) Implemented Feature #299 - new flag TPNOABORT
0345 
0346 - Version 5.3.9 released on 09/04/2018 (development) Fixed Bug #300
0347 
0348 - Version 5.3.10 released on 10/04/2018 (stable) Fixed Support #301
0349 
0350 - Version 5.3.12 released on 12/04/2018 (stable) Support #302 - added last 8 chars of hostname to logfile
0351 
0352 - Version 5.3.14 released on 21/04/2018 (stable) Bug #306
0353 
0354 # Build configurations
0355 
0356 ## Configure make with: 
0357 
0358 $ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/dist .
0359 
0360 ### Flags:
0361 
0362 - To disable GPG_ME, pass additional flag to cmake '-DDEFINE_DISABLEGPGME=ON'
0363 
0364 - To disable documentation building add '-DDEFINE_DISABLEDOC=ON'
0365 
0366 - To enable poll() use instead of epoll() in Linux use '-DDEFINE_FORCEPOLL=ON'
0367 
0368 - To disable platform script building use '-DDEFINE_DISABLEPSCRIPT=ON'
0369 
0370 - To do release build, use '-DDEFINE_RELEASEBUILD=ON'
0371 
0372 - To force use emulated message queue, add '-DDEFINE_FORCEEMQ=ON'
0373 
0374 - To log the memory allocation to user log add '-DNDRX_MEMORY_DEBUG=1' 
0375 
0376 - To trace of the Object-API use '-DNDRX_OAPI_DEBUG=1' 
0377 
0378 - To enable file descriptor based treating of the queues 
0379 '-DDEFINE_FORCEFDPOLL=ON'
0380 
0381 - To trace of the Semaphore handling use '-DNDRX_SEM_DEBUG=1' 
0382 
0383 - To enable test047 with Oracle PRO*C database access use '-DENABLE_TEST47=ON'
0384 Note that *proc* must be in path and ORACLE_HOME must be set. Also Oracle DB 
0385 libraries must be present in LD_LIBRARY_PATH (or equivalent environment for 
0386 target OS).