Performance

  • Most edi files are just a few kilobyes. An edi file of 5Mb is very large (edifact, x12). If you encounter larger ones: please let me know.

  • AFAIK there are no issues with performance. If you run into this: please inform me eg via mailing list.

Get More Performance

  • (Bots >= 2.2.0) Cdecimals (Extern library) speeds up bots. This library will be included in python 3.3, but can be installed in earlier python versions.

  • (bots >= 3.1) Do not use get_checklevel=2 in config/bots.ini. This does extended checking of mpath’s, use this during development only.

  • (bots >= 3.0) Schedule bots-engine via the job queue server.

  • For xml: check if the c-version of python’s elementtree is installed and used.

  • Enough memory is important for performance: disk-memory swapping is slow. Actual memory usage depends on size of edi-files.

  • Check mappings for slow/inefficient algorithms.

  • Bots works with pypy, see below on this page.

  • Use SSD for faster reading/writing. In config/bots.ini the botssys-directory can be set, in config/settings.py the place of the SQLite-database.

Strategy for bigger edi volumes

  • Best strategy is to schedule bots-engine more often.

  • (bots >= 3.0) Schedule bots-engine via the jobqueue server.

  • Routes can be scheduled independently.

  • Set-up good scheduling, keeping volumes in mind.

  • EDI in the real world has often large peaks.

  • Dome edi transactions are time critical (eg orders), others not so much (eg invoices)

  • Check where the large volumes are (size and number of edi-transactions)

  • Look at the sending pattern of your customers. Often edi is send in night jobs, so you might receive lot of volume early in the morning.

  • Check where you send large volumes. Send this at a time that does not interfere with other flows.

  • Incoming volumes can be limited per run. This way the time bots-engine runs is predictable.

    • The max time a channel fetches incoming files is a parameter for each channel.

    • This is dependent upon the communication type used; eg file system I/O is much faster than SFTP.

    • Files “left behind” will be fetched on subsequent runs.

  • (Bots >= 3.0) Limit for max file-size (set in bots.ini). If an incoming file is larger, bots will give error. This is to prevent accidents.

Performance/throughput testing

  1. Tests are done using file system I/O (no testing of communication performance).

  2. Tests done in one run of bots-engine.

  3. Test system: Intel Q9400 2.66GHz; 4Gb memory; ubuntu 10.04(lucid); python 2.7; default SQLite database.

  4. Please note that these tests are artificial: if you have such high volumes and big files look at good scheduling.

  5. Tests are with edifact; x12 performance is the same.

Description

File Count

Total Size

Message Count

Time(bots2.0)

Speed(bots2.0)

Time(bots2.2)

Speed(bots2.2)

Time(bots3.2)

Speed(bots3.2)

01 edifact2fixed

32

305Mb

32

1:01:39

82 kb/s

0:50:57

100 kb/s

0:44:01

115 kb/s

02 edifact2fixed

116

300Mb

116

1:14:18

68 kb/s

0:36:20

137 kb/s

0:37:25

133 kb/s

03 edifact2fixed

94048

295Mb

141072

0:47:21

104 kb/s

0:39:54

125 kb/s

0:42:30

115 kb/s

04 fixed2edifact

14244

300Mb

78342

1:04:11

78 kb/s

0:33:21

150 kb/s

0:32:40

153 kb/s

05 xml2edifact

17424

300Mb

17424

0:41:24

121 kb/s

0:35:48

139 kb/s

0:35:20

141 kb/s

06 edifact2xml(1to1)

14609

300Mb

74919

1:23:03

60 kb/s

0:58:19

85 kb/s

0:44:38

112 kb/s

Conclusions of performance measurements:

  1. Memory usage is stable (no leakage).

  2. Memory usage is directly related to the size of the edi-files. In test 01 (edifact files of 9.5Mb) bots-engine uses 1.5Gb memory.

  3. Tested with edifact files of 120Mb; memory usage is stable at 4.5Gb.

  4. Performance is reasonably independent from the size of edi-files/messages.

  5. An edifact file of 9.5Mb takes about 85sec to be processed.

  6. For outgoing edi files: writing to one file or multiple file does not significantly affect performance.

Testing with pypy

pypy s a python implementation that is faster by using a JIT. Results of the first tests with pypy (beta-versions of pypy 2.0):

  • Bots works with pypy.

  • Comparing some stress tests: much faster, 2-3 times faster.

  • Did not run all test-sets. Probably I will do that with definitive version of pypy 2.0 and/or release of bots 3.1.

  • Problem might be that not all libraries/dependencies work with pypy.

    • SQLite3 database connector: OK

    • MySQL database connector: version 1.2.5 does. Note that bots 3.1.0 gave am error with this version, a patch was easy.

    • paramiko (for SFTP/SSH): no, dependency pycrypto is not supported.

This looks like a very interesting development.