Bots 3.0.0

Bots 3.0.0 was released 2013-02-04
Bigger changes, and a database migration is needed.

Migration Notes

Introduction

Version 3.0 is a bigger update for bots, view all changes in next section. Overview:

Django 1.1 and 1.2 are not supported anymore. Supported are django 1.3 and 1.4.

Settings.py is changed. Advised: use the new settings.py, and do your customization in the new setting.py (eg for error reports, maybe database and timezone).

The database has changed. A script is included to change the database. I had no database issues while testing this migration.

lots of changes in bots.ini. The ‘old’ bots.ini is OK, but it is advised to use new bots.ini, and do your customizations there.

Excel input: does work; but is now an incoming messagetype and not via ‘preprocessing’.

Most user scripts will work; many user scripts are not needed anymore because the functionality is provided by bots now.

Some functions that may be used in user scripting have changed:

botslib.change() -> botslib.changeq(). This function is used to in processing incoming 997’s and CONTRL

botsengine routescripts: for ‘new’ runs in routescript botsengine.py called function postnewrun(routestorun) -> postnew(routestorun)

communication.run(idchannel,idroute) -> communication.run(idchannel,command,idroute). Command is one of: ‘new’,’automaticretrycommunication’,’resend’,’rereceive’.

transform.run(idchannel,idroute) -> transform.run(idchannel,command,idroute). Command is one of: ‘new’,’automaticretrycommunication’,’resend’,’rereceive’.

Note

My experiences: after changing settings.py and database migration, all works (except for and issues mentioned above).

Summary of procedure

make a backup!

rename existing installation.

do a fresh install.

copy old data to new installation.

change settings

update the database.

Note

It is critical to change the settings before updating the database. Bots finds the right database via the settings.

If you use MySQL or PostGreSQL: same procedure. The bots-updatedb script also updates MySQL or PostGreSQL.

Tested this for migration from bots2.2.1 -> 3.0.0, but works for all bots2.* installations.

Windows procedure

make a backup!

rename existing installation

existing bots-installation is in C:\Python27\Lib\site-packages

renamed bots directory to bots221

also renamed existing directories for cherrypy, django, genshi.

do a fresh install of bots3.0.0 installer (bots-3.0.0.win32.exe)

copy old data to new installation.

in C:\Python27\Lib\site-packages new directories have been installed (bots, django, cherrypy, genshi)

copy directories botssys and usersys from bots221-directory to bots directories. Everything can be overwritten.

change settings

use new config/bots.ini, adapt for your own values.

use new config/settings.py, adapt for your own values. Especially the database settings are important; the format is slightly different (but similar enough to give no problem); critical is using the ‘ENGINE’-value of the new settings.py.

update the database.

use command-prompt/dos-box

goto directory C:\Python27\Scripts

command-line: C:\Python27\python bots-updatedb.py

should report that database is successful changed.

Note

If you use a 64-bits version of windows another option is to use the 64-bits versions of python and bots.

Linux procedure

make a backup!

rename existing installation

existing bots-installation is in /usr/local/lib/python2.7/dist-packages

renamed bots directory to bots221

for libraries: check you use at least django 1.3

do a fresh install: see installation procedure

copy old data to new installation.

in /usr/local/lib/python2.7/dist-packages new bots-directory is installed.

copy directories botssys and usersys from bots221 directory to bots directories. Everything can be overwritten.

mind your rights.

change settings

use new config/bots.ini, adapt for your own values.

use new config/settings.py, adapt for your own values. Especially the database settings are important; the format is slightly different (but similar enough to give no problem); critical is using the ‘ENGINE’-value of the new settings.py.

update the database.

command-line: bots-updatedb.py

should report that database is successful changed.

Changes

Changes in database format since version 3.0.0rc

Alas; the database as used in the 3.0.0rc version has changed. Changed is:

table channel: field ‘testpath’ is added

table report: field ‘acceptance’ is added

tabel ccode: field ‘rightcode’->70pos

tabel ccode: field ‘attr1’->70pos

Wrapping of user script functionality into GUI

For outgoing filenames: can include partnerID, messagetype, botskey, data/time, etc in channel.

Pass-through is option in route.

Zip and unzip files as option in channel.

SSL keyfile and certificate as options in channel.

Excel as incoming messagetype (instead of via preprocessing).

Option in channel to indicate edi file in email should be in body.

Add: communication-out type ‘trash’ to discard of edi files.

Improved ‘run’ options

Run options in GUI are simpler: new, rereceive, resend. (Automatic recommunication is possible via bots-engine).

Communication errors are visible in outgoing-screen and can be ‘resend’ manually.

Add ‘resend all’ and ‘rereceive all’ in incoming/outgoing screen; espc. useful in combination with selections.

Indicate in screens a file has been resend; Keep track of number of resends of an outgoing file. Filer works ‘over’ resends now.

Dropped ‘retry’ option. This was not useful and confusing. Use ‘rereceive’.

Dropped ‘retry communication’. Use ‘resend’, easier and more consistent.

Add: use ‘channel’ in selects (eg useful for resend selects).

Configuration change management :doc:`see wiki <../deployment/change-management>`

Use tools to push changes test-> production

Via GUI: write database configuration to usersys in order to compare environments.

Via GUI: read the changed database configuration in usersys after pushing changes

Isolated acceptation tests: run an acceptance test set without changes.

option in bots.ini to active ‘acceptance test’

bots prevents communication to ‘outside

use seperate path to read/write (‘testpath’)

do not change counters etc

results of acceptance tests can be deleted/removed

etc

Many improvements in GUI, eg:

‘Detail’ screen has a better layout.

Improve: show name partner in selectlist (in configuration).

Improve: show channel-type in selectlist and routes (in configuration). I find this very convenient.

Improve: better sorting in configuration for in django 1.4, eg for routes, codelists, trasnlations.

References in a configuration are better guarded. Eg: when deleting partner that is used in partner-specific translation: user is warned, has to confirm.

Improve: removed charset from channel. This was not needed.

Improve: selections in screens not only for partners but also for partner groups

Fix: download an edi-file via filer could give not-correct file.

Add: keep track of file-size of incoming/outgoing files.

Extended partner functionality

Add: function ‘partnerlookup’ for use in mapping scripts to look-up/convert partner related functions.

Add: extra partner fields: address, user-defined fields.

Add: send a message to any bots partner from mapping script (use partners as the email address book).

Improve: multiple email addresses in cc field.

Confirmations

Fix: bug in confirmation logic for frompartner/topartner (I had them reversed…).

Better CONTRL-message.

Option to run user exit either to generate CONTRL message or change the generated CONTRL message.

messagetype (for incoming) is now similar to other uses of message-type; edi-type is removed (is not needed)

Fix: better indexing for reference/botskey. This improves performance for confirmations.

Plugins

Always make configuration backup when reading new plugin.

Dropped the date-indication for files that are overwritten.

Index file in plugin always starts with type of plugin and layout is sorted/predictable.

Fix for performance problem when generating plugin for big plugins.

Fixed bug for relations in database. Bug only occurs when reading ‘hand-changed’-plugins.

Better handling of ‘database is locked’/crash recovery

First of all: use the jobqueue-server for more complicated scheduling (prevents running multiple engine)

Different way of detecting another instance of engine is running via locking of port.

If bots-engine finds no other engine is running, but the database is locked this indicates the previous run was ended unexpectedly (eg computer crash).

In this case bots will do an automatic ‘crash recovery’. A warning is still given in logs or via email. Only ONE email is send.

Technical

An 64-bits installer for windows is available.

Runs with django 1.4 now ; dropped support for django 1.1, 1.2

Database has changed. A script is added to upgrade the database.

Other changes

Add: startup script for bots-webserver using apache. Multiple bots-environments can run over one apache server.

Add: user exit for cleanup.

Improve: use external file name in archive.

Add: archive as zip-files.

Fix: port was not used in initializing PostgreSQL.

Less statuses in processing (simpler, faster).

Improve: use unlimited text fields in database for errortext and persist.

Improve: if message in incoming edi file has error in mapping script and/or writing outgoing file: process rest of edi file (there is a compatibility option in bots.ini).

Reworked errors for edi files (parsing, generating). All errors have numbers now (for referencing).

Always read incoming files sorted by name. Reason: predictability. Formerly the read order was not predicable.

Fix: better handling of import errors for user scripts. This could lead to confusing errors/situations.

Add: access to whole envelope in mapping.

Fix: nr of messages was not used correctly when writing multiple UNH in one mapping script.

Fix: bug for numerical fields with more than 4 decimal positions.

Add: counter(s) per bots-run in mapping script; eg useful for UNH/ST counter per interchange.

Fix: enveloping for edifact with keca-character-set.

Add: concatenate function for usage in mapping scripts.

Improve: updated grammar-handling to allow for UNS-UNS construction.

Improve: set explicit namespace when generating xml.

Add: determine translation via user scripting.

Add: user scripts can easy detect what type of run is done via routedict[‘command’]

Dropped: intercommit connector.