new openIMIS batch platform

Description

Today openIMIS doesn't have batch processing (scheduled, ordered,...) ability.
Legacy openIMIS relies on Windows services/scheduler.

To port batches into openIMIS, the architecture was initially foreseeing AirFlow.
One year later, the choice has to be re-considered (confirmed?).
Amongs alternatives:

  • new django module (in django 3)?

  • OpenHIM

 

Expected features:

  • scheduling of processing (cron-based + activation/de-activation dates)

  • processing = service call in a (backend) openIMIS module (note: we can expose a ‘scheduled-trigger’ generic API endpoint if scheduler is to be managed ‘outside’ openIMIS tech stack)

  • orchestrate processes (sequence of jobs, with conditional branching,…)

  • ability to transmit (and update from scheduler console) potentially sensitive parameters (like FTP credentials,…)

  • monitor/alert in case of error (and ability to restart)

  • dashboard overview (next runs,…)

  • audit/view logs of executed processus (troubleshooting)

Nice to have:

  • combine openIMIS services with other processes (like download/upload files on S3 or FTP server or …)

Activity

Show:
Xavier Gillmann
June 5, 2020, 12:59 PM

Although OpenHIM has orchestration capabilities:

  • there is no scheduling (yet?)… which makes it not so well suited for batch scheduling

  • it doesn’t look like well suited for ‘long running transactions’ either

Xavier Gillmann
June 12, 2020, 12:17 PM

Looking at what OpenMRS has done: they made their own scheduler inside OpenMRS:

Assignee

Unassigned

Reporter

Xavier Gillmann

Labels

None

Priority

Medium
Configure