A. Technology – IT Processes
1. Is your Software Development Life Cycle methodology documented covering traditional SDLC components including, but not limited to, quality assurance, coding standards, change control?
As we are a company driven by quantitative modelling, we operate in 4 phases, quite similar to the SLDC approach:
1. R&D: A research step, during which key concepts are articulated and refined.
2. BUILD: We then articulate the building blocks, which draw from different methodological approaches (e.g. data cleaning or data smoothing). From this, we are able to draw a flowchart of the various transformation stages of the data.
Each of the previous steps is coded on its own and tested accordingly on data to make sure it delivers appropriate results. It is the time during which we produce technical papers which include a clear explanation of the quantitative approach and the key results obtained.
3. DELIVERY: The version of the code, which was up to this step driven by research is transformed into a properly written according to professional quality standards, optimized and documented production code. The results of this latter code are compared with those of the research code for validation purposes.
4. DEPLOYMENT: The version of the code is loaded on the production servers and is also stored on SVN (Apache) in order to be able to keep track of all past versions. There is a phase during which the code is run in a testing mode every day, in order to make sure it remains robust, even when the data is incomplete or mistaken.
Additional Comments
- There is no formal document on this process, but it is real and effective, as this corresponds to the ongoing process on which the firm and people are organised. This organisation is related to this separation of duties and competencies.
- The CEO is having oversight on this proxy SDLC process, having different teams operating at different steps.
- Up to phase 2, the code is being developed and tested purely by the R&D team. In phase 3 (Delivery), the code is then revisited and tested by the production team, with the support of the R&D team.
- It is an internal Production Committee decision which triggers the deployment in the production environment, with testing being performed independently by the production team, which informs the committee about its comfort with respect to the methodology itself, the robustness of the code and the management of the data.
- The Production Committee drives the change control too.
2. Please describe your software change management process. What processes are in place to validate changes implemented into the system (Upgrades, enhancements, Release, etc…)?
Because we have adopted the philosophy of running stable strategies, our goal is to be as minimalist as possible in terms code changes.
The main reasons why there would be changes is to integrate partial upgrades on a low frequency basis (yearly?).
Continuous Integration Process: Processes are however in place to make sure that version releases are properly and automatically implemented both at the pre-prod and the prod levels.
Additional Comments
- A production version of the code cannot be changed without prior Production Committee approval. This committee includes R&D, IT and Production people and is chaired by the CEO of the firm. There needs to be full consensus within the committee in order to have the change integrated in production.
3. How is the logical separation of duties implemented within the technology team?
In terms of separation of duties, we have three functions in the firm:
- IT and database organization (building, maintenance, development)
- (Non IT) Research and preproduction (from ideation to the delivery of the code at the level of the pre-prod level)
- (Non IT) Production (From preproduction to daily production)
These three functions are relatively independent from each other, however, there are interfaces and dialogue among these three teams.
Additional Comments
- The IT team focuses on both the Data environment and the stability of the production environment. Its work is to set up a robust automated environment, and never to work through ad hoc human interventions related to data loads or updates. It makes sure the right versions of the languages and packages are used. It makes sure that the connection between the different IT constituents (database / servers) is safe and sufficiently duplicated. It drives the evolution of the IT architecture overall and the choice of the IT tools used.
- The R&D team does not have production access.
- The production team is independent from the IT team. It has strong coding and operative skills but has no direct say on the IT architecture. It has front-end access to the database, but not back-end access.
4. Are procedures and controls for the separation of duties documented?
These elements of separation of duties are based on a separation of competencies. They occur naturally, given the size of the firm. Each team has a separate focus and delivers outcomes related to its competencies. Attention to the focus of these outcomes constitutes a good way to insure the separation of duties.
- Research and preprod produce the documentation explaining the methodology.
- Production produces the readme files and documents the production code on a line by line basis
- IT / database sets up the production environment and documents it. The explanatory graphs embedded in this document are produced by IT.
In terms of controls, controls are more related to the existing completeness of the process (no gap) than anything else. We complement the effort based on a review of the complete execution process and we therefore monitor the ongoing stability of the daily runs and try to address in a joined manner issues which would not be related to data issues, until the algos are fully stabilized.
B. Methodology, Operations and Data
1.1 Backup Management- What types of controls are implemented to ensure data files and libraries are backed up on a periodic basis to ensure availability of data files/system?
Backups are stored every day in the different locations. It is based on automated routines.
The focus we emphasise here is related to the quality of what is being stored.
Quality control on stored data: we check the quality of the process at three stages. First the input file has to respect a strict format. If it is not the case, the process cannot run and we have to revisit the input files. Moreover, we monitor the number of daily input (number of options) to check for outliers. Second, during the computation process itself, we save all intermediary files to be able to confirm the relevance of all of the different intermediary steps.
Replicability of external packages: In terms of full transparency of what is going on, the open architecture libraries we use are very standard and their outcome can be checked manually.
1.2 Overview of processes that are done outside the calculation system and who performs these functions. What are the QA Checks for these processes?
Nothing is done outside of the algorithms.
1.3 Incident Management- how are they handled, how often are they reviewed, how are clients notified?
Generally, incident management is related to unclean data jamming the system. There are sufficient alarm bells to mitigate the issue before the output has had to go out.
Again, the fact that the issuance is on a daily frequency basis gives time to adjust.
1.4 Aside from the processes described in A1.2, describe your change control process related to methodology / modelling / algorithms / artificial intelligence. What processes are in place to identify the need for calculation / process changes (including upgrades, enhancements, new releases)?
On a general basis, there is no need to change the process.