Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Sure, I'm trying to take the idea of merging continuous integration with workflow/pipelines. It's all stuff that I learned at Google and is non-proprietary. The idea is have presubmit checks that invoke a full instance of a complex pipeline, but on canned (synthetic or pseudoanonymized or somehow not directly connected to the prod system) data, as an integration test. This catches many errors that would be hard to debug later in a prod workfflow.

In a sense, I see software testing/big web data and modern large scale data processing in science as a continuum and I want to bring the practices from the big web data and testing fields to bear on science pipelines.



Apart from a shift in mental attitude, is it primarily about getting a dataset for the integration test?


also making sure the testing is hermetic (not breaking prod) and all the components are actually reproducible.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: