Cell lineage data

Just links:

Relevant meeting notes

Whitepaper

Github issue

Updates, Persistence, and Berkeley DB store

For persisting updates to data through PyOpenWorm, we need some stable store. For now I'm looking at how we can save the updates locally. From there we would have mechanisms to write back local changes to the shared repository, possibly with staging for review by other members, like a Github pull request.

Stephen L. has suggested the integration of git with the database system, in particular through presentation of changes through diffs in SQL statements. With the scheme above this could involve dumping to SQL statements on commit to be legible in the diff. To extract a meaningful diff from from these dumps, we'd need to sort the values first so that no-change would mean matching subsequences for diff to find. Just from a cursory assessment, I don't think this too difficult, and we can already see that it would be valuable for exposing the changes in a human readable format and for integrating into existing Github-based workflows.

I have started experimenting with Berkeley DB using the "Sleepycat" store for RDFlib. I've done a little profiling and found that performing queries is about 10 times slower (.02 ms for in-memory v. .17ms for Sleepycat). What's interesting is that query times fall precipitously after a number of repeated queries, suggesting some caching. I'd like to read up on Berkeley DB to get a fuller picture. I'll also be looking at times for writes back to the database.

Coming from the data meeting held today, I looked further into the movement validation repo.

I will review later today.

Data sources and collection

This is a list of data sources I've identified as actually being used in projects, starting from here. The goal is to understand the different types of data source to know how we can access them and what input/output formats should be targeted:

• MySQL database: mysql_31129_celegans at my01.winhost.com
• Ion Channel Spreadsheet includes citations from Pubmed showing relationships of ion channels to genes.
• Wormbase, generally. There seems to be some query functionality, but it uses its own 'special' query language -- seems not very complicated; pattern based.
• Time series data?
• Movement validation data pointing to the ftp from Laura Grundy
• Comments on some of the spreadsheet cells -- not exported from google drive.

I've also found a lot of scripts for extracting data from different formats:

• Extracting from the Laura Grundy ftp
• From spreadsheets(CElegansNeuroML project)

I'm still looking around. I'll make a new post with any updates.

neuroConstruct and CElegansNeuroML

I am thinking that a good model to start working from is the CElegansNeuroML project. For right now I'm going to browse around to figure out how it all fits together. One of the problems I have is that my NeuroConstruct build can't open the 3D display. I'm using Xephyr in XMonad so I can get a non-tiling WM in here (Java AWT programs don't play well with Xmonad it seems). Mayhaps I can use it in a different session.

For right now, I'm going to explore some more data sources, start looking at where there are shared entities. Regardless of what we end up doing, we'll need to handle this somehow.

I'm thinking of trying out the RDFlib store API

PyOpenWorm and NeuroML

I'm trying to make a neuroml file from the generated rdf database in PyOpenWorm. Trying to make sense of the schema. Everything is related through integer keys and I haven't found a human description yet.

Update 1:
All of the methods read through to the a RDF graph built up in the network module. However, the neuron is not checked for existence in the databse on creation. I'm going to add a primary access method with the check.

Update 2:
Thinking about the structure of these modules. I'm guessing a config file would be good for moving the configurations around. PyOpenWorm should be a library that can be used to make the wrappers, so it shouldn't be tied to any specific database. Sounds like a job for Dependency Inversion (i.e. parametrizing the network class).

Update 3:
Learning libneuorml from Padraig G's examples in the github repo. Right now, I'm leaning towards separating the file-type specific components from PyOpenWorm proper. Eventually I want to stop opening up the library files (closely tied to the schema) and spend my time with the file-specific converters.