Index of /SISTA/aerts

GCC

Current situation GCC 1.0

Roche 454

Current

cluster

Per run:

~ 1 Mio reads

~ 2Gb raw data

New sequencer: 1000x increase

1.1TB / run (200Gbp)

~1000 Mio reads

8 days run!

Basic analysis of 1 full run

< 1 week on 3 nodes with 48Gb RAM and 8 CPU cores each (and needs 7TB space)

Full capacity sequencing = full capacity 24 cpu cores

Meta-analyses & post-analyses

•

_{Several fold higher needs than basic run analyses}

•

_{Integrate multiple runs (e.g,. patient versus controls,}

families, etc)

•

_{Integrate with previous data}

•

_{Integrate with publicly available data}

–

_{RNA-Seq + gene expression data from GEO}

•

_{Integrate with other data sources}

–

_{DNA-Seq + RNA-Seq + Methyl-Seq}

•

_{Integrate with genome browsers}

–

_{Galaxy, UCSC, Ensembl}

•

_{Make analysis pipelines available to users as a service}

•

_{Custom analyses as a service or in collaboration}

Ideal computing setup

High Performance

Computing (HPC)

UZ-GBIOMED-VSC

- Additional RAM (32Gb or 48 Gb per node) - Additional storage? DAS or NAS? Dell, NetApp? DAS or NAS? Dell, NetApp? Open-MPI SGE Distributed computing Torque/PBS Distributed computing Flexible computing ~ 100 cpu 6Gb RAM/core NetApp +DDN Storage NetApp +DDN Storage - Servers - Storage - Switches Software: - Academic tools - CLCBio? Software:

- CASAVA (parall. by user) - Academic: bowtie, bwa, … - CLCBio? UZ-Patient data Software: - CASAVA - CLCBio - Roche

- Computing (0,5 EUR / cpu-hour) - Storage (750-1500 EUR / TB)

VSC

gbiomed

To be discussed

• _{How can HiSEQ2000 choose between UZ and KULeuven network to send run data} to storage?

– _1Gb

– _{350 Gb / run compressed}

• _{Where to store data after secondary analysis?}

– _{Cheap storage} – _{External HDD} – _tape

• _{Who does what?}

– _{Jeroen / Jan for UZ?}

– _{Stein / Gert / Raf for Biomed?}

• _{Can we already buy additional RAM for UZ cluster?}

• _{Can we connect gbiomed servers directly to UZ storage?}

– _{What are the requirements?}

• _{Estimate load over 3 levels}

– _{# users}

– _{# run}

What’s next

•

_{Decide on gbiomed hardware}

•

_{List of things needed at UZ}

•

_{Start testing CASAVA on UZ system and on VSC}

•

_{Test CLCBio on UZ system for Illumina data}

Storage

•

_{How much do we need?}

–

_{1.1 TB per run}

–

_{7 TB space during analysis}

•

_{BUT: keep only runs that are being analyzed}

–

_{~ 3 at a time?}

–

_{10-15 TB}

•

_{After analysis:}

–

_{Data delivered to client}

–

_{Data compressed and moved to offline storage}

• _{Cheap HDD array?} • _Tape?

Proposal for GCC2.0

(ideas under construction)

Patient-related data Non-patient-related data (e.g., model organisms, cell lines, …)

32C 256Gb 8C 48Gb 8C 48Gb gbiomed computing nodes Fast interconnect; high I/O bandwidth

Illumina HiSEQ2000 Roche 454 ICTS/VSC NetApp +DDN Storage ICTS/VSC NetApp +DDN Storage

VSC

!

Non-patient-related data

!

GCC2.0 features

•

_{Divide and conquer: solution at 3 levels}

– _{UZ: for UZ-patient-related data (protected)}

– _{Gbiomed: ad hoc, flexible computing for research (non-UZ-patient related data)} – _{VSC: high-performance computing (non UZ-patient related data)}

•

_{Storage (too expensive to duplicate)}

– _{VSC storage with Gbiomed access (create 10Gb fast interconnect from ICTS to gbiomed)} – _{UZ storage with Gbiomed access (create ‘open-access’ policy for non-patient related data)} – _{Gbiomed ad hoc storage (HDDs in the local servers)}

•

_Computing

– VSC for HPC

– _{Servers in UZ (patient-related data)}

– _{Servers in gbiomed (for research-related ad hoc analyses, web services, development,} software testing, …)

GCC2.0 Cost, Timing & Effort

estimates

• _{Budget from Stichting tegen Kanker}

– _{200-250 K left for computing}

• Solution for the first 3 years should be possible (excluding bioinformatics manpower)

• Budget spread between VSC-gbiomed-UZ: to be decided internally in genomics core

• VSC x%

– Storage (86.400 EUR for 32 TB; ~80 TB is needed for 25 runs per year)

– Computing time (29.594 EUR for 55.000 cpu-hours)

• Gbiomed local servers and local storage y%

• _{UZ additional storage z%}

• _{Software licenses (CLCBio) (price quote requested)}

– _{More investments needed over time (e.g., new hardware is only for 3 years)}

• _{Timing: 31 August 2010?}

• _{Estimated effort (to be discussed)}

– _VSC:

• Create 10Gb ethernet link to gbiomed (cost?)

• … mandays for startup and testing (network connections, storage, software)

• Maintenance included in price

– _{Genomics Core bioinformaticians (VRC, CME)}

• … mandays for startup and testing

– _{Gbiomed IT:}

• _{… mandays for setting-up local servers & integration with ICTS storage} • _{… FTE for maintenance of local servers}

Hurdles to overcome

•

_{1) 10Gb ethernet link between VSC and gbiomed}

– _{For non-UZ-patient related data} – _{To transfer Illumina data to VSC}

– _{To run ad hoc analyses on local gbiomed servers, connected to the VSC} storage, without the need to duplicate the storage solution and the data (too costly)

– _{An absolute requirement} – _{Currently not available}

– _{A necessary investment for future VSC-BMW interactions}

•

_{2) UZ-Patient-related data cannot be transferred to VSC storage, nor}

computed at VSC

– _{Can VSC provide a secure transfer, storage and computing environment for} UZ-data? If not, data analysis and storage for UZ-data remains in UZ.

•

_{3) Link between UZ storage and gbiomed for non-patient related data}

– _Gbiomed-UZ

– _{10Gb link is possible in principle. Perhaps during transition period (while waiting} for 10Gb link VSC-gbiomed)?

Alternatives

•

_{All-in-one solution}

•

_PSSCLabs

Bioinformatics analyses

•

_{Estimated effort from Genomics Core bioinformatician for basic}

analysis of 1 run: ~2-3 mandays

–

_{Included in service fee?}

–

_{This analysis will not be satisfactory for most projects}

•

_{Fee-based bioinformatics and data analysis service for more}

advanced analyses?

•

_{Many users have a bioinformatician in the group or already}

collaborate with bioinformaticians

•

_{Contribution in the service fee for GCC hardware & maintenance}

cost, and software licenses

•

_{Estimated effort:}

– _{Either only basic analysis services are offered: ½ FTE bioinformatics postdoc} – _{Or basic plus advanced bioinformatics services will be offered: 1 FTE}