Whole Brain Mapping with X-rays

Whole Brain Mapping with X-rays

Yeukuang Hwu 胡宇光

Institute of Physics, Academia Sinica

台大物理 27/10/2020

Collaborators: A true international effort!

• ^Taiwan

• Keng S. Liang, Ting-Kuo Lee, Maw-Kun Wu, Shih-Chang Lee, En-Te Hwu, Ming-Li Chu, Chih-Hsiun Lin (Academia Sinica): phase retrieval, reconstruction, magnetic nanoparticles, microfluidity

• Ann-Shyn Chiang, Chia-Wei Li (Life Science, Tsing Hua U.): cell biology, shell fish, fossil, fire fly

• Hong-Ming Lin (Tatung U): electron chemistry, battery

• Yu-Tai Ching (NCTU), David Chien (UCS-St. Marcos): Image Processing, reconstruction

• Chung-Shi Yang, Kelvin K. C. Tsai (NHRI): nanomedicine, microangiogenesis

• Ann Chen, Maria Ka, Dueng-Yuan Hong (TSGH): CKD vessel imaging

• Y. F. Hu, S. K. Tsai (TTY Pharmaceutics): (drug delivery, pharmacokinetics)

• Y. C. Yang, Hong-I Yeh, Yu-Jen Chen (Mackay Memorial Hospital): (artery disease, tumor development)

• Japan: T. Ishikawa, Y. Kohmura, K. Sawada, Y. Joti (RIKEN/SPring-8 Center)

• Korea: Jung-Ho Je (POSTECH), Doyoung Noh (GIST), Jun Lim, Jae-Hong Lim (PAL)

• China: Jun Hu, Chunhai Fan, Lihua Wang, Xiaoqing Cai, Chichao Zhang, Ying Zhu

• Singapore: Eng Soon Tok, Alvin Teo, Chian-Ming Low, Gan-Moog Chow, H. O. Moser (NUS)—drug delivery, polymer blend, electrochemistry, neurobiology.

• France: Cyril Petibois, Sophie Javerzat, Michele Moenner (U. Bordeaux, brain tumor angiogenesis), Patrick Soukiassian

• USA: Yong S. Chu, Wah-Keat Lee, Qun Shen, B. Lai, (APS), Wen-An Chiou (U. Maryland), John Boeckl (US Air Force Lab)–TXM

• Switzerland:

• G. Margaritondo (EPFL)

• Rolf Gruetter (EPFL)

NanoX Team (Current Members)

• X-ray microscopy:

• Hsiang-Hsin Chen (陳翔欣), Shun-Min Yang (楊舜閔), Tsung-Tse Lee (李宗 澤), Cheng-Huan Hsu (徐晟桓), Ching-Yu Chiou(邱鏡宇)

• X-ray Nanosynthesis:

• Ming-Tsang Lee (李旻倉)

• Nanofabrication of X-ray optics:

• Mei-Chun Chen (陳玫君), Yu-Ting Jian (簡郁庭)

• Bioimaging:

• Shun-Min Yang (楊舜閔), Hsiang-Hsin Chen (陳翔欣), Yi-Yun Chen (陳怡云), Chia-Ru Chang (張家如), Ya-Sian Wang (王雅嫻), Shiou-Jin Chiou (邱繡謹), Cheng Jyun Yang (楊程鈞)

•

Visitors: Cyril Petibois, Pei-Feng Chen, Keng S. Liang, Benoit Recur

Innovation in physics and instrumentation has opened new eras of biology and medicine

• ^Microscopy

• Spectroscopy

• X-rays scattering

• Mass spectrometry

• Nuclear Medicine

• NMR, SPECT, PET, CT, ultrasound, OCT

• ^…

And, More Recently..

•

Laser confocal scanning microscopy + fluorescent protein

•

Protein crystallography

•

Cryo-electron microscopy

The breakthroughs in x-ray imaging

•

Phase contrast

•

Nano-resolution

•

Elemental contrast

•

High speed for 3D imaging

Historical increase of the x-ray source brightness (in standard units)

Taiwan Photon Source 3^rd Gen. SR

sources

It took decades to obtain the first x-ray free

electron lasers, but the results are fantastic:

The second breakthrough: nanoscale

resolution

Transmission x-ray microscopy with

Fresnel zone plate optics

20/450 nm & 40/950 nm Au zone

plates

Star Pattern Image

Finest line width: 20nm Au thickness: 300nm Pixel size: 2.51nm FOV: 5.14um No binning 8 sec exposure 60 frames averaged 4 pieces of diffuser

Zone Plate applied (ZP m2-6):

8.000keV

20nm outermost zone width 50um diameter

400nm thickness

SEM

Enlarge

Finest line width = 25nm 20 nm 15 nm

Si/WSi₂

Si/W multilayer test patterns

H. R. Wu, et al. Nanoresolution Radiology of Neurons, J. Phys. D 45, 242001 (2012).

HeLa cells with AuNPs on culture dish

Element mapping with nanoresolution

J Electrochem Soc 157, B783 (2010) Figure 2

• Ni–YSZ (yittria-

stabilized zirconia) anode

• Ni K-edge: 8.333 keV

• Spatial resolution: 38.5 nm

• Box size: 5 µm

16 eV below Ni K-edge 24 eV above Ni K-edge

Historical increase of the x-ray source

brightness (in standard units)

Taiwan Photon Source 3^rd Gen. SR

sources

Phase contrast: the first breakthrough

The XFEL: Toward the ultimate x-ray

source

“Detect and Destroy”

In collaboration with Prof. Yoshinori Nishino of Hokkaido University and SACLA

Blank liposome ((NH

₄

)

₂

SO

₄

inside)

4

Doxorubicin loaded liposome

Coherent diffraction imaging (XCDI)

A single 40 nm AuNP Two 10 nm AuNP

A single 200 nm liposome

Tumor Induced Angiogenesis

Cyril Petibois et al. (U. Bordeaux), DY Hong et al. (TSGH),

Large tissue, high resolution and speed

Whole-brain blood vessels in mouse

Res. < 0.5 mm Size > 400 mm³ Time < 10 min

X-ray imaging of angiogenic

microvasculature of glioblastoma

Solid glioma tumors

Diffused glioma tumors

TXM Images of a microvessel

What do we expect to learn?

• Small differences in microvasculature between different models

• Effect of treatment

• Inhomogeneity in microvasculature with respect to tumor

• Microvasculature with respect to the growth and metastasis of tumor

• Drug development and usage

There are more problems in brains with neurons (dementia) than vessels (cancer)

• Can X-rays image neurons?

• Can X-rays image blood vessels at the same time?

• Can X-rays study neuro-vascular interaction?

• On dead specimens, of course.

WHY BRAIN?

Mapping the Brain:

an Historical Mission for Science and Technology

• A fundamental research objective

• An effective way to understand and cure brain diseases

• Potential for a broad social impact

• Important technological byproducts:

- Advanced imaging technologies - New computational strategies

- Artificial intelligence

THE CHALLENGES:

…we must do better!

An effort to image the neural network

of a whole human brain with proven

performance of synchrotron x-ray

tomography and a collective effort

SYNAPSE: an international

partnership with a milestone research mission

First objective:

mapping the neuron

connections of an

entire human brain

by 2023

Mapping the Brain:

an Historical Mission for Science and Technology

• A fundamental research objective

• An effective way to understand and cure brain diseases

• Potential for a broad social impact

• Important technological byproducts:

- Advanced imaging technologies

- New computational strategies

- Artificial intelligence

IT TAKES DECADES FOR

MAPPING ONE HUMAN BRAIN WITH THE PRESENT X-RAY

TOMOGRAPHY PERFORMANCES!

•

Parallel image taking at multiple facilities

•

Automated specimen sectioning

•

Automated specimen mounting and alignment

•

High-speed data transfer and sharing for all partners

•

New, efficient strategies for reconstruction

SYNAPSE is the solution

SYNAPSE:

core partners with top-level

synchrotron facilities

NUS/SSLS

AS

RIKEN/Spring8 SARI/SSRF POSTECH/PAL

In addition to x-ray microtomography facilities, the core partners bring with them a very powerful

extended coalition

CONCENTRATE ON 3D IMAGING:

New radiology techniques combined with other advanced microscopies

• For overall connection mapping: phase-contrast microtomography (0.3 µm resolution in all 3D directions)

• To explore in detail synapses and neuron connections: nanotomography with <10 nm resolution

• To obtain detailed 3D maps of special regions: electron microscopy/continuous sectioning, optical super-resolution microscopy, with nm resolution

• To analyze the whole drosophila brain and large parts of the mouse brain

obtaining functional information: Confocal (or Light Sheet) + FocusClear imaging

• For single-molecule mapping of drosophila and mouse brains: Super-resolution microscopy with FocusClear

• high resolution functional imaging: Raman, IR spectromicroscopy

• Fusing-in low resolution functional imaging, i.e. fMRI

• …plus other tasks using other frontline techniques

Pohang Light Source II 7C

SPring-8 BL32B2

Taiwan Photon Source BL2A

Singapore Synchrotron Light Source PCXT

Shanghai Synchrotron Radiation Facility BL 09

SYNAPSE:

core partners with top-level

synchrotron facilities

NUS/SSLS

AS

RIKEN/Spring8 SARI/SSRF POSTECH/PAL

In addition to x-ray microtomography facilities, the core partners bring with them a very powerful

extended coalition

Concentrate on 3D imaging: new radiology techniques combined

with other advanced microscopies

• For overall connection mapping: phase-contrast

microtomography (0.3 µm resolution in all 3D directions)

• To explore in detail synapses and neuron connections:

nanotomography with <10 nm resolution

• To obtain detailed 3D maps of special regions: cryo-electron microscopy with nm resolution

• To analyze the whole drosophila brain and large parts of the mouse brain obtaining functional information: Confocal

FocusClear imaging

• For single-molecule mapping of drosophila and mouse brains:

Super-resolution microscopy with FocusClear

• …plus other tasks using other frontline techniques

Lightsheet Localization super-resolution Microscopy

Communications Biology, 2, 177 (2019) 2.7 × 10⁴ µm³/s

250 image averaging, 15 min 75 nm resolution

Lattice lightsheet microscopy w/ tissue clearing

Brain size

Volume (mm³)

Number of Neurons

brain/body Ratio

Number of Synapse

Density of neurons

(/mm³ ) drosophila 2x10^-2 2.5x10⁵

(1.3x10⁵) 1.25x10⁷

mouse 450 7.1x10⁷ 1:40 10¹¹ 1.57x10⁵

marmoset 6.4x10⁸

human 1.2x10⁶ 8.6x10¹⁰ 1:50 10^14-15 7.2x10⁴ Source of information: Wikipedia

Big Brain, Big Data

Nature

http://www.nature.com/nature/journal/v541/n7638/full/541559a.html#references

Nature blogs http://blogs.nature.com/naturejobs/2017/01/26/new-neuroscience-tools- for-team-science-in-big-data-era/

Scientific American https://www.scientificamerican.com/article/neuroscience-big- brain-big-data/

Small Brain, Big Data

Fluorescence confocal microscopy on connectome mapping

• Drosophila Brain: ~135,000 neurons

• Resolution：500 nm

• Data amount： ~10 TB

• Time to complete： ~20 year/10 microscopes

• Human brain: 85 billion neurons

• Time to complete： ~17,000,000 年

500 mm × 300 mm × 200 mm

3D X-ray data

• Single fly brain: ~100GB

• Single mouse brain: ~360TB

• Complete the map with 100 brains: ~36 PetaByte (PB)!

• Human/mouse brain volume: 2500 !

1 m 100 10 1 mm 100 10 1 µ m 100

SuperRes

*+

Pr ob e D ep th

CT

SuperRes

µCT

TEM TEM+

SuperRes*

10 1 100 10 1 100 10 1

mm µ m nn

- - - - - - - -

Phase Contrast

CT

PC – phase contrast

* – Tissue clearing

+ – Continuous sectioning

Resolution

10

⁰

10

^-1

10

^-2

10

^-3

10

^-4

10

^-5

10

^-3

10

^-2

10

^-1

10

⁰

10

¹

10

²

10

³

Resolution (mm)

Specimen size (mm)

400 nm Mouse brain 20 nm

Mouse brain

Mouse brain Mouse brain

X-ray imaging — From Animals, Organs, Neurons to Synapses

< 0.5 mm resolution (mCT) 1μm ^0.5μm

ccelerated -ray

bservation for

eurons

AXON Implementation: TLS, APS, PLS-II, TPS..

Whole brain imaging without sectioning

Merging Different Brains

DAL neurons

enzmet/gold toning/gold enhance

Large tissue, high resolution and speed

Whole-brain blood vessels in mouse

Res. < 0.5 mm Size > 400 mm³ Time < 10 min

AXON on Mouse Brains

X-ray imaging of a mouse Purkinje neuron with 20 nm resolution

X-ray imaging of a whole mouse with 0.5 µ m resolution.

Tomography image shows blood vessels (golden) and neurons (green)

2 mm

200 µm 100 µm

20 µm

Recent progress – whole mouse brain imaging

Mouse brain slice, 400 µm thick

Mouse Perkinji cell in a thick tissue slice

Same specimen, different resolution

10 µm

Same specimen, different resolution

20x, NA = 0.75, resolution ~0.5 µm 40x, NA = 0.95, resolution ~0.3 µm

And speed..

Mouse brain, (0.8

^µ

m)

³

Technology impacts — Beyond connectome

•

X-ray imaging: phase contrast radiology and tomography, nanotomography

•

Super-resolution imaging

•

3D continuous sectioning

•

3D spectromicroscopy

•

3D pathology

•

Big Data analysis

•

Neuromorphic computing & AI

Brain inspired computation, neurosynaptic

computing, neuromorphism and artificial brains

•

Power consumption

• 86 billion neurons, ~1000 synapses per neuron

• >trillion transistor, >10 GW

• Human brain 20 W!

•

Simulate Brain from neuron (HBP)

•

Neurosynaptic computer using neuromorphic chips

•

Artificial Intelligence

Flybrain Simulators

•

130,000 neurons

•

Build with real connectome map

•

With input and output terminals for simulations

•

Functional information included in the simulations

•

Understand how fruit fly brain function (with a super computer)

•

Better algorithm and computer architecture for AI

•

Simulate and understand brain diseases, and the cure

Next steps:

• Sub-10 nm imaging

• Multi-modality imaging

• X-ray molecular/functional imaging

• X-FEL applications?

• Laser wavefront accelerators?

Toward Sub-10 nm

•

Must be an international effort

•

Mobilization of resources and user base

•

Focus on novel configurations and applications

•

A consortium of beamlines at NSRRC, SPring-8, PLS-II,

BSRF (Beijing), SSRF (Shanghai), SSLS, NSLS-II (Singapore)

…

Objectives – to map in 3 years:

• One whole human brain

• 200 whole mouse brains

• Develop the microtomography technique for >100x improvement in speed to enable complete human connectome mapping

Whole brain w/ 1 m m resolution

SYNAPSE

2023: groundbreaking progress in human brain knowledge brought by the Asia-Pacific countries

Thank you!!

Go fast and go far with friends!

(from China, France, Germany, Japan, Korea, Singapore, Switzerland, US, ..)

Thanks for the funding support from:

• Ministry of Science & Technology

• National Program for Nanoscience and Nanotechnology

• Academia Sinica Thematic Projects

• ANR-MOST, INSERM-MOST

• RIKEN-MOST

• ^USAF-MOST

d

The SYNAPSE strategy:

data acquisition

and management

Data size:

• Our experience from the tomography of a 0.5³ mm³ volume containing one drosophila brain:

Size of each raw projection image: 32 MB

Tomography set (1000 projection images): 32 GB

Reconstructed images for volume rendering: ≈128 GB

• Scaling to one mouse brain: [450/(0.5)³] x 128 GigaByte ≈ 460 TB

• Limited staining rate (~5%) 100 mouse brains required for complete map: ~46 PB

• Scaling to one human brain (2,700 times the mouse volume): 460 TB x 2,700 ≈ 1,240 PB

Total image-taking times at current speed:

• ~5 min for a (0.5)³ mm³ volume:

For one mouse brain: (450/0.5³) x 5 min

≈ 1.8 x 10⁴ minutes ≈ 12.5 days

For 100 mouse brains: 1250 days ≈ 3.4 years

• ~15 min for a 1 mm³ volume (with a 4K x 4K detector):

For one human brain: 1.2 x 10⁶ x 15 min

≈ 1.8 x 10⁷ minutes ≈ 34 years

For 100 human brains: 3,400 years

We must increase the throughput.

Without new technologies, by:

• Increasing the number of synchrotron beamlines used in parallel, coordinated within SYNAPSE

• Reducing the number of projection images for each tomography, from 1000 to 100.

With new algorithms

With artificial intelligence

It becomes thus realistic to map one human brain in 4 years – the first

SYNAPSE objective

Further throughput increases

possible with new technologies:

• Staining rate increase from 5% to 30% (demonstrated but not optimized). The number of brains for a

complete map could decrease to <20.

• Faster imaging: Higher x-ray flux (10x), higher detector sensitivity (5x) and better algorithms (to further

reduce the number of projections) – overall, the image acquisition speed could be increased by two orders of magnitude.

The image taking time for full human

brain mapping could be reduced to a

few years

Image taking is not all: data managing is a big challenge!

• Tomography reconstruction for drosophila: currently, 10 times slower than image acquisition

• To perform one automatic segmentation and tracing within a comparable time, it takes now the largest computing facility in Taiwan, NCHC

• Commercial graphic workstations cannot handle the

rendering and visualization of the reconstructed data set

• Another big challenge: creating and managing the database

• Further complication: adding functional information to the map

Crucial data handling tasks:

• Image morphing, warping and fusion

• Correlation identification

• Tracing, segmentation

• Database architecture and optimization

• Database access

• Visualization

Adding functional information:

• Using the connectome as the skeleton/grid.

• Adding local high-resolution 3D information on synapse connections with:

X-ray tomography with 10 nm resolution.

Super-resolution 3D fluorescence microscopy (with tissue clearing)

Super-resolution vibrational spectromicroscopy (Nano-IR and Nano-Raman)

Continuous sectioning with electron and optical microscopy

• Chemical information in large region:

Functional MRI, PET & SPECT imaging (with 100 µm resolution)

3D IR (with <3 µm resolution)

• Simulations based on the structural and functional information

Other technological developments targeted by SYNAPSE:

• Algorithms and related software

• Computer hardware: machine-learning graphic station

• Automation in specimen handling

• Automation in image acquisition

• Artificial Intelligence for automation of image processing

• Image taking hardware: specimen manipulation, detectors and high-brightness X-ray sources

Why us?

We are uniquely qualified!

• We pioneered the special techniques required for the SYNAPSE initiative, including:

X-ray phase-contrast tomography World-record x-ray microscopy

Deep tissue super-resolution microscopy

• We have a solid record of relevant previous

accomplishments and all the required know-how for the core and support techniques

• The new partnership is built on long-standing and very successful collaborations

Sizes of Brains:

Volume (mm³)

Number of neurons

Brain/

body ratio

Number of synapses

Density of neurons

(mm^-3 ) drosophil

a 2x10^-2 2.5x10⁵

(1.3x10⁵) 1.25x10⁷

mouse 450 7.1x10⁷ 1:40 10¹¹ 1.57x10⁵

marmoset 6.4x10⁸

human 1.2x10⁶ 8.6x10¹⁰ 1:50 10¹⁴-10¹⁵ 7.2x10⁴

Physics opens the new era of

biology and medicine, again and again.

• Microscopy

• Spectroscopy

• X-rays

• Protein crystallography

• NMR, SPECT, PET, CT, ultrasound, ..

• Mass spectrometry

• Nuclear Medicine

• Digital Camera

• …

540 million year old embryo fossil

Concentrate on imaging

Structure and Network:

• X-ray phase contrast microtomography, 0.3 µm resolution in all 3D directions: to provide the wiring map

• X-ray nanotomography: 10 nm to provide the structure and

location information of targeted synapse and neuron connections

• Confocal + FocusClear^® : Full drosophila brain and regions of mouse brain with functional information

• Cryo-EM: small region 3D map down to nm resolution.

• Super-Res + FocusClear^® : Full drosophila brain and small region of mouse brain but concentrate on single molecule detection

Stereographic 3D

Volumetric view

ONE STEP FURTHER…

Chian Ming Low, Eng Soon Tok, Alvin Teo, Tin-wee Tan (National University of Singapore)

Zhong Zhong & Hua Hau (中中與华华)

CA 1 CA

3 DG Sub

EC CA

2 Hippocampal

Formation

DG – Dentate Gyrus CA – Cornu Ammonis Sub – Subiculum EC – Entorhinal Cortex

Note: hippocampal formation consists of these regions – DG, CA3, CA2, CA1, Subiculum &

Entorhinal

cortex…you can also call it hippocampus (a more ‘general’ term)

Monkey Hippocampus

(Macaca Fascicularis)

CA 1

CA1 Rat Monkey

# Divisions 1-2 7

# Cell Thick 5 10-15

Boundary between PCL -SR

Clear Less clear

Boundary between CA1- Subiculum

Clear Less clear

Entorhinal Cortex – laminar organization

Less clear (e.g.

Layers V- VI) 2 divisions

Clear (e.g. Layers

V-VI) 7 divisions

hf

hf – hippocampal fissure (a ‘line’ that DG is separated from CA1)

Arrowhead – neuron soma

CA 3

Dentate Gyrus

Mouse brain 2x2x20 mm

³

Besides Image Taking:

• Full biological characterization

• Specimen preparation: novel labeling procedures

• Coordinated use of different imaging facilities at partner countries, to reach the required

throughput

• Standardization and mutual validation for all participating facilities

• Advanced image processing with new computer techniques

• Full data access for all partners and eventually to public

The core partners bring with them a very powerful extended coalition:

• Through NUS/SSLS: research teams from university (NUS, NTU, NYP), public institutions (NCSS)

• Through SARI/SSRF: research teams from universities (Shanghai Jiao Tong U., Shanghaitech U.), public institutions (SINAP, IoN)

• Through POSTECH/PAL: research teams from universities (POSTECH, KAIST), medical institutions (ASAN, Samsung, SNU, Yonsei)

• Through RIKEN/Spring8: public institutions (RIKEN QBiC-BDR, BSI)

• Through ANSTO: research teams from universities (U. Sydney, U.

Wollongong), public institutions (Australian Synchrotron, Brain & Mind Center)

• Through AS: research teams from universities (NTU, NTHU, NCTU, NCKU), public institutions (ITRI, NHRI, NHPC), medical institutions (NTUH, Chang Gun H, Mackay H, TSGH, VGH, Chinese Medical UH, Taipei, Medical UH, Kaohsung MU, Cheng Kung UH) and from the private sector (TTY Pharma, Delta Electronics)