SiPAM: Silicon Photonic Accelerated Memory Pooling Architecture