We revisit the problem of camera relocalization under memory budget through a combination of product quantization and map compression. This achieves high compression rates but leads to performance drop. To address memory performance tradeoff, we train a light-weight scene-specific auto-encoder network that performs quantization-dequantization in an end-to-end differentiable manner updating both product quantization centroids and network parameters. Unlike standard L2 reconstruction loss for training auto-encoder network, we show that additional margin-based metric losses are key to achieve good performance. Results show that for a descriptor memory of 1 MB, we can achieve competitive performance on Aachen Day with only 5 % drop in performance.