I presented my work on Pommerman at the Deep Reinforcement Learning Workshop at NeurIPS this year: "Accelerating training in Pommerman with Imitation and Reinforcement Learning" (with Omkar Shelke, Richa Verma and Harshad Khadilkar). The main highlight for me was the RL social — a new addition by NeurIPS for informal discussions with prominent people in the field. I had a detailed discussion with Richard Sutton, David Silver, Martha White, and Michael Littman over topics ranging from causality in RL to moving away from the MDP framework entirely. The common notion that resonated with all of them: have a big picture in mind before delving into a very specific problem. If building AGI is the ultimate goal, place your work in that context.
In RL, people have been looking at sample-efficient RL, batch RL, meta-learning, and ablation studies of existing algorithms. What follows is a summary of talks and papers I found interesting. Recordings are available at slideslive.com/neurips/.
Tutorials
Imitation Learning and its Application to Natural Language Generation
Kyunghyun Cho, Hal Daume III
Focused on using imitation and reinforcement learning in NMT, dialogue, and story generation. The main challenge with beam search is lack of diversity — typically tackled by adding noise. RL with stochastic policies can help significantly here, especially for natural dialogue generation.
Efficient Processing of Deep Neural Networks: from Algorithm to Hardware
Vivienne Sze
Insights into designing efficient hardware for DL under constraints of speed, latency, energy, and cost. Discussion of CPUs, GPUs, FPGAs, and task-specific architectures, mostly focused on vision tasks.
Reinforcement Learning: Past, Present and Future Perspectives
Katja Hofmann
An elaborate session from basic MDP formulation to multi-agent RL, with a focus on generalization and policy evaluation. The Minecraft case study was particularly interesting for long-term reward and exploration challenges.
Invited Talks & Keynotes
Celeste Kidd: How to Know
Kidd's work centers on how humans (and babies) form beliefs — what they look at, where they focus, what predictions they make. Key takeaways:
- Humans continuously form beliefs — it's a probabilistic, ongoing process, not a one-shot decision.
- Certainty diminishes interest: agents learn most efficiently in the intermediate zone between fully known and fully unknown.
- Certainty is driven by feedback — without it, our models can be wildly off. Less feedback may encourage overconfidence.
- Humans form beliefs quickly, meaning the algorithms pushing content online have profound impacts on what we believe.
Her discussion of how small, confirmatory feedback loops solidify wrong assumptions resonated deeply. Standing ovation from the crowd.
Yoshua Bengio: From System 1 Deep Learning to System 2 Deep Learning
Inspired by Kahneman's "Thinking Fast and Slow":
- System 1: Intuitive, fast, unconscious, habitual — DL is good at this.
- System 2: Slow, logical, sequential, conscious, algorithmic — DL is not equipped for this.
What's missing in DL: out-of-distribution generalization, high-level cognition (causality), and world models that enable knowledge-seeking. The talk proposed moving toward sparse factor graphs and a "consciousness prior."
Michael Littman: Assessing the Robustness of Deep RL Algorithms
Built saliency maps by masking portions of the state space (DQN on Atari). Conclusion: DQN does not learn the state space as we see it. Although it has strategies to win games, it doesn't understand them. Evaluation metrics discussed: Value Estimation Error and total accumulated reward.
Selected Papers
Towards Explaining the Regularization Effect of Initial Large Learning Rate
Proves that lower learning rates learn myopic/local structure, while larger rates learn macro structure. Demonstrated on CIFAR with a superimposed image patch — high-LR networks learn the macro (generalizable) pattern.
Uniform Convergence May Be Unable to Explain Generalization in Deep Learning (Outstanding New Directions Paper)
Challenges the standard explanation for why over-parameterized networks generalize. Shows that generalization error also depends on training set size in ways uniform convergence cannot capture.
Causal Confusion in Imitation Learning
Behavioral cloning fails catastrophically on distributional shift. More information doesn't always help: a policy trained with brake-light information learns to brake whenever the brake light is on — even when it should accelerate. Targeted intervention by expert queries helps resolve this.
Learning to Control Self-Assembling Morphologies
Lego-style modules where each takes inputs + messages and produces outputs + messages. Shared policies across modules lead to robust behavior. The video is remarkable — the agent completes its task even after losing modules.
Weight Agnostic Neural Networks
How much can network structure alone achieve without training? Using random weights with fixed structure, they show that architecture alone yields 82% on MNIST — raising deep questions about the role of inductive biases.
Better Exploration with Optimistic Actor Critic
Policy gradient is too greedy, leading to conservative policies. Providing an upper bound (optimistic estimate) to critic values rather than a lower bound leads to better exploration. Improvement over SAC.
Hindsight Credit Assignment
Credit assignment is typically done over temporal scale with noisy proxies. This work explicitly learns relevant credit using posterior probabilities — analogous to figuring out why you got wet hours after the fact, rather than just associating it with the most recent actions.
Selected Posters
Reinforcement Learning
- Doubly-Robust Lasso Bandit — Kim & Paik, Seoul National University
- Multi-agent Common Knowledge Reinforcement Learning — Schroeder de Witt et al.
- Measuring the Reliability of RL Algorithms — Chan et al.
- Learning Efficient Representations for Intrinsic Motivation — Zhao, Tiomkin, Abbeel
- Benchmarking Safe Exploration in DRL — Ray, Achiam, Amodei
NLP
- Text-Based Interactive Recommendation via Constraint-Augmented RL — Zhang et al.
- Can Unconditional Language Models Recover Arbitrary Sentences? — Subramani, Bowman, Cho
Misc
- Adversarial Examples Are Not Bugs, They Are Features — Ilyas et al., Madry group (MIT)
- Putting an End to End-to-End: Gradient Isolated Learning — Löwe et al.