Worked solutionOne step at a time
Count the delay in control steps
The policy acts every 20 ms. A 100 ms delay means each command takes effect five steps after the observation it was based on.
Why late corrections make it fall
Balance is feedback: lean left, push right. If the push arrives after the body has already swung back, it adds to the swing instead of cancelling it. For a wobble of 2.5 cycles a second, 100 ms is a quarter of a cycle.
Why the trained policy cannot cope
In training, cause and effect were one step apart. Nothing in the policy's experience says a command might land five steps later, so it keeps correcting a posture the robot has already left. The wobble feeds itself and grows.
Put the delay into training
Hold each action in a buffer and release it after a random delay, say 0 to 120 ms, drawn fresh for each episode. It also helps to give the policy its own last few actions as input, so it can tell what is still on its way to the motors.
Answer: C, randomise the delay
This is domain randomisation applied to time. The policy stops relying on an instant response and becomes robust across the whole range, which includes the real robot.
What the simulation showedOne number changed between the two runs
The reel runs the same Unitree G1 walking policy twice in MuJoCo.
- With commands applied instantly it walks.
- With every command held back by 100 ms, and nothing else changed, it goes down in about two seconds.
- No motor got weaker and no sensor got noisier. Timing alone is enough to break a policy that was trained without it.
Wrong answersWhy each one is tempting
| Pick | What went wrong |
|---|---|
| A · bigger network | Treats a missing-data problem as a capacity problem. |
| B · higher gains | The instinct from tuning a loop with no delay. With delay, gain eats your stability margin. |
| D · more reward | Changes what the policy wants, not what it has experienced. |
Follow-ups an interviewer asks nextTry these before opening them
Why randomise the delay instead of setting it to exactly 100 ms?
The real delay is not one number. It jitters with bus load, differs between robots and changes with firmware. A policy trained on one exact value can be as brittle as one trained on zero.
What else gets randomised for sim-to-real transfer?
Ground friction, link masses, motor strength, sensor noise, the terrain, and random pushes. Each one covers a way the simulator is wrong about the real robot.
What can you fix on the robot instead of in training?
Cut the delay itself: a faster bus, running the policy on the robot's own computer, and tighter motor driver loops. You can also predict the state forward by the known delay before feeding it to the policy.
How does a policy actually cope with delay?
With its recent actions as input, or a recurrent memory, it can estimate where the body will be when the command lands and act for that moment instead of for now.
