Yes, you need to be in the pen before the banner drops. They utilise a normal race mode in zwift to run the TTT’s, and you can’t join a race after it’s started, even if for the TTT your “start time” isn’t until some time later.
You can hit A to take you to the pairing screen and carry on staying loose if you want. Then just press Esc to get straight back to the game. It can help stop your power meter, etc, going to sleep.