I know I'm young at 24, but thought I could throw something out there. As said earlier, us guys love it when girls are blunt. And now that you guys are talking more, I think you should just ask him if he wants to do something after work. If you leave the option open for during work hours, he probably just sees it as lunch with a coworker. And let's be honest, that's a HUGE difference.
When a coworker asks me to lunch (granted it's always just guys) I usually respond with something like "Not a good day, sorry." Especially if I'm legit busy that day. He could be interested, he could not be. So you might as well find out for sure. And even if he says no, you don't work with him anyways, so it's not like you'll see him everyday.
Good luck